WO2024099313A1 - 一种云边端协同智能婴儿护理系统及方法 - Google Patents

一种云边端协同智能婴儿护理系统及方法 Download PDF

Info

Publication number
WO2024099313A1
WO2024099313A1 PCT/CN2023/130232 CN2023130232W WO2024099313A1 WO 2024099313 A1 WO2024099313 A1 WO 2024099313A1 CN 2023130232 W CN2023130232 W CN 2023130232W WO 2024099313 A1 WO2024099313 A1 WO 2024099313A1
Authority
WO
WIPO (PCT)
Prior art keywords
edge
cloud
recognition
baby
devices
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/130232
Other languages
English (en)
French (fr)
Inventor
谭明奎
陈良玮
陈耀佛
胡玮文
游增
曾景辉
吉冬昱
连梓豪
邓炜滔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Publication of WO2024099313A1 publication Critical patent/WO2024099313A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
    • A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning

Definitions

  • the present invention relates to the field of infant monitoring technology, and in particular to a cloud-edge-end collaborative intelligent infant care system and method.
  • an object of the present invention is to provide a cloud-edge-end collaborative intelligent infant care system and method.
  • a cloud-edge-device collaborative intelligent infant care system comprising:
  • a sensor module used for collecting image information and sound information from the baby crib
  • the edge device is used to perform crying recognition and physical sign recognition based on the collected image information and sound information, wherein the physical sign recognition includes at least one of heart rate and respiratory rate measurement, mouth and nose cover detection, height measurement, skeleton recognition or body temperature detection. kind;
  • Cloud devices are used to train and update the detection model and output prompt information based on the recognition results
  • a cross-edge cloud collaborative framework is adopted to realize application management, peripheral management, deployment and communication between edge and cloud devices.
  • the crying sound recognition and vital sign recognition are implemented using a neural network model;
  • the edge device and the cloud device reduce the model parameters that are not sensitive to the task by means of parameter sharing and clipping, thereby reducing the model storage and communication overhead, reducing the network scale and reducing the computing power requirements;
  • the neural network model When the network communication is good, the neural network model will be segmented, and the cloud device will first perform the front-end computing tasks of the model, and then the cloud device will transmit the intermediate results to the edge device, which will perform the remaining detection work and obtain the final result;
  • the edge end takes on all detection work to ensure that the final results can be obtained in real time and quickly;
  • cloud devices will train models based on the data uploaded by edge devices and update the models on edge devices; edge devices will use the latest models to perform detection tasks based on the information collected by sensors.
  • the sensor module includes an infrared camera, an RGB camera, a thermal sensor and a microphone.
  • the recognition results include baby crying, quiet scenes, noisy scenes and baby laughing;
  • the support vector machine is obtained in the following way;
  • the support vector machine is obtained based on the extracted features.
  • Spectral bandwidth refers to the second-order spectral bandwidth and is defined as follows:
  • Sk represents the spectrum of the kth window
  • fk represents the frequency corresponding to the kth data point of the spectrum.
  • the cloud device is a server with a Linux operating system and a high-performance computing device.
  • the device is an embedded device equipped with a series of sensors;
  • the waveform data, timestamp, and sampling rate are packaged as sound information through the message queue and sent to the subsequent processing process.
  • the sound information is processed as follows:
  • the audio waveform data in the buffer reaches or exceeds n seconds, the data are spliced and sent to the model for crying recognition.
  • the specific steps of measuring height according to the image information include:
  • the joint point information includes pixel coordinates of seven key skeleton joint points, namely, the top of the infant's head, cervical spine, pelvis, left and right knees, and left and right heels;
  • the camera parameters obtain the spatial three-dimensional coordinates of the key skeleton joint points in the coordinate system with the camera optical center as the origin;
  • the distances between the joints are calculated based on the three-dimensional coordinates, and the height of the baby is obtained based on the distances between the joints.
  • the length of the left and right lower limbs was averaged to obtain the final lower limb length, that is:
  • L HN is the distance between the head vertex node and the cervical node
  • L NP is the distance between the cervical node and the pelvic node
  • L PK is the distance between the pelvic node and the knee node
  • L KF is the distance between the knee node and the heel node.
  • the method for building a cloud-edge-device collaborative intelligent baby care system as described above includes the following steps:
  • the cloud device is a server with Linux operating system and the edge device is an embedded device equipped with a series of sensors;
  • the present invention collaborates between the edge and the cloud, reasonably optimizes the task allocation strategy, disassembles the intelligent algorithm, uses the powerful computing power of the cloud to undertake public computing tasks, reduces the computing pressure at the edge, and meets the requirements of intelligent algorithms and computing power.
  • FIG1 is a structural block diagram of a cloud-edge-device collaborative intelligent infant care system according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of measuring the height of an infant in an embodiment of the present invention.
  • orientations such as up, down, front, back, left, right, etc.
  • orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
  • this embodiment provides a cloud-edge-end collaborative intelligent infant care system, which is built through the following steps: the first step is to build and configure the crib hardware, the cloud device is a server with a Linux operating system with a high-performance computing device (such as NVIDIA GPU), and the edge device is an embedded device equipped with a series of sensors (such as NVIDIA Jetson Nano); the second step is to build a crib cloud-edge-end collaborative cluster.
  • the present invention will achieve application management and deployment between edge and cloud based on a cross-edge-cloud collaborative framework (such as KubeEdge); the third step is algorithm model training and engineering.
  • the system can realize the training and deployment of six intelligent applications such as baby cry recognition, baby height measurement, baby heart rate and respiratory rate measurement, baby skeleton recognition, baby mouth and nose cover detection, and baby temperature detection, and adopts a high-performance deployment framework (such as NVIDIA TensorRT) for inference acceleration.
  • six intelligent applications such as baby cry recognition, baby height measurement, baby heart rate and respiratory rate measurement, baby skeleton recognition, baby mouth and nose cover detection, and baby temperature detection
  • a high-performance deployment framework such as NVIDIA TensorRT
  • the solution of the present invention uses a computer compatible with the Linux operating system as a cloud device, and the edge devices mainly include edge reasoning embedded devices and sensor devices.
  • the edge reasoning embedded device collects sensor device data for end-to-end reasoning or cloud-edge collaborative reasoning, and obtains the final reasoning results and feeds them back to the user;
  • the sensor devices include infrared binocular cameras (such as the MYNT binocular camera depth version D1000-120 series and 850nm near-infrared flood light source), RGB cameras (such as ZED binocular cameras, Hikvision DS-E12 cameras), thermal sensors (such as Dali SD 16W miniaturized thermal imaging body temperature rapid screening module), and microphones (such as external dual microphones).
  • Edge inference development equipment The NVIDIA series of edge inference embedded devices used in the present invention are characterized by high computing density, high energy efficiency, and strong deep learning algorithm inference capabilities of edge devices.
  • the device has an operating power as low as 10 watts, a performance of up to 32TOPS, and low heat generation, and can be used in smart home scenarios such as cribs. Its internal storage space can store multiple deep learning models, can run complex applications and enhance their real-time processes.
  • the device supports developers to use NVIDIA's rich development tools and workflows, which can help developers quickly train and deploy neural networks.
  • the infrared imaging unit used in the present invention can be used for a variety of measurement and detection tasks, such as height measurement, posture recognition, etc.
  • the camera has the following features: it has infrared/visible light imaging function, so it can be used with active IR structured light to solve the recognition problem of indoor white walls and textureless objects; at the same time, it can be used with IR floodlight to achieve three-dimensional measurement at night; the camera has its own chip to calculate the depth map, which does not consume the computing power of the computing platform, and has IR fill light to enhance the depth map effect; the camera has a 120-degree field of view and a large observation range.
  • the main visible light imaging device of the present invention has the main function of providing visible light video stream.
  • the imaging device used in the present invention needs to take into account both visible light and near-infrared imaging, so the infrared filter is eliminated from its lens.
  • the present invention uses an additional visible light camera as a supplement.
  • the additional visible light camera has the following characteristics: high resolution, long measurement distance, and can provide clear video streams.
  • the main parameters of the camera are 1920x1080 resolution, 3.6 mm lens, higher resolution and measurement distance, and can provide measurement data for multiple tasks, and provide extensive SDK support in three-dimensional measurement, positioning, bone recognition, etc. However, this module does not have night imaging function.
  • the thermal imaging temperature measurement module used in the present invention provides a non-contact temperature measurement function.
  • the module has a resolution of 160x120, is equipped with a 3.85 mm lens, and has a large measurement field of view.
  • the infrared array sensor independently developed and produced can be integrated with advanced algorithms such as non-blocking algorithm, thermal imaging precision temperature measurement, and ambient temperature adaptive calibration according to application requirements.
  • the SPI interface is used to output digital infrared images, and the universal serial port is used to communicate and control the components, which is convenient for integration and docking with other MCU-controlled devices.
  • the module also has the advantages of light weight, low power consumption, and high cost performance. It is suitable for various infrared temperature measurement and monitoring fields such as over-temperature alarm of industrial online temperature measurement equipment, flow meters, handheld instruments, fire monitoring alarms, etc.
  • the present invention uses external dual microphones as sound collection devices to provide functions such as baby crying detection, intention analysis, sleep state detection, and ambient sound detection.
  • the present invention adopts a cross-edge-cloud collaborative framework to achieve application management, peripheral management, deployment and communication between the edge and the cloud.
  • cloud computing and edge computing need to process and analyze data in real time or faster, save network traffic, run offline and support breakpoint resumption, and provide higher security protection for local data.
  • Cloud-edge collaborative cluster working mode adopts a cross-edge-cloud collaborative framework.
  • the neural network model will be divided.
  • the cloud will first perform the computing tasks of the model front end, and then the cloud will transmit the intermediate results to the edge end, and the edge end will perform the remaining detection work to obtain the final result.
  • the cloud will also train the model based on the data uploaded by the edge end, and will update the edge end model from time to time.
  • the edge end collects edge device data and uses the latest model to perform detection tasks; in the case of poor network communication, the edge end will take on all detection work to ensure that the final result can be obtained in real time and quickly.
  • the present invention reduces model parameters that are insensitive to the task through methods such as parameter sharing and clipping, thereby reducing model storage and communication overhead, reducing network scale and reducing computing power requirements; at the same time, the present invention also quantifies the model weights to reduce the calculation of the model on the edge cloud.
  • the collaborative framework used in this invention relies on Kubernetes' container orchestration and scheduling capabilities to provide an open source system with infrastructure support for network, application deployment, and metadata synchronization between cloud and edge.
  • the collaborative framework combines the core advantages of cloud native. It solves the challenges in edge computing scenarios, supports edge device management, and adapts to complex edge-cloud network environments.
  • edge-cloud collaborative deep learning Supports edge-cloud collaborative data set management and model management, making it easier for developers to quickly develop edge-cloud collaborative intelligent applications;
  • the cloud-edge collaborative cluster provides memory function through the cloud knowledge base, allowing the sample knowledge accumulated at the edge to be continuously updated and persisted, thereby dealing with the problem of catastrophic forgetting; combined with incremental training and multi-task training, it simultaneously realizes knowledge transfer across time and scenarios, thereby better handling unknown tasks.
  • the present invention completes the training and engineering deployment of six intelligent algorithms.
  • the six intelligent algorithms include: baby cry recognition, baby height measurement, baby heart rate and respiratory rate measurement, baby skeleton recognition, baby mouth and nose cover detection, and baby temperature detection. Inference acceleration is performed based on an efficient deployment framework.
  • Baby cry recognition Crying is the main way for babies to communicate with the outside world. Babies convey their needs to the outside world through crying, and guardians need to meet their needs based on the baby's crying.
  • the training dataset of the present invention includes four categories of scene audio: baby crying, quiet scene, noisy scene and baby laughing, wherein the baby crying audio comes from the ESC-50 dataset, and the audio clips in the dataset are manually extracted from the public audio data collected by the Freesound.org project.
  • the ESC-50 dataset is a dataset for sound classification, consisting of 2000 labeled audios, each of which is about 5 seconds long and can be divided into 50 categories.
  • S102 Feature extraction First, the input audio signal is preprocessed, and the audio of the training set is divided into 5-second segments. Then, based on the characteristics of the baby's crying, the input audio signal is extracted with Mel frequency cepstrum coefficients, spectrum roll-off points, spectrum centroid, zero crossing rate, spectrum bandwidth, root mean square energy and other features.
  • S103 model training and engineering Design a detection algorithm based on support vector machine and use the data set proposed by S-1 for training.
  • the designed algorithm is simple in principle, easy to implement and has fast calculation speed. It is more suitable for real-time processing and has impressive performance, achieving an accuracy rate of 97.22% on the crying recognition data set.
  • the specific steps of baby crying recognition are as follows:
  • S1-1 Call the ALSA driver interface of the Linux operating system, initialize the microphone device, and set acquisition parameters such as sampling rate.
  • ALSA stands for Advanced Linux Sound Architecture, which is the current mainstream audio architecture in the Linux operating system. It can be used for common audio operations such as recording and playing audio. Common sound card devices on the market are all supported by ALSA, and it is also the way most applications currently interact with audio hardware.
  • This embodiment uses the recording function of ALSA through the official user space alsa-lib function library of ALSA.
  • the microphone device When in use, the microphone device must first be initialized with specified parameters, where the hardware parameters mainly include: the data format is a little-endian 16-bit integer, the number of channels is 1, the sampling rate is 48000Hz and other similar sampling rates can be selected as needed, and automatic resampling is disabled; the software parameters mainly include: turning on the timestamp, the timestamp type is GETTIMEOFDAY. Finally, the snd_pcm_prepare function is called to complete the initialization process.
  • the sampling rate indicates the number of times the microphone hardware completes signal sampling per second in the process of converting the analog signal of the mechanical wave of the sound into a digital signal.
  • S1-2 Get the sound card hardware timestamp from ALSA.
  • the snd_pcm_htimestamp function can obtain the number of audio samples in the current hardware queue and the timestamp recorded by the hardware at the same time.
  • the timestamp of the first sample that has not been processed in the hardware queue can be converted by the sampling rate.
  • S1-3 Read audio waveform data fragments from ALSA, that is, the original samples of the sound card sampling the mechanical waves of the sound.
  • the snd_pcm_readi function can read a specified number of samples in the hardware queue, and wait if the number of samples is insufficient. This implementation reads 1024 samples each time, converting about 20ms of audio waveform data according to the sampling rate. By calling snd_pcm_recover, abnormal situations such as hardware queue overflow caused by insufficient software running speed are ignored. Subsequent processing can determine whether there is data loss based on the timestamp.
  • S2 Buffers audio segments and manages buffer content based on load.
  • n is the pre-specified detection window length. A longer window can improve the accuracy of detection, but it will also increase the algorithm's computational complexity and latency. In this implementation, n is set to 5 seconds.
  • S2-3 Determine whether the current audio segment is continuous with the previously buffered content based on the timestamp. If not, clear the buffer and detection result queue.
  • the discontinuity may be caused by the active discarding of data packets in the previous step, or by the overflow of the buffer queue or message queue of the microphone hardware.
  • S3-1 Before the system is deployed, a dataset consisting of audio of crying babies and random environmental sounds is collected. The audio is segmented and resampled, and then features such as Mel frequency cepstral coefficients, spectrum roll-off points, spectrum centroid, zero crossing rate, spectrum bandwidth, and root mean square energy are extracted, and a support vector machine is fitted with these features.
  • the zero-crossing rate refers to the number of times the waveform crosses the zero point as a proportion of the total number of samples, and is calculated using a sliding window with a window size of 2048 samples and a step size of 512 samples.
  • the root mean square energy represents the square root of the mean of the square of the vibration amplitude at each sampling point, and is calculated using a sliding window with a window size of 2048 samples and a step size of 512 samples.
  • MFCC Mel-frequency cepstral coefficients
  • the spectrum roll-off point indicates that 90% of the energy is distributed at this frequency and below, and is calculated with a sliding window of 2048 samples as the window size and 512 samples as the step size.
  • Spectral bandwidth represents the second-order spectral bandwidth, defined as The sliding window calculation is performed with a window size of 2048 samples and a step size of 512 samples.
  • S3-2 During runtime, receive the n-second audio clip output by S2, perform resampling (if the sampling rate is different from that during training), extract features, and classify using the support vector machine fitted in the previous step.
  • the recognition method of this embodiment further extracts more robust features based on the Mel spectrum and other methods. It is more robust than simple circuits or methods based only on the Mel spectrum and can better adapt to complex practical application environments. Compared with methods based on deep learning, it is simpler in training and deployment, and has low computing power requirements, which helps to reduce application costs.
  • a predefined alarm action is triggered, which may include: displaying an alarm message on the terminal; connecting to a remote server and sending a push message to the guardian's smartphone; playing music to automatically soothe the baby, etc.
  • Infant skeleton recognition mainly provides support for other intelligent infant monitoring functions, such as dangerous action recognition, infant height measurement, etc.
  • the existing lightweight skeleton recognition network is used to reduce the inference cost in low-power devices.
  • This lightweight skeleton recognition network optimizes the OpenPose method and can be inferred in real time on edge devices with negligible accuracy loss.
  • the detection model can detect 18 key points of the human body: ears, eyes, nose, neck, shoulders, elbows, wrists, hips, knees and ankles. Its average accuracy in the skeleton recognition task in COCO 2017 is 2.38%. The accuracy can reach 40%.
  • Infant height measurement The infant stage is the first peak period of children's growth. Height measurement can effectively reflect the growth and development of infants and is an important criterion for measuring health.
  • S201 collects baby pictures and their corresponding depth maps through a depth camera.
  • the infrared binocular camera used in the present invention can collect high-definition visible light images of babies in real time, and the camera has its own chip to calculate the depth map corresponding to the image, effectively reducing the computing burden of the edge device.
  • S202 uses a skeleton extraction algorithm to extract skeleton key points.
  • the present invention uses the above-mentioned infant skeleton recognition algorithm to obtain the coordinates of key points such as the top of the infant's head, knees, ankles, etc. And according to the key point coordinates, the depth information of the corresponding position of the depth map is converted into a three-dimensional point cloud, that is, the XYZ coordinates of the key point in the three-dimensional coordinate system with the camera optical center as the origin.
  • S203 estimates the height of the baby based on the three-dimensional point cloud coordinates. Based on the three-dimensional point cloud coordinates of the key points measured by S-2, the length of the lower limbs, the length of the trunk and the length of the head can be calculated. The height of the baby can be expressed as the sum of the length of the lower limbs, the length of the trunk and the length of the head. In order to reduce the measurement error, the length of the left and right lower limbs is averaged to obtain the final length of the lower limbs. This algorithm effectively solves the problem of the baby's weird posture and lack of cooperation.
  • the specific steps of measuring the height of the baby are as follows:
  • S1-1 Obtain the baby's RGB image and the corresponding depth image in real time through a deep binocular camera.
  • the deep binocular camera can be any common model currently available on the market, and no additional restrictions are imposed here.
  • S1-2 Use a skeleton joint extraction algorithm (such as the OpenPose algorithm) to perform real-time detection and processing on the infant's RGB image to extract the coordinates of the infant's skeleton joints.
  • the skeleton joint extraction algorithm can use any of the currently available mature algorithms, and no additional restrictions are imposed here.
  • S2-1 Obtain the internal parameters of the camera sensor by reading the hardware information provided by the camera manufacturer, including the camera's pixel resolution (w, h), pixel size (d) and lens focal length (f). If the camera manufacturer does not provide internal parameter information, it is necessary to obtain the above parameters through the camera calibration method. The current calibration method is very mature and will not be described here. Usually, the internal parameters of the camera sensor are fixed after leaving the factory and generally do not change during use. If recalibration is required, any of the currently available mature calibration methods can be used, and there is no restriction here;
  • the pinhole model is usually used to describe the process of a camera mapping three-dimensional spatial coordinates to two-dimensional planar coordinates. Effective description, according to the optical principle of camera imaging, the pixel coordinates (u, v) of the key skeleton joint points and their corresponding three-dimensional spatial coordinates (X, Y, Z) satisfy the following relationship:
  • u 0 and v 0 represent the horizontal and vertical pixel differences between the center pixel coordinates and the origin pixel coordinates of the RGB image, that is,
  • the height of an infant can be expressed as the sum of the head length, trunk length and lower limb length, among which the head length can be expressed as the distance from the top of the head (H) to the cervical vertebra (N), the trunk length can be expressed as the distance from the cervical vertebra (N) to the pelvis (P), and the lower limb length can be expressed as the sum of the distances from the pelvis (P) to the knee (K) and from the knee (K) to the heel (F).
  • any two points A and B in space if their three-dimensional coordinates are known to be (X A , Y A , Z A ) and (X B , Y B , Z B ) respectively, the length of the line connecting the two points AB can be expressed as:
  • the lengths of the left and right lower limbs are averaged to obtain the final lower limb length, that is:
  • Infant heart rate and respiratory rate measurement The structure and function of the cardiac conduction system of newborns are still in the development stage after birth, and the incidence of arrhythmia is relatively high. Heart rate monitoring can detect such diseases early.
  • the infant heart rate and respiratory rate measurement of the present invention are both monitored by visual information, without the need to introduce additional edge devices.
  • Infant heart rate measurement The color of facial blood vessels, especially the forehead area, will produce slight color changes with the heartbeat.
  • the infant heart rate measurement algorithm first extracts the pixel values of specific areas of the face, then goes through a series of signal processing and noise reduction, and finally measures the change frequency to derive the heart rate.
  • the breathing rate is determined by the frequency of chest rise and fall.
  • the algorithm first constructs a Laplace-Gaussian image pyramid for each frame in the video, and then performs Fourier transform on each pixel at each scale to achieve Euler amplification, amplify subtle movements, and record the movement area. The frequency of periodic motion in the area is then measured. Since the frequency of the signal is low, in order to reliably detect the peak, a low-pass filter is used to create a smoother signal to derive the breathing rate.
  • Infant mouth and nose covering detection During sleep, infants may have their mouths and noses covered by bedding, pillows, etc. due to poor sleeping posture, which may cause huge safety hazards. Infant caregivers should take measures to ensure the safety of infants. Since the facial key point detection of the two tasks of mask recognition and ear and nose covering is basically the same, the present invention considers using a mask detection dataset as a supplement to the training dataset, which includes nearly 8,000 (mouth and nose covered) face annotated images, and the images are from the WIDER Face and MAFA datasets. During the training process, a lightweight detection model is used to reduce the amount of edge computing, and its average accuracy can reach 89.6%, which can meet most of the needs of real scenes.
  • Infant temperature detection Infant temperature is an important indicator for judging whether the baby is healthy. Real-time temperature detection can help guardians find abnormalities in the baby's physical condition in time.
  • the present invention uses a thermal sensor to detect the temperature of the object being tested in real time. First, the face of the object being tested is detected, and then the temperature of a fixed area of the forehead is taken to take the maximum value of the pixel points of 35-42 degrees as the final detection result. This detection method has the advantages of fast reasoning speed and no need to contact the baby.
  • the solution of the present invention adopts a cloud-edge collaborative cluster.
  • the edge is responsible for collecting data and uploading it to the cloud.
  • the neural network model will be segmented.
  • the cloud undertakes the computing tasks of the front end of the model, and then transmits the intermediate results to the edge.
  • the edge continues to perform the prediction work and obtains the final result.
  • the cloud will also design and train intelligent models based on the data uploaded by the edge and upgrade the models from time to time for use by the edge; when the network conditions are poor, the edge undertakes the model reasoning task.
  • This solution can collaborate between the edge and the cloud, reasonably optimize the task allocation strategy, disassemble the intelligent algorithm, use the powerful computing power of the cloud to undertake public computing tasks, reduce the computing pressure at the edge, and meet the computing power requirements of the intelligent algorithm.
  • this embodiment has the following advantages and beneficial effects:
  • the solution of the present invention supports multiple intelligent algorithms such as baby cry recognition and baby height measurement, and provides low-latency and high-throughput deployment reasoning for deep learning applications through high-performance software frameworks such as TensorRT to meet the needs of real-time reasoning at the edge.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Physiology (AREA)
  • Cardiology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Dentistry (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Pulmonology (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

一种云边端协同智能婴儿护理系统及方法，其中系统包括：传感器模块，用于采集婴儿床上的图像信息和声音信息；边缘设备，用于根据采集的图像信息和声音信息进行哭声识别以及体征识别；云端设备，用于对检测模型进行训练以及更新，以及根据识别结果输出提示信息；其中，采用跨边云的协同框架来实现边缘端与云端设备之间的应用管理、外设管理、部署以及通信。在边缘与云端之间进行协同，合理优化任务分配策略，拆解智能算法，利用云端强大的计算能力承担公共的计算任务，减轻边缘的计算压力，满足智能算法、算力需求，可广泛应用于婴儿监护技术领域。

Description

一种云边端协同智能婴儿护理系统及方法 技术领域
本发明涉及婴儿监护技术领域，尤其涉及一种云边端协同智能婴儿护理系统及方法。
背景技术
每年都有成千数万名婴儿突然的意外死亡，其中婴儿猝死综合症(又称摇篮死亡)仍然是导致全球婴儿死亡的主要因素。最常见的原因是床上用品阻塞了婴幼儿的呼吸道和监护者身体压到婴儿身上导致婴儿窒息死亡。为了降低婴儿睡眠相关的死亡风险，美国儿科学会发表的最新申明指出，婴儿出生第一年应与父母睡在同一房间，但婴儿应睡在自己的空间里(婴儿床)。近年来，随着深度学习和物联网等前沿技术的快速发展，智能化已成为婴儿床市场新趋势。一般来说，智能婴儿床在在传统婴儿床的基础上，附加了监控护理功能，可以实时监控婴儿在智能婴儿床上的状态，帮助监护人实时了解婴儿动态。
然而，现有智能婴儿床产品大多基于传感器采集信息进行简单的信息处理和逻辑判断，难以识别和应对复杂场景，无法满足现实场景下的婴儿看护需求。为了解决上述问题，部分产品尝试利用深度学习算法实现智能感知与决策，但是深度学习算法的资源需求大，婴儿床等边缘设备处理能力往往受限于计算资源难以满足。而且智能任务需求多样，包括数据采集、预处理、计算、控制和反馈等，边缘端设备往往仅具有单一能力，极大约束了智能化设备的广泛应用。为了寻求更高的算力解决，部分产品使用云端服务器进行推理，然而云端设备实时性不够，带宽不足，能耗较大，不利于数据安全和隐私。
发明内容
为至少一定程度上解决现有技术中存在的技术问题之一，本发明的目的在于提供一种云边端协同智能婴儿护理系统及方法。
本发明所采用的技术方案是：
一种云边端协同智能婴儿护理系统，包括：
传感器模块，用于采集婴儿床上的图像信息和声音信息；
边缘设备，用于根据采集的图像信息和声音信息进行哭声识别以及体征识别，其中体征识别包括心率与呼吸频率测量、口鼻遮掩检测、身高测量、骨架识别或体温检测中的至少一 种；
云端设备，用于对检测模型进行训练以及更新，以及根据识别结果输出提示信息；
其中，采用跨边云的协同框架来实现边缘端与云端设备之间的应用管理、外设管理、部署以及通信。
进一步地，所述哭声识别和体征识别采用神经网络模型来实现；边缘设备和云端设备通过参数共享和裁剪的方式，减少对任务不敏感的模型参数，从而降低模型存储和通信开销、减少网络规模和降低算力需求；
在网络通信良好的情况下，神经网络模型将会被分割，先由云端设备执行模型前端的计算任务，然后云端设备将中间结果传输给边缘设备，由边缘端执行剩下的检测工作，得出最终结果；
在网络通信不好的情况下，边缘端承担任全部的检测工作，以确保能够实时、快速地获得最终结果；
此外，云端设备还会根据边缘设备上传的数据来训练模型，并更新边缘设备上的模型；边缘设备根据传感器采集到的信息，使用最新的模型执行检测任务。
进一步地，所述传感器模块包括红外摄像头、RGB摄像头、热传感器和麦克风。
进一步地，根据声音信息进行哭声识别的具体步骤如下：
对声音信息进行预处理，获取音频信号；
对音频信息进行重采样以及提取特征，根据提取的特征和预设的支持向量机进行分类识别，获得识别结果；所述识别结果包括婴儿哭声、安静场景、嘈杂场景和婴儿笑声；
所述支持向量机通过以下方式获得；
获取数据集，对数据集进行重采样，并进行特征提取；其中，提取的特征包括梅尔频率倒谱系数、频谱滚降点、频谱质心、过零率、频谱频宽、均方根能量；
根据提取的特征拟合获得支持向量机。
进一步地，所述频谱质心定义如下：
C＝∑kSkfk/∑kSk
频谱频宽表示二阶频谱频宽，定义如下：
其中，Sk表示第k个窗口的频谱，fk表示频谱第k数据点对应的频率。
进一步地，云端设备是一台带有高性能计算设备的Linux操作系统的服务器，边缘端设 备是一台装有一系列传感器的嵌入式设备；
通过以下方式采集获取声音信息：
调用Linux操作系统的ALSA驱动接口，初始化麦克风，设置采集参数；
从ALSA获取声卡硬件时间戳，以及读取音频的波形数据，即声卡对声音的机械波进行采样的原始样本；
通过消息队列将波形数据和时间戳、采样率进行打包，作为声音信息并发送至后续处理进程。
进一步地，在进行哭声识别前，对声音信息进行以下处理：
从消息队列中读取消息；
对比当前读取到的消息的时间戳和当前时间，若时间相差超过预设阈值，判定目前检测延迟过高，丢弃所有时间戳在n秒前的未处理的数据包；其中，n为预设的检测窗口长度；
根据时间戳判断当前音频片段是否和之前缓冲的内容连续，若不连续则清空缓冲和检测结果队列；
将当前消息中的音频波形数据加入缓冲；
当缓冲中的音频波形数据达到或超过n秒后，将这些数据进行拼接，并送入模型中进行哭声识别。
进一步地，根据图像信息进行身高测量的具体步骤包括：
获取深度图像，根据深度图像获取婴儿的关节点信息；其中，关节点信息包括婴儿的头顶、颈椎、盆骨、左右膝盖以及左右脚后跟共7个关键骨架关节点的像素坐标；
根据相机的参数，获取关键骨架关节点在以相机光心为原点的坐标系下的空间三维坐标；
根据三维坐标计算各关节点间距离，根据各关节点间距离计算获得婴儿的身高。
进一步地，最终身高的计算公式如下：
Height＝LH-N+LN-P+LP-K+LK-F
为了减少测量误差，对左右下肢的长度取平均得到最终的下肢长度，即：

其中，LH-N为头顶节点到颈椎节点之间的距离，LN-P为颈椎节点到盆骨节点之间的距离，LP-K为盆骨节点到膝盖节点之间的距离，LK-F为膝盖节点到脚后跟节点之间的距离。
本发明所采用的另一技术方案是：
如上所述的一种云边端协同智能婴儿护理系统的搭建方法，包括以下步骤：
搭建与配置硬件设备，其中，云端设备是一台Linux操作系统的服务器，边缘端设备是一台装有一系列传感器的嵌入式设备；
搭建边缘设备和云端设备之间的协同集群，根据基于跨边云的协同框架实现边缘设备和云端设备之间的应用管理与部署；
在边缘设备和云端设备上部署模型，以及训练模型。
本发明的有益效果是：本发明在边缘与云端之间进行协同，合理优化任务分配策略，拆解智能算法，利用云端强大的计算能力承担公共的计算任务，减轻边缘的计算压力，满足智能算法、算力需求。
附图说明
为了更清楚地说明本发明实施例或者现有技术中的技术方案，下面对本发明实施例或者现有技术中的相关技术方案附图作以下介绍，应当理解的是，下面介绍中的附图仅仅为了方便清晰表述本发明的技术方案中的部分实施例，对于本领域的技术人员而言，在无需付出创造性劳动的前提下，还可以根据这些附图获取到其他附图。
图1是本发明实施例中一种云边端协同智能婴儿护理系统的结构框图；
图2是本发明实施例中婴儿身高测量的示意图。
具体实施方式
下面详细描述本发明的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，仅用于解释本发明，而不能理解为对本发明的限制。对于以下实施例中的步骤编号，其仅为了便于阐述说明而设置，对步骤之间的顺序不做任何限定，实施例中的各步骤的执行顺序均可根据本领域技术人员的理解来进行适应性调整。
在本发明的描述中，需要理解的是，涉及到方位描述，例如上、下、前、后、左、右等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。
在本发明的描述中，若干的含义是一个或者多个，多个的含义是两个以上，大于、小于、超过等理解为不包括本数，以上、以下、以内等理解为包括本数。如果有描述到第一、第二 只是用于区分技术特征为目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量或者隐含指明所指示的技术特征的先后关系。
本发明的描述中，除非另有明确的限定，设置、安装、连接等词语应做广义理解，所属技术领域技术人员可以结合技术方案的具体内容合理确定上述词语在本发明中的具体含义。
如图1所述，本实施例提供一种云边端协同智能婴儿护理系统，该系统通过以下步骤进行搭建：第一步搭建与配置婴儿床硬件，云端设备是一台带有高性能计算设备(如NVIDIA GPU)的Linux操作系统的服务器，边缘端设备是一台装有一系列传感器的嵌入式设备(如NVIDIA Jetson Nano)；第二步搭建婴儿床云边端协同集群，本发明将基于跨边云的协同框架(如KubeEdge)实现边云之间的应用管理与部署；第三步算法模型训练和工程化。通过该系统能够实现婴儿哭声识别、婴儿身高测量、婴儿心率与呼吸频率测量、婴儿骨架识别、婴儿口鼻遮掩检测、婴儿体温检测等六种智能应用的训练与部署，并采用高性能部署框架(如NVIDIA TensorRT)进行推理加速。下面将结合图1对图中的每个步骤进行详细说明：
婴儿床硬件搭建与配置
本发明方案使用一台兼容Linux操作系统的计算机作为云端设备，边缘端设备主要包括边缘推理嵌入式设备和传感器设备。边缘推理嵌入式设备其作用是收集传感器设备数据进行端对端推理或云边端协同推理，并得到最终推理结果反馈到用户；传感器设备包括红外双目摄像头(如小觅双目摄像头深度版D1000-120系列及850nm近红外泛光光源)、RGB摄像头(如ZED双目摄像头、海康威视DS-E12摄像头)、热传感器(如大立SD 16W微型化热成像体温快速筛检模组)、麦克风(如外置双麦克风)。
(1)边缘推理开发设备：本发明使用的NVIDIA系列的边缘推理嵌入式设备，其特点是边缘设备的计算密度高、能效高、深度学习算法推理能力强。改设备运行功率低至10瓦，性能高达32TOPS，发热小，可适用于婴儿床等家居智能场景中。其内部存储空间可以存储多个深度学习模型，能够运行复杂的应用程序并增强其实时流程。并且该设备支持开发者使用NVIDIA丰富的开发工具和工作流，能够帮助开发人员快速训练和部署神经网络。
(2)红外双目摄像头：本发明使用的红外成像单元可被用于多种测量和检测任务，如身高测量、姿态识别等。该摄像头具备以下特点：具备红外/可见光成像功能，因此可搭配主动IR结构光，解决室内白墙和无纹理物体的识别难题；同时配合IR泛光光源，可以实现夜间三维测量；该摄像头自带芯片计算出深度图，不耗费计算平台的算力，并且带IR补光提升深度图效果；该摄像头具有120度视场角，观测范围大。
(3)可见光双目摄像头：本发明的主要可见光成像设备其主要作用是提供可见光视频流。 本发明使用的成像设备由于需要兼顾可见光/近红外成像，因此其镜头取消了红外滤光片。为了避免可能存在的日间拍摄偏色问题，本发明使用额外的可见光摄像头作为补充。额外的可见光摄像头具备以下特点：分辨率高，测量距离远，可以提供清晰的视频流，该摄像头主要参数为1920x1080分辨率，3.6毫米镜头，具有更高的分辨率和测量距离，能够为多个任务提供测量数据，在三维测量、定位，骨骼识别等方面提供广泛的SDK支持。但是该模组不具备夜间成像功能。
(4)热传感器：本发明使用的热成像测温模组提供非接触式测温功能。该模组具有160x120分辨率，配备3.85毫米镜头，具有较大的测量视野。采用自主研发生产的红外阵列传感器，根据应用需求可集成无挡片算法、热像精准测温、环境温度自适应校准等先进算法。采用SPI接口输出数字红外图像，通用串口对组件进行通讯控制，便于与其它MCU主控的设备集成对接。该模组并具有重量轻，功耗低，性价比高等优势。适合应用于工业在线测温设备过温报警、人流量计、手持仪器、火灾监控报警等各种红外测温监控领域。
(5)麦克风：本发明使用外置双麦克风作为声音采集设备，提供婴儿哭声检测、意图分析、睡眠状态检测以及环境音检测等功能。
婴儿床云边端协同集群的搭建
本发明采用跨边云的协同框架，从而实现边缘端与云端之间的应用管理、外设管理、部署与通信等功能。为了最大化云计算与边缘计算的应用价值与更好地满足各种需求场景的需求，云计算与边缘计算需要实时或更快速地处理和分析数据、节省网络流量、可离线运行并支持断点续传、给本地数据提供更高安全保护等。
云边端协同集群工作方式：本发明采用跨边云的协同框架，在网络通信良好的情况下，神经网络模型将会被分割，先由云端执行模型前端的计算任务，然后云端将中间结果传输给边缘端，由边缘端执行剩下的检测工作，得出最终结果。此外，云端还会根据边缘端上传的数据来训练模型，并会不定期更新边缘端的模型，而边缘端收集边端设备数据，并使用最新的模型执行检测任务；而在网络通信不好的情况下，边缘端承担任全部检测工作，确保能够实时、快速得出最终结果。
为了实现边缘端的有效训练，需要减少边端与云端的计算开销，本发明通过参数共享和裁剪等方法减少对任务不敏感的模型参数，从而降低模型存储和通信开销、减少网络规模和降低算力需求；同时，本发明还对模型权重进行量化，减少模型在边云的计算。
本发明采用的协同框架依托于Kubernetes的容器编排和调度能力，为云边之间的网络、应用部署和元数据同步提供基础架构支持的开源系统。协同框架结合了云原生的核心优势， 解决了边缘计算场景下的挑战，并支持边缘设备管理、适应复杂的边云网络环境等。
本发明使用的跨边云的协同框架具有以下特点：
(1)提供边云协同深度学习基础框架。支持边云协同数据集管理、模型管理，方便开发者快速开发边云协同智能应用；
(2)提供边云协同训练基础模块和丰富的推理框架。包括：A)联合推理：针对边缘端资源需求大，或边侧资源受限，将推理任务迁移到云端，以提升系统整体的推理性能。B)增量训练：针对小样本和边缘数据异构的问题，模型可以在云端或边缘进行跨时间自适应优化。C)联邦学习：针对数据大，原始数据不出边缘端，隐私要求高等场景，模型在边缘训练，参数在云端聚合，能够有效解决数据孤岛的问题。D)终身学习：针对小样本和边缘数据异构的问题。云边端协同集群通过云端知识库提供记忆功能，让边缘积累的样本知识持续更新，持久化，从而处理灾难性遗忘问题；结合增量训练和多任务训练，同时实现跨时间与跨情景的知识迁移，从而更好地处理未知任务。
算法模型训练和工程化
本发明完成六种智能算法的训练以及工程化部署。六种智能算法包括：婴儿哭声识别、婴儿身高测量、婴儿心率与呼吸频率测量、婴儿骨架识别、婴儿口鼻遮掩检测、婴儿体温检测。并基于高效部署框架进行推理加速。
婴儿哭声识别：哭声是婴儿与外界交流的主要途径，婴儿通过哭声向外界传达需求，监护人则需根据婴儿哭声来满足其需求。
S101构建数据集：本发明的训练数据集包括四个类别的场景音频：婴儿哭声、安静场景、嘈杂场景和婴儿笑声，其中婴儿哭声的音频来源于ESC-50数据集，该数据集中的音频片段是从Freesound.org项目收集的公共场合音频数据中人工提取的。ESC-50数据集是一个用于声音分类的数据集，由2000个带标签的音频构成，每个音频约5秒长，可分为50个类别。
S102提取特征：首先对输入音频信号进行预处理，讲训练集的音频切分为5秒长的片段。然后针对婴儿哭声的特点，对输入音频信号提取了梅尔频率倒谱系数、频谱滚降点、频谱质心、过零率、频谱频宽、均方根能量等特征。
S103模型训练与工程化：设计基于支持向量机的检测算法，并使用s-1提出的数据集进行训练，所设计算法原理简单、易于实现和计算速度快，比较适合实时处理，并且其性能可观，在哭声识别数据集上达到97.22％的准确率。
进一步作为可选的实施方式，婴儿哭声识别的具体步骤如下：
S1：从麦克风设备驱动程序读取音频片段。
S1-1：调用Linux操作系统的ALSA驱动接口，初始化麦克风设备，设置采样率等采集参数。
ALSA全称高级Linux声音架构(Advanced Linux Sound Architecture)，是Linux操作系统中当前主流的音频体系架构，可用于录音和播放音频等常见音频操作。市面上常见的声卡设备均有ALSA的支持，也是目前大多数应用程序与音频硬件交互所使用的方式。
本实施例通过ALSA官方的用户空间alsa-lib函数库使用了ALSA的录音功能。在使用时，需先以指定参数初始化麦克风设备，其中硬件参数主要包括：数据格式为小端序16位整数，通道数为1，采样率为48000Hz并允许按需选择近似的其他采样率，禁用自动重采样；软件参数主要包括：开启时间戳，时间戳类型为GETTIMEOFDAY。最后调用snd_pcm_prepare函数完成初始化过程。采样率表示麦克风的硬件在将声音的机械波的模拟信号转换成数字信号的过程中每秒完成信号采样的次数。
S1-2：从ALSA获取声卡硬件时间戳。snd_pcm_htimestamp函数可获得当前硬件队列中的音频样本数量，以及在同一时刻硬件记录的时间戳。通过采样率可换算得到硬件队列中尚未处理的首个样本的时间戳。
S1-3：从ALSA读取音频波形数据片段，即声卡对声音的机械波进行采样的原始样本。snd_pcm_readi函数可读取硬件队列中指定数量的样本，若样本数量不足则等待。本实现每次读取1024个样本，按采样率换算约20ms的音频波形数据。通过调用snd_pcm_recover以忽略软件运行速度不足导致的硬件队列溢出等异常情况。后续处理流程可根据时间戳判断是否有数据丢失。
S1-4：通过消息队列(如ZeroMQ)，将波形数据和时间戳、采样率、通道数等元数据打包发送至后续处理进程。回到S1-2不断循环。
S2：缓冲音频片段，并根据负载管理缓冲内容。
S2-1：从消息队列读取消息。
S2-2：对比当前读取到的消息的时间戳和当前时间，若时间相差超过预设阈值，则说明目前检测延迟过高，丢弃所有时间戳在n秒前的未处理的数据包。n为预先指定的检测窗口长度，较长的窗口可提升检测的准确度，但也会提升算法的运算量和延迟。在本实现中设定n为5秒。
S2-3：根据时间戳判断当前音频片段是否和之前缓冲的内容连续，若不连续则清空缓冲和检测结果队列。该不连续可能是由于上一步主动丢弃数据包，也可能是由于麦克风硬件的缓冲队列或消息队列溢出导致的。
S2-4：将当前消息中的音频波形数据加入缓冲。
S2-5：当缓冲中的音频波形数据达到或超过n秒后，将这些数据拼接，并送入啼哭检测。
S3：多特征啼哭检测。
S3-1：在系统部署之前，收集包含婴儿哭声的音频和随机的环境声音组成数据集。将这些音频分段、重采样，然后提取梅尔频率倒谱系数、频谱滚降点、频谱质心、过零率、频谱频宽、均方根能量等特征，以这些特征拟合一个支持向量机。
1)过零率表示波形跨过零点的次数，占总采样次数的比例，以2048个样本为窗口大小，512个样本为步长的滑动窗口计算。
2)均方根能量表示每个采样点处，震动的幅值的平方的均值的平方根，以2048个样本为窗口大小，512个样本为步长的滑动窗口计算。
3)梅尔频率倒谱系数(MFCC)表示对数梅尔功率谱经过DCT后的前13个系数。
4)频谱质心定义为C＝∑kSkfk/∑kSk，其中S表示当前窗口的频谱，f表示频谱各数据点对应的频率，以Hz为单位，频谱质心以2048个样本为窗口大小，512个样本为步长的滑动窗口计算。
5)频谱滚降点表示90％的能量分布于该频率及以下，以2048个样本为窗口大小，512个样本为步长的滑动窗口计算。
6)频谱频宽表示二阶频谱频宽，定义为以2048个样本为窗口大小，512个样本为步长的滑动窗口计算。
S3-2：运行时，接收S2输出的n秒音频片段，同样进行重采样(如果采样率与训练时不同的话)，特征提取，并使用上一步拟合的支持向量机进行分类。
本实施例的识别方法在梅尔频谱等的基础上，进一步提取了更为鲁棒的特征，较简单电路或仅基于梅尔频谱的方法鲁棒性更强，更能适应复杂的实际应用环境。而和基于深度学习的方法相比，在训练和部署上较为简单，且算力要求低，有助于降低应用成本。
若判定婴儿正在啼哭，则触发预定义的报警动作。可选地包括：在终端上显示报警消息；连接到远程服务器，向监护人的智能手机发送推送消息；播放音乐以自动安抚婴儿等。
婴儿骨架识别：婴儿骨架识别主要是为其他智能婴儿监控功能提供支撑，如危险动作识别、婴儿身高测量等。采用现有的轻量级的骨架识别网络以降低在低功耗设备中的推理代价。这个轻量级的骨架识别网络优化了OpenPose方法，能够在边缘设备上实时推断，且精度下降可以忽略不计。输入单帧图片，该检测模型能够检测人体18个关键点：耳朵、眼睛、鼻子、脖子、肩膀、肘部、手腕、臀部、膝盖和脚踝。其在COCO 2017中的骨架识别任务中平均精 准度可以达到40％。
婴儿身高测量：婴幼儿阶段是儿童成长第一高峰期，身高测量可以有效地反映婴儿成长发育情况，是衡量健康的一项重要标准。
S201通过深度相机采集婴儿图片及其对应的深度图。本发明采用的红外双目摄像头能够实时采集婴儿高清晰度的可见光图像，并且该摄像头自带芯片计算出图像对应的深度图，有效减轻边缘端设备的计算负担。
S202使用骨架提取算法提取骨架关键点。本发明采用上述的婴儿骨架识别算法，得到婴儿的头顶，膝盖，脚踝等关键点坐标。并根据关键点坐标将深度图对应位置的深度信息转换成三维点云，即为该关键点在以相机光心为原点的三维坐标系中的XYZ坐标。
S203根据三维点云坐标估算婴儿身高。根据s-2测量的关键点三维点云坐标，可计算下肢长度、躯干长度和头的长度。婴儿身高可以表示为下肢长度、躯干长度和头的长度的总和。为了减少测量误差，对左右下肢的长度取平均得到最终的下肢长度。该算法有效解决婴儿姿势古怪、不配合的问题。
进一步作为可选的实施方式，婴儿身高测量的具体步骤如下：
S1：获取深度图像，以及获取婴儿关节点信息。
S1-1：通过深度双目相机实时获取婴儿RGB图像以及对应的深度图像，该深度双目相机可采用目前市面上任意一种常见型号，此处不作额外限制。
S1-2：利用骨架关节点提取算法(如OpenPose算法)对婴儿RGB图像进行实时检测和处理，提取婴儿的骨架关节点坐标，该骨架关节点提取算法可采用目前任意一种已有的成熟算法，此处不作额外限制。选择所有骨架关节点中部分关键骨架关节点，如图2所示，包括婴儿的头顶H、颈椎N、盆骨P、左右膝盖KL,KR以及左右脚后跟FL,FR共计7个关键骨架关节点，分别记录其对应的像素坐标，记为(u,v)，并记录深度图像中相应像素坐标位置的深度值，记为Z。
S2：生成关键点三维点云数据。
S2-1：通过读取相机厂商提供的硬件信息，得到相机传感器的内部参数，包括相机的像素分辨率(w,h)，像素尺寸(d)以及镜头焦距(f)。若相机厂商未提供内部参数信息，则需要通过相机标定方法得到上述参数。目前标定方法已经十分成熟，此处不再赘述。通常相机出厂后其传感器内部参数已固定，在使用过程中一般不会发生变化，若需要重新进行标定，可采用目前任意一种已有的成熟标定方法，此处不作限制；
S2-2：目前通常可采用针孔模型对相机将空间三维坐标映射到平面二维坐标的过程进行 有效描述，由此可根据相机成像的光学原理得到，关键骨架关节点的像素坐标(u,v)及其对应的空间三维坐标(X,Y,Z)满足如下关系：
其中，u0以及v0分别表示RGB图像的中心像素坐标和原点像素坐标之间横向和纵向的像素差值，即
对上式进行转换和化简可得：
由此，根据已知的相机传感器的内部参数，对于婴儿的每一个关键骨架关节点，将其像素坐标(u,v)和在深度图像中相应像素坐标位置的深度值Z代入上式，即可得到该关键骨架关节点在以相机光心为原点的坐标系下的空间三维坐标(X,Y,Z)。
S3：计算各关节点间距离。
S3-1：婴儿身高可表示为头部长度、躯干长度和下肢长度的总和，其中，头部长度可表示为头顶(H)至颈椎(N)的距离，躯干长度可表示为颈椎(N)至盆骨(P)的距离，下肢长度可表示为盆骨(P)至膝盖(K)以及膝盖(K)至脚后跟(F)的距离之和。
参见图2，对于空间中任意两点A、B，已知其空间三维坐标分别为(XA,YA,ZA)以及(XB,YB,ZB)，则计算AB两点连线长度可表示为：
由此，可根据婴儿关键骨架关节点的空间三维坐标分别计算婴儿的头部、躯干以及下肢长度，最终身高计算公式可表示为：
height＝LH-N+LN-P+LP-K+LK-F
其中，为了减少测量误差，对左右下肢的长度取平均得到最终的下肢长度，即：

婴儿心率与呼吸频率测量：新生儿出生后心脏传导系统的结构和功能仍处于发育阶段，心律失常发生率较高，对其进行心率监控可以及早发现该类疾病。本发明的婴儿心率与呼吸频率测量均是采用视觉信息来监测，无需引入额外的边缘端设备。
婴儿心率测量：面部血管尤其是额头区域的颜色会随着心跳而产生微小的颜色变化，婴儿心率测量算法首先提取人脸特定区域像素值，然后通过一系列信号处理和降噪后，最后测定变化频率从而得出心率。
婴儿呼吸频率测量：呼吸频率则是利用胸部的起伏频率，算法首先对视频中的每一帧构造一个拉普拉斯-高斯图像金字塔，然后对每个像素在每个尺度上进行傅里叶变换，从而实现欧拉放大，将细微动作放大，并且记录运动区域，随后测定该区域周期运动频率，由于信号的频率较低，为了可靠地检测峰值，使用低通滤波创建更平滑的信号，从而得出呼吸频率。
婴儿口鼻遮掩检测：在睡眠过程中，婴儿可能因为睡姿不良导致口鼻被被褥枕头等遮掩从而造成巨大的安全隐患，婴儿照料者应采取措施保证婴儿生命安全。由于口罩识别和耳鼻遮掩两个任务的人脸关键点检测基本一致，本发明考虑采用口罩检测数据集作为训练数据集的补充，其包括近8000张(口鼻遮掩)人脸标注图片，图片来自于WIDER Face和MAFA数据集。在训练过程中，采用轻量化的检测模型，以减少边缘端计算量，并且其平均精准度能够达到89.6％，能够满足真实场景的大部分需求。
婴儿体温检测：婴儿体温是判断婴儿是否健康的一项重要指标，实时的体温检测可以为监护人及时发现婴儿身体状况的异常。本发明采用热传感器能够实时检测到被测对象温度，首先检测被测对象的人脸，然后取额头固定区域的体温取35-42度的像素点的最大值作为最终检测的结果，该检测方法具有推理速度快，无需接触婴儿的优点。
总的来说，现有的智能婴儿床产品因其算法简单、计算消耗低，能够在边缘处理数据，但难以满足用户对智能看护的需求。与现有发明方案不同，为部署与使用多种深度学习算法，本发明方案采用云边端协同集群，当网络条件良好，边缘端负责搜集数据并上传至云端，同时神经网络模型将会被分割，云端承担模型前端的计算任务，然后将中间结果传输给边缘端，边缘端继续执行预测工作，得出最终结果。云端还会根据边缘上传的数据来设计、训练智能模型并且不定期升级模型以供边缘端使用；当网络条件不佳，边缘端承担模型推理任务。本方案能够在边缘与云端之间进行协同，合理优化任务分配策略，拆解智能算法，利用云端强大的计算能力承担公共的计算任务，减轻边缘的计算压力，满足智能算法的算力需求。
综上所述，本实施例相对于现有技术，具有如下优点及有益效果：
(1)支持云边端协同体系，减轻边缘端计算压力。能够在边缘与云端之间进行协同，合理优化任务分配策略，拆解智能算法，利用云端强大的计算能力承担公共的计算任务，减轻边缘的计算压力，满足智能算法、算力需求。
(2)支持多种边缘端设备协同，提供全方面看护能力。本发明方案利用物联网设备多功 能、空间分布多样化的优势，对任务进行协同处理，增强系统的整体能力。本发明方案通过边缘与云端之间协同，增强对外服务能力，提升服务的多样性。
(3)支持多种深度智能算法，提供智能化看护服务。本发明方案支持婴儿哭声识别、婴儿身高测量等多种智能算法，并通过TensorRT等高性能软件框架为深度学习应用提供低延迟、高吞吐率的部署推理，以满足边缘端实时推理的需求。
在本说明书的上述描述中，参考术语“一个实施方式/实施例”、“另一实施方式/实施例”或“某些实施方式/实施例”等的描述意指结合实施方式或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。
尽管已经示出和描述了本发明的实施方式，本领域的普通技术人员可以理解：在不脱离本发明的原理和宗旨的情况下可以对这些实施方式进行多种变化、修改、替换和变型，本发明的范围由权利要求及其等同物限定。
以上是对本发明的较佳实施进行了具体说明，但本发明并不限于上述实施例，熟悉本领域的技术人员在不违背本发明精神的前提下还可做作出种种的等同变形或替换，这些等同的变形或替换均包含在本申请权利要求所限定的范围内。

Claims (10)

  1. 一种云边端协同智能婴儿护理系统，其特征在于，包括：
    传感器模块，用于采集婴儿床上的图像信息和声音信息；
    边缘设备，用于根据采集的图像信息和声音信息进行哭声识别以及体征识别，其中体征识别包括心率与呼吸频率测量、口鼻遮掩检测、身高测量、骨架识别或体温检测中的至少一种；
    云端设备，用于对检测模型进行训练以及更新，以及根据识别结果输出提示信息；
    其中，采用跨边云的协同框架来实现边缘端与云端设备之间的应用管理、外设管理、部署以及通信。
  2. 根据权利要求1所述的一种云边端协同智能婴儿护理系统，其特征在于，所述哭声识别和体征识别采用神经网络模型来实现；边缘设备和云端设备通过参数共享和裁剪的方式，减少对任务不敏感的模型参数，从而降低模型存储和通信开销、减少网络规模和降低算力需求；
    在网络通信良好的情况下，神经网络模型将会被分割，先由云端设备执行模型前端的计算任务，然后云端设备将中间结果传输给边缘设备，由边缘端执行剩下的检测工作，得出最终结果；
    在网络通信不好的情况下，边缘端承担任全部的检测工作，以确保能够实时、快速地获得最终结果；
    此外，云端设备还会根据边缘设备上传的数据来训练模型，并更新边缘设备上的模型；边缘设备根据传感器采集到的信息，使用最新的模型执行检测任务。
  3. 根据权利要求1所述的一种云边端协同智能婴儿护理系统，其特征在于，所述传感器模块包括红外摄像头、RGB摄像头、热传感器和麦克风。
  4. 根据权利要求1所述的一种云边端协同智能婴儿护理系统，其特征在于，根据声音信息进行哭声识别的具体步骤如下：
    对声音信息进行预处理，获取音频信号；
    对音频信息进行重采样以及提取特征，根据提取的特征和预设的支持向量机进行分类识别，获得识别结果；所述识别结果包括婴儿哭声、安静场景、嘈杂场景和婴儿笑声；
    所述支持向量机通过以下方式获得；
    获取数据集，对数据集进行重采样，并进行特征提取；其中，提取的特征包括梅尔频率倒谱系数、频谱滚降点、频谱质心、过零率、频谱频宽、均方根能量；
    根据提取的特征拟合获得支持向量机。
  5. 根据权利要求4所述的一种云边端协同智能婴儿护理系统，其特征在于，所述频谱质心定义如下：
    C＝∑kSkfk/∑kSk
    频谱频宽为二阶频谱频宽，定义如下：
    其中，Sk表示第k个窗口的频谱，fk表示频谱第k个数据点对应的频率。
  6. 根据权利要求1所述的一种云边端协同智能婴儿护理系统，其特征在于，云端设备是一台Linux操作系统的服务器，边缘端设备是一台嵌入式设备；
    通过以下方式采集获取声音信息：
    调用Linux操作系统的ALSA驱动接口，初始化麦克风，设置采集参数；
    从ALSA获取声卡硬件时间戳，以及读取音频的波形数据；
    通过消息队列将波形数据和时间戳、采样率进行打包，作为声音信息并发送至后续处理进程。
  7. 根据权利要求6所述的一种云边端协同智能婴儿护理系统，其特征在于，在进行哭声识别前，对声音信息进行以下处理：
    从消息队列中读取消息；
    对比当前读取到的消息的时间戳和当前时间，若时间相差超过预设阈值，判定目前检测延迟过高，丢弃所有时间戳在n秒前的未处理的数据包；其中，n为预设的检测窗口长度；
    根据时间戳判断当前音频片段是否和之前缓冲的内容连续，若不连续则清空缓冲和检测结果队列；
    将当前消息中的音频波形数据加入缓冲；
    当缓冲中的音频波形数据达到或超过n秒后，将这些数据进行拼接，并送入模型中进行哭声识别。
  8. 根据权利要求1所述的一种云边端协同智能婴儿护理系统，其特征在于，根据图像信息进行身高测量的具体步骤包括：
    获取深度图像，根据深度图像获取婴儿的关节点信息；其中，关节点信息包括婴儿的头顶、颈椎、盆骨、左右膝盖以及左右脚后跟共7个关键骨架关节点的像素坐标；
    根据相机的参数，获取关键骨架关节点在以相机光心为原点的坐标系下的空间三维坐标；
    根据三维坐标计算各关节点间距离，根据各关节点间距离计算获得婴儿的身高。
  9. 根据权利要求8所述的一种云边端协同智能婴儿护理系统，其特征在于，最终身高的计算公式如下：
    Height＝LH-N+LN-P+LP-K+LK-F
    为了减少测量误差，对左右下肢的长度取平均得到最终的下肢长度，即：

    其中，LH-N为头顶节点到颈椎节点之间的距离，LN-P为颈椎节点到盆骨节点之间的距离，LP-K为盆骨节点到膝盖节点之间的距离，LK-F为膝盖节点到脚后跟节点之间的距离。
  10. 如权利要求1-9任一项所述的一种云边端协同智能婴儿护理系统的搭建方法，其特征在于，包括以下步骤：
    搭建与配置硬件设备，其中，云端设备是一台Linux操作系统的服务器，边缘端设备是一台装有一系列传感器的嵌入式设备；
    搭建边缘设备和云端设备之间的协同集群，根据基于跨边云的协同框架实现边缘设备和云端设备之间的应用管理与部署；
    在边缘设备和云端设备上部署模型，以及训练模型。
PCT/CN2023/130232 2022-11-08 2023-11-07 一种云边端协同智能婴儿护理系统及方法 Ceased WO2024099313A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211398455.4 2022-11-08
CN202211398455.4A CN115715671A (zh) 2022-11-08 2022-11-08 一种云边端协同智能婴儿护理系统及方法

Publications (1)

Publication Number Publication Date
WO2024099313A1 true WO2024099313A1 (zh) 2024-05-16

Family

ID=85255430

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/130232 Ceased WO2024099313A1 (zh) 2022-11-08 2023-11-07 一种云边端协同智能婴儿护理系统及方法

Country Status (2)

Country Link
CN (1) CN115715671A (zh)
WO (1) WO2024099313A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118466238A (zh) * 2024-05-31 2024-08-09 浙江捷特佳智能科技有限公司 一种智能家居系统
CN118746940A (zh) * 2024-06-17 2024-10-08 振宁(无锡)智能科技有限公司 一种基于工业互联网的跨平台数据协同优化系统
CN119655879A (zh) * 2025-02-07 2025-03-21 宁夏医科大学总医院 基于人工智能的甲状腺结节外科手术规划与导航系统
CN119808890A (zh) * 2024-12-12 2025-04-11 广州准捷电子科技有限公司 一种采用分布式能源的ai集群训练方法以及相关装置
CN119893167A (zh) * 2025-03-27 2025-04-25 浙江省公安科技研究所 一种云边协同的智能视频处理方法
CN120723779A (zh) * 2025-06-16 2025-09-30 苏州辰瓴信息技术有限公司 基于云升级的ai质检系统
CN120939405A (zh) * 2025-10-17 2025-11-14 深圳市医友天下科技有限公司 安抚系统的控制方法、安抚设备和存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115662476B (zh) * 2022-11-08 2025-08-26 华南理工大学 一种婴儿声音检测方法、装置和存储介质
CN115715671A (zh) * 2022-11-08 2023-02-28 华南理工大学 一种云边端协同智能婴儿护理系统及方法
CN117608591A (zh) * 2023-10-23 2024-02-27 华南理工大学 一种云边端协同大模型轻量化部署平台系统及方法
CN117503090A (zh) * 2023-11-09 2024-02-06 长春理工大学 一种边缘智能婴幼儿监护方法
CN118587660B (zh) * 2024-08-07 2025-02-07 嘉兴元贝贝科技有限公司 一种基于机器学习的婴幼儿行为动作监护方法及系统
CN119583387B (zh) * 2024-12-03 2025-10-17 四川大学 基于预训练语言模型的云边端时间序列异常检测系统及方法
CN120582342B (zh) * 2025-06-11 2026-02-13 赣州康晋电气设备有限公司 一种自动测控的智能箱变控制方法及系统

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109008440A (zh) * 2018-08-31 2018-12-18 钟祥博谦信息科技有限公司 智能婴儿床系统及其控制方法
CN109982046A (zh) * 2019-04-04 2019-07-05 成都易通酷物联网科技有限公司 一种便于婴幼儿护理的云自动摄像监控系统及方法
CN114172809A (zh) * 2021-12-13 2022-03-11 重庆邮电大学 一种基于目标追踪的视频计算云边协同任务调度方法
CN114528987A (zh) * 2022-02-15 2022-05-24 中国科学院上海微系统与信息技术研究所 一种神经网络边缘-云协同计算分割部署方法
CN114564312A (zh) * 2022-03-03 2022-05-31 北京航空航天大学云南创新研究院 一种基于自适应深度神经网络的云边端协同计算方法
KR20220092761A (ko) * 2020-12-24 2022-07-04 (주)끌렘컴퍼니 인공지능 기반 아기 수면 모니터링 방법 및 시스템
CN115137324A (zh) * 2022-05-20 2022-10-04 西安交通大学 一种基于云-边-端架构的人体心脏健康监护系统
CN115715671A (zh) * 2022-11-08 2023-02-28 华南理工大学 一种云边端协同智能婴儿护理系统及方法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103854646B (zh) * 2014-03-27 2018-01-30 成都康赛信息技术有限公司 一种实现数字音频自动分类的方法
CN107192342A (zh) * 2017-05-11 2017-09-22 广州帕克西软件开发有限公司 一种非接触式体型数据的测量方法及系统
CN108540855B (zh) * 2018-04-18 2021-09-28 王健 一种适用于网络直播场景下的自适应低延时流媒体播放方法
CN110909687B (zh) * 2019-11-26 2022-10-21 爱菲力斯(深圳)科技有限公司 动作特征有效性判定方法、计算机存储介质和电子装置
CN112073823B (zh) * 2020-09-02 2022-07-12 深圳创维数字技术有限公司 丢帧处理方法、视频播放终端及计算机可读存储介质
CN114926957B (zh) * 2022-04-13 2024-01-19 西安理工大学 基于智能家居的婴幼儿监护系统及监护方法
CN114841952B (zh) * 2022-04-28 2024-05-03 华南理工大学 一种云边协同的早产儿视网膜病变检测系统及检测方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109008440A (zh) * 2018-08-31 2018-12-18 钟祥博谦信息科技有限公司 智能婴儿床系统及其控制方法
CN109982046A (zh) * 2019-04-04 2019-07-05 成都易通酷物联网科技有限公司 一种便于婴幼儿护理的云自动摄像监控系统及方法
KR20220092761A (ko) * 2020-12-24 2022-07-04 (주)끌렘컴퍼니 인공지능 기반 아기 수면 모니터링 방법 및 시스템
CN114172809A (zh) * 2021-12-13 2022-03-11 重庆邮电大学 一种基于目标追踪的视频计算云边协同任务调度方法
CN114528987A (zh) * 2022-02-15 2022-05-24 中国科学院上海微系统与信息技术研究所 一种神经网络边缘-云协同计算分割部署方法
CN114564312A (zh) * 2022-03-03 2022-05-31 北京航空航天大学云南创新研究院 一种基于自适应深度神经网络的云边端协同计算方法
CN115137324A (zh) * 2022-05-20 2022-10-04 西安交通大学 一种基于云-边-端架构的人体心脏健康监护系统
CN115715671A (zh) * 2022-11-08 2023-02-28 华南理工大学 一种云边端协同智能婴儿护理系统及方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118466238A (zh) * 2024-05-31 2024-08-09 浙江捷特佳智能科技有限公司 一种智能家居系统
CN118746940A (zh) * 2024-06-17 2024-10-08 振宁(无锡)智能科技有限公司 一种基于工业互联网的跨平台数据协同优化系统
CN119808890A (zh) * 2024-12-12 2025-04-11 广州准捷电子科技有限公司 一种采用分布式能源的ai集群训练方法以及相关装置
CN119655879A (zh) * 2025-02-07 2025-03-21 宁夏医科大学总医院 基于人工智能的甲状腺结节外科手术规划与导航系统
CN119893167A (zh) * 2025-03-27 2025-04-25 浙江省公安科技研究所 一种云边协同的智能视频处理方法
CN120723779A (zh) * 2025-06-16 2025-09-30 苏州辰瓴信息技术有限公司 基于云升级的ai质检系统
CN120939405A (zh) * 2025-10-17 2025-11-14 深圳市医友天下科技有限公司 安抚系统的控制方法、安抚设备和存储介质

Also Published As

Publication number Publication date
CN115715671A (zh) 2023-02-28

Similar Documents

Publication Publication Date Title
WO2024099313A1 (zh) 一种云边端协同智能婴儿护理系统及方法
Chen et al. Fall detection system based on real-time pose estimation and SVM
CN112861624A (zh) 一种人体姿态检测方法、系统、存储介质、设备及终端
US20190279481A1 (en) Subject detection for remote biometric monitoring
CN117058713B (zh) 基于头戴式毫米波雷达的第一人称姿态估计方法及系统
CN107909023B (zh) 运动参数的识别方法、装置、终端及存储介质
CN105686833A (zh) 基于无线体域网的可穿戴式多节点三维运动监测系统
CN114926957B (zh) 基于智能家居的婴幼儿监护系统及监护方法
CN108187333A (zh) 上肢康复训练中的多惯性节点无线监测与评价系统及方法
CN106843002A (zh) 一种基于智能机器人头部的控制系统
CN109614899A (zh) 一种基于李群特征和卷积神经网络的人体动作识别方法
CN114662606A (zh) 行为识别方法及装置、计算机可读介质和电子设备
CN114067256B (zh) 一种基于Wi-Fi信号的人体关键点检测方法及系统
CN106853638A (zh) 一种基于增强现实的人体生物信号远程控制系统及方法
CN208969808U (zh) 基于人脸检测和声音检测的婴儿监护器
CN115280371A (zh) 图像处理装置和运动图像数据生成方法
CN109547706A (zh) 眼镜设备和系统
CN103371798A (zh) 耳挂式眼控装置
CN110757475A (zh) 一种居家养老服务机器人
Chen et al. Human motion tracking with wireless wearable sensor network: experience and lessons.
CN207218881U (zh) 基于异构嵌入式平台的眼动追踪系统
CN121583006B (zh) 面向康复场景的多视角跨模态三维人体姿态估计方法及系统
Dhole et al. Review of deep learning models for mask detection and medical sensors for iot based health care system
CN210811071U (zh) 一种基于三轴传感器的人体行为识别系统
CN120298731B (zh) 一种基于vr全景的图中实体对象匹配系统及其方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23888002

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23888002

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 23888002

Country of ref document: EP

Kind code of ref document: A1