CN118076292A

CN118076292A - Computer-based system for interacting with infants and method of using same

Info

Publication number: CN118076292A
Application number: CN202280043493.XA
Authority: CN
Inventors: Y·什瓦尔茨曼; A·凯哈特
Original assignee: Nutritech Corp
Current assignee: Nutritech Corp
Priority date: 2021-04-20
Filing date: 2022-04-20
Publication date: 2024-05-24

Abstract

A system, comprising: a memory; an optical subsystem; an audio system; a plurality of sensors that output sensor data; a communication circuit; and a processor. The processor is configured to: inputting image data, audio signal data, sensor data, and baby-specific personal data associated with the baby to a baby-specific behavioral state detection machine learning model; receiving an output of the infant anxiety and/or about to wake up from the infant specific behavioral state detection machine learning model; transmitting instructions based on the output that cause the audio system and/or the optical subsystem to perform at least one of: (i) generating a pacifying sound when the infant is anxious about to wake, (ii) generating a sleep-aiding sound when the infant is about to wake, or (iii) projecting a relaxed image to the infant for viewing by the infant when the infant is anxious about to anxiety.

Description

Computer-based system for interacting with infants and method of using same

RELATED APPLICATIONS

The present application claims the benefit of U.S. provisional application Ser. No. 63/177,171, U.S. provisional application Ser. No. 63/186,869, and U.S. provisional application Ser. No. 63/222,001, filed on Ser. No. 63/177,171, filed on Ser. No. 5/11, 2021, and filed on 15, 2021, both of which are incorporated herein by reference in their entirety for all purposes.

Technical Field

The present disclosure relates to a computer-based system for interacting with an infant and methods of use thereof.

Background

Baby monitoring systems have evolved from simple devices (such as having only a recording system) to complex devices that can provide real-time video feeds of babies. The baby monitor may be connected to the mobile device, and thus, the caretaker may monitor the baby remotely via the mobile device.

Disclosure of Invention

In some embodiments, the present disclosure provides an exemplary technology-improved computer-based system comprising at least the following components:

A nonvolatile memory;

at least one electronic resource, which may include a database;

Wherein the database may comprise:

(i) Infant specific stimulus data for a plurality of infant specific stimuli provided to a plurality of infants,

(Ii) Infant-specific response data of a plurality of infant-specific responses acquired in response to the plurality of infant-specific stimuli provided to the plurality of infants, and

(Iii) Baby-specific personal data for each baby in the plurality of babies;

an optical subsystem, which may include an imaging device, a projection device, or both;

Wherein the optical subsystem may be configured to perform at least one of:

(i) Acquiring image data of an image of at least one infant from the plurality of infants from the imaging device, or

(Ii) Projecting, by the projection device, at least one visual image for viewing by the at least one infant;

an audio system, which may include a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(i) Receiving audio signal data from the at least one infant by the microphone, or

(Ii) Generating at least one sound for the at least one infant by the speaker;

A plurality of sensors that output sensor data;

a communication circuit configurable to communicate with at least one communication device of at least one user associated with the at least one baby over a communication network; and

At least one processor configured to execute code stored in the non-volatile memory, the code causing the at least one processor to:

Receiving the image data, the audio signal data, and the sensor data associated with the at least one infant;

determining infant-specific physiological data of the at least one infant based on the image data, the sensor data, and the audio signal data;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Inputting the image data, the audio signal data, the sensor data, the infant-specific physiological data, and the infant-specific personal data associated with at least one infant to at least one infant-specific behavioral state detection machine learning model;

wherein the at least one infant-specific behavioral state detection machine learning model is trainable using a dataset that is based at least in part on the infant-specific stimulus data, the infant-specific response data, and the infant-specific personal data associated with the plurality of infants;

Receiving output of the at least one infant anxiety disorder, about to wake up, or both from the at least one infant-specific behavioral state-detecting machine learning model;

transmitting the sensor data, the at least one infant anxiety reminder, the at least one infant about to wake up reminder, or any combination thereof, to the at least one communication device of the at least one user over the communication network;

transmitting instructions based on the output that cause the audio system, the optical subsystem, or both to perform at least one of:

(i) Generating a pacifying sound by said speaker when said at least one infant is anxious,

(Ii) Generating a sleep-aiding sound by the speaker when the at least one infant is about to wake up, or

(Iii) When the at least one infant is anxious, a relaxed image is projected by the projection device for viewing by the at least one infant.

A nonvolatile memory;

At least one electronic resource, the at least one electronic resource comprising at least one database;

Wherein the at least one database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

An imaging device configured to acquire image data of an image of at least one infant from the plurality of infants;

a microphone configured to receive audio signal data from the at least one infant;

A plurality of sensors that output sensor data;

A communication circuit configured to communicate with at least one communication device of at least one user associated with the at least one baby over a communication network;

A temperature controller; and

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Wherein the at least one infant-specific behavioral state detection machine learning model is trained using a dataset based at least in part on the infant-specific stimulus data, the infant-specific response data, and the infant-specific personal data associated with the plurality of infants;

Receiving at least one indication of hunger of the at least one infant from the at least one infant-specific behavioral state detection machine learning model;

transmitting a reminder to feed the at least one infant, the sensor data, or both, to the at least one communication device of the at least one user over the communication network; and

Transmitting instructions that cause the temperature controller to change a predefined temperature of at least one food item in preparation for feeding the at least one infant.

A nonvolatile memory;

At least one electronic resource, which may include at least one database;

wherein the at least one database may comprise:

(i) A plurality of infant-specific educational programs for a plurality of infants,

(Ii) Based on infant-specific stimulus data of a plurality of infant-specific stimuli provided to the plurality of infants by the plurality of infant-specific education programs,

(Iii) Infant-specific response data of a plurality of infant-specific responses acquired in response to the plurality of infant-specific stimuli provided to the plurality of infants, and

(Iv) Baby-specific personal data for each baby in the plurality of babies;

wherein the optical subsystem is configured to perform at least one of:

wherein the audio system is configured to perform at least one of:

(Ii) Generating at least one sound for the at least one infant by the speaker;

A plurality of sensors that output sensor data;

a communication circuit configured to communicate with at least one communication device of at least one user associated with the at least one baby over a communication network; and

projecting, by the projection device, the at least one visual image to the at least one infant based on an infant-specific educational plan from the plurality of infant-specific educational plans for the at least one infant;

generating, by the audio system, the at least one sound associated with the at least one visual image;

Receiving the image data, the audio signal data, the sensor data, or any combination thereof associated with the at least one infant;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Inputting the image data, the audio signal data, the sensor data, the at least one visual image, the at least one sound, the infant-specific physiological data, the infant-specific personal data associated with the at least one infant to at least one infant-specific educational machine learning model;

Wherein the at least one infant-specific educational machine learning model is trainable using a dataset that is based at least in part on the infant-specific stimulus data, the infant-specific response data, the infant-specific personal data, the plurality of infant-specific educational programs, or any combination thereof associated with the plurality of infants;

Receiving output from the at least one infant-specific educational machine learning model;

Wherein the outputting may include:

(i) The at least one infant understands or does not understand at least one indication of the at least one visual image and the at least one sound associated with the at least one visual image according to the at least one infant-specific educational plan for the at least one infant;

(ii) At least one infant-specific educational recommendation based at least in part on the at least one indication;

Transmitting the at least one indication, the at least one baby-specific educational recommendation, the sensor data, or any combination thereof, to the at least one communication device of the at least one user over the communication network; and

Performing at least one of the following based on the at least one infant-specific educational recommendation:

(i) Modifying the at least one infant-specific education program when the at least one indication indicates that the at least one infant is not understood, or

(Ii) Continuing to execute the infant-specific educational program for the at least one infant.

Drawings

Various embodiments of the present disclosure may be further explained with reference to the appended figures, wherein like structure is referred to by like numerals throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the disclosure. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the one or more illustrative embodiments.

FIG. 1 illustrates a first exemplary embodiment of a computer-based system for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure;

FIG. 2A is a first exemplary screen shot of a graphical user interface of a mobile device in a system configured to pacify an infant in accordance with one or more embodiments of the present disclosure;

FIG. 2B is a second exemplary screen shot of a graphical user interface of a mobile device in a system configured to prepare food products for feeding an infant in accordance with one or more embodiments of the present disclosure;

FIG. 2C is a third exemplary screen shot of a graphical user interface of a mobile device reminding a user to feed an infant in accordance with one or more embodiments of the present disclosure;

FIG. 2D is a fourth exemplary screen shot of a graphical user interface of a mobile device in a system for educating an infant by projecting images in accordance with one or more embodiments of the present disclosure;

Fig. 3 is a table of input data sources and output peripherals/IoT devices used by the system to implement the following three use cases in accordance with one or more embodiments of the present disclosure: feeding cases, pacifying cases and education cases;

FIG. 4A is a first exemplary list of input-output data features in a dataset for training an infant-specific behavioral state detection machine learning model and/or an infant-specific educational machine learning model, according to one or more embodiments of the present disclosure;

FIG. 4B is a second exemplary list of input-output data features in a dataset for training a baby-specific behavioral state detection machine learning model, according to one or more embodiments of the present disclosure;

FIG. 5A is an algorithm flow diagram of a computer-based system for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure;

FIG. 5B illustrates an exemplary width and depth neural network model for modeling a smile rate of an infant in accordance with one or more embodiments of the present disclosure;

FIG. 6A is a flow diagram of a method for pacifying an infant in accordance with one or more embodiments of the present disclosure;

Fig. 6B is a flow diagram of a method for feeding an infant in accordance with one or more embodiments of the present disclosure;

FIG. 6C is a flow diagram of a method for educating an infant in accordance with one or more embodiments of the present disclosure;

FIG. 7A illustrates a second exemplary embodiment of a computer-based system for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure;

FIG. 7B illustrates a diagram of data elements of an Infant Condition Description Vector (ICDV) in accordance with one or more embodiments of the present disclosure;

FIGS. 7C and 7D illustrate exemplary embodiments of a detection reaction loop for pacifying an infant in accordance with one or more embodiments of the present disclosure;

FIG. 7E schematically illustrates an embodiment of an audio generator response agent of a reactive pacifying loop according to one or more embodiments of the present disclosure;

FIG. 8A illustrates a second embodiment of an infant monitoring and interaction (BMID) device according to one or more embodiments of the present disclosure;

FIG. 8B illustrates a third embodiment of an infant monitoring and interaction (BMID) device according to one or more embodiments of the present disclosure;

fig. 9A is a flow diagram of a method for educating an infant in accordance with one or more embodiments of the present disclosure.

FIG. 9B is a flow diagram of a method for testing an infant to assess infant learning according to one or more embodiments of the present disclosure;

FIG. 10A is a diagram illustrating a heart rate estimation system in accordance with one or more embodiments of the present disclosure;

FIG. 10B is a diagram illustrating an object of interest (OOI) and a region of interest (ROI) on a subject in accordance with one or more embodiments of the present disclosure;

FIG. 10C is a schematic block diagram illustrating video data according to one or more embodiments of the present disclosure;

FIG. 10D is a schematic block diagram illustrating a temporal sequence of superpixels according to one or more embodiments of the present disclosure;

FIG. 11A is a schematic block diagram illustrating video data in accordance with one or more embodiments of the present disclosure;

FIG. 11B is a schematic block diagram illustrating data according to one or more embodiments of the present disclosure;

FIG. 11C is a schematic block diagram showing ROI data according to one or more embodiments of the present disclosure;

FIG. 11D is a schematic block diagram illustrating superpixel data in accordance with one or more embodiments of the present disclosure;

FIG. 11E is a schematic block diagram illustrating a superpixel model in accordance with one or more embodiments of the present disclosure;

FIG. 12 is a flow diagram illustrating a heart rate estimation process according to one or more embodiments of the present disclosure;

FIG. 13 is a schematic block diagram illustrating a computer according to one or more embodiments of the present disclosure;

FIG. 14 is a flow diagram of a method of cardiac property estimation according to one or more embodiments of the present disclosure;

Fig. 15A is a schematic block diagram of a respiratory event recognition system according to one or more embodiments of the present disclosure;

fig. 15B is a schematic block diagram of a respiration report according to one or more embodiments of the present disclosure;

FIG. 15C is a schematic block diagram of a motion report according to one or more embodiments of the present disclosure;

Fig. 15D is a schematic block diagram of respiratory data according to one or more embodiments of the present disclosure;

FIG. 16A is a schematic diagram illustrating regions in an image frame in accordance with one or more embodiments of the present disclosure;

FIG. 16B is a schematic diagram illustrating regions in an image frame in accordance with one or more embodiments of the present disclosure;

FIG. 16C is a schematic diagram illustrating regions in an image frame in accordance with one or more embodiments of the present disclosure;

FIG. 16D is a schematic diagram illustrating a user selected region in an image frame in accordance with one or more embodiments of the present disclosure;

fig. 16E is a schematic block diagram of a respiratory event identification state according to one or more embodiments of the present disclosure;

fig. 17A is a schematic block diagram of a respiratory event identification process according to one or more embodiments of the present disclosure;

Fig. 17B is a schematic block diagram of a respiratory event identification process according to one or more embodiments of the present disclosure;

FIG. 17C is a schematic diagram of a video stream of a moving object generation process in accordance with one or more embodiments of the present disclosure;

FIG. 18A is a diagram illustrating a heat map according to one or more embodiments of the present disclosure;

FIG. 18B is a diagram illustrating a heat map according to one or more embodiments of the present disclosure;

FIG. 18C is a schematic block diagram of a computer according to one or more embodiments of the present disclosure;

fig. 18D is a schematic block diagram of a neural network according to one or more embodiments of the present disclosure;

fig. 19A is a flow diagram of a respiratory signal estimation method according to one or more embodiments of the present disclosure;

fig. 19B is a flowchart of a respiratory event identification method according to one or more embodiments of the present disclosure; and

Fig. 19C is a flow chart of a respiratory event communication method according to one or more embodiments of the present disclosure.

Detailed Description

Various detailed embodiments of the present disclosure are disclosed herein in connection with the accompanying drawings; however, it will be understood that the disclosed embodiments are merely illustrative. In addition, each of the examples given in connection with the various embodiments of the present disclosure are intended to be illustrative, and not limiting.

The term "infant" hereinafter may refer to the body of the system. The term infant may refer to a newborn, infant or toddler, but the infant may be a child, adult or elderly person in need of care or stimulation, or the infant may be an animal, such as a pet that may become noisy when boring or an animal in a veterinary care facility.

The term "user" hereinafter may refer to a person using the system to monitor and care for an infant and also to provide a stimulus, instruction, or developmental means for the infant.

Embodiments of the present disclosure describe a computer-based system for monitoring and interacting with infants and methods of use thereof. The system may be implemented with artificial intelligence (e.g., a machine learning model) that can collect data and personalize the complete collaborative ecosystem in terms of data collection functionality and feedback, thereby providing a personalized care system. The system may be configured for care of newborns, infants and toddlers. The system may be configured for use with humans and may be configured for use with members of a group of children, adolescents, adults, elderly people, or any combination thereof. In other embodiments, the system may be configured for use with animals, such as laboratory animals, pets, farm animals, zoo animals, animals under veterinary care, and any combination thereof.

Embodiments disclosed herein address the technical problem of monitoring and interacting with an infant without connecting a sensing device to the infant's body. Using output data from any or all of an imaging device (camera), an audio device (sound transducer, such as a microphone), a plurality of sensors applied to a machine learning model trained for a particular use case, the system can determine whether the infant is hungry, anxious, or understands images and/or sounds projected to the infant according to the use case. The machine learning model is trained to output environmental parameters to change the infant's environment to optimize a particular behavior according to use cases. Thus, the infant may be pacified, fed or presented with images and sounds in a particular manner to enhance education to the infant, all without the need for any physical connection of sensing devices, transducers, etc. to the infant.

In some embodiments, the system can monitor and interact with infants for at least three use cases: pacify, feed and aid in the development or education of the infant.

In some embodiments, the user of the system may be a parent or caretaker, healthcare worker, or child care worker. The user may be on the baby's side (e.g., physically close to the baby) or remote from the baby. The system may be used in healthcare facilities, infant care facilities, and other remote use options.

Fig. 1 illustrates a first exemplary embodiment of a computer-based system 5 for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure. The system 5 may include a first embodiment of an infant monitoring and interaction device (BMID) 10 for monitoring and interacting with an infant 15. The infant 15 may lie on a bed or mattress 37 on a crib, which may include a vibration unit 40 for applying a vibration stimulus to the infant 15. In other embodiments, the vibratory unit 40 may be placed in any suitable location that is operably coupled to the infant and/or may be placed in an infant toy (such as a teddy bear). BMID 10 may include a sensor 14 configured to monitor an infant 15 positioned within the sensing region 12. BMID 10 may include a speaker 16 to generate audio signals for listening by the infant 15.BMID 10 may include a projection device 18 to project 22 an image 25 for viewing by an infant.

In some implementations, BMID may include a processor, a plurality of sensors 80, analog-to-digital (a/D) circuitry 75, memory 95, communication circuitry 98, peripheral/IoT (internet of things) control circuitry 70, and peripheral/IoT device 71. peripheral/IoT devices 71 may include, but are not limited to, speakers 16, projector 18, vibration unit 40, lights 27, ioT devices 76, and food temperature controller 35. Any of the peripheral/IoT devices 71 may be located in BMID a 10, outside BMID a located near the baby 15 or remote from the baby 15, but remain communicatively coupled with BMID a via any suitable wired or wireless communication protocol.

In some embodiments, any of the plurality of sensors and/or peripheral/IoT devices 71 (e.g., speakers 16, projector 18, vibration unit 40, lights 27, and/or IoT devices 76) may be located within BMID devices or external to BMID, implemented as a stand-alone unit having any suitable combination of peripheral/IoT devices 71 but controlled by peripheral/IoT control circuitry 70.

In some implementations, the processor 60 may execute software modules that may be stored in the memory 95 that, when executed by the processor 60, cause the processor to perform the functions as described herein. The software modules may include a sensor processing manager 62, a three-dimensional (3D) graph generator 61, a peripheral/IoT control manager module 63, a GUI manager 64, and/or a Machine Learning Model (MLM)/algorithm module 65. The MLM/algorithm module 65 may include any of the following algorithm blocks, which may be based on machine learning: physical algorithms 66, time-driven pipelines 67, behavioral models 68, and/or feedback environment generators 69.

In some embodiments, sensor processing manager 62 may be used to process digital output signal data from a/D circuit 75, which may receive analog signal inputs from any of a plurality of sensors 80. In other implementations, any of the plurality of sensors 80 may include a/D functionality that directly outputs digital sensor data.

The communication circuitry 98 may be configured to enable BMID to communicate 31 over the communication network 30 and/or to communicate 31 via any suitable wired and/or wireless communication protocol, such as bluetooth and/or wireless fidelity (WiFi), for example, with any of the peripheral/IoT devices (e.g., the speaker 16, projector 18, vibration unit 40, lighting lamp 27, and/or IoT device 76) and/or the mobile device 50 associated with the user 45, such as a caretaker and/or parent of the infant 15. For example, the projector 18 may be located in a first unit and the imaging device of the infant 15 may be located in a second unit separate from the first unit.

In some embodiments, mobile device 50 may display a Graphical User Interface (GUI) 55 to user 45 that may be remotely controlled by GUI manager 64 and/or send a reminder to user 40 via mobile device GUI 55 of mobile device 50. BMID 10 may be configured to evaluate the distance D52 between the infant 15 and the user 45 to determine whether the user 45 is near the infant 15 or far from the infant 15 based on the distance D.

In some embodiments, BMID may include any suitable sound transducer (such as microphone 82) and/or imaging device (such as imaging camera 81) for generating still image data and/or video data (e.g., video data having image frames in a predefined timeline) for infant 15. Other sensors of the plurality of sensors 80 may include a thermal device 83, such as an Infrared (IR) camera, a laser radar device 84, and/or a Radio Frequency (RF) device 85. Output data from the imaging camera 81, thermal imager 83, lidar device 84, and/or RF device 85 may be used by the 3D map generation module 61 to generate a time-domain three-dimensional (3D) map of the infant 15 within the sensing region 12. The time-domain three-dimensional (3D) map of the infant 15 may be used by the MLM/algorithm 65 to detect the position and movement of the infant 15 over any time interval.

For example, the system 5 may include peripheral devices and IoT devices such as speakers 16, projector 18, vibration unit 40, illumination lamp 27, and/or IoT devices (such as food temperature controller 35) to maintain the temperature of the baby bottle 32. The peripheral devices and IoT devices may be controlled by peripheral/IoT control circuitry 70.

The food temperature controller 35 may be controlled by BMID a 10 to control the temperature of the baby bottle 32 or any suitable baby food placed in the food temperature controller 35. For example, if the food product is infant milk, the temperature of the infant milk may be stored at a predefined storage temperature of, for example, 4 ℃. When BMID detects infant hunger, the system 5 may transmit instructions to the food temperature controller to change the predefined temperature from the predefined storage temperature to a predefined feeding temperature of, for example, 37 ℃. In other embodiments, a food preparation controller (not shown) may be used to prepare meals for infants in any suitable manner and is not limited to food temperature control. However, the food preparation controller may include the functionality of the food temperature controller 35, for example for warming a baby bottle.

In some embodiments, the illumination lamp 27 (e.g., an indoor lamp) may be controlled by BMID to illuminate 28 the infant 10, for example, at different light frequencies (e.g., colors) and/or light illumination levels. BMID 10 can control the illumination lamp 27, which can be dynamically changed to provide different colors and/or illumination levels to the infant 15 at any time interval.

In some embodiments, the system 5 may include a plurality of electronic resources 110, denoted as N electronic resources: ELECTR RESR 1a, … …, ELECTR RESRN 110B, where N is an integer, these electronic resources may include N databases denoted DB 1a, … …, DBN 112B, respectively. The memory 95 may also include a local database 96.

In some implementations, the optical subsystem may refer to the imaging camera 81, the projector 18, or both. The audio system may refer to microphone 82, speaker 16, or both.

In some embodiments, the system 5 may include a server 100 in communication 31 with any of the elements shown in fig. 1 via a communication network 30. The server 100 may include a processor 105, communication circuitry 106, memory 107, and/or input/output (I/O) devices 108 (such as a display, keyboard, mouse, etc.). The processor 105 of the server 100 may be configured to perform any or all of the functions performed by BMID, such as controlling the GUI manager 101 of the mobile device associated with the plurality of users associated with the plurality of infants and/or the MLM/algorithm module 102 for processing data of the plurality of infants. Additionally, in other embodiments, the server 100 may control a plurality of infant monitoring and interaction devices (BMID) for use with different ones of the plurality of infants using the system 5.

In some embodiments, the system 5 may provide a movement and personalization system that may include bi-directional video communication with the infant 15 via a plurality of sensors 80 and/or peripheral/IoT devices, an Artificial Intelligence (AI) -based infant and toddler development system, an AI-based infant and toddler education system, an AI-based infant and toddler entertainment system, or any combination thereof for pacifying. AI-based refers to the use of machine learning models and algorithms.

In some embodiments, the physical algorithm module 66 may process output data from the plurality of sensors 80 to implement the functionality of a breath detector, a temperature detector, a heart rate detector, other vital sign detector, or any combination thereof. The physical algorithm module 66 may use a respiration monitoring algorithm and a heart rate monitoring algorithm to implement respiration detection and heart rate detection, respectively, from the output sensor data, as will be described below.

Fig. 2A is a first exemplary screen shot of a graphical user interface 55 of a mobile device 50 in a system 5 configured to pacify an infant 15 in accordance with one or more embodiments of the present disclosure. The first exemplary screenshot of GUI 55 may be used in a pacifying use case. GUI 55 may include indicia of a sleep state 120 of the infant, a heart rate 122 (in heart beats per minute BPM), and a volume level 124 of an audio signal played to infant 15 via speaker 16. GUI 55 may include graphical objects 125 (e.g., icons to be pressed) for user 45 to activate to perform different interactive functions with infant 15. GUI 55 may display real-time images and/or video of infant 15 to user 45 in area 57.

Fig. 2B is a second exemplary screen shot of a graphical user interface 55 of the mobile device 50 in the system 5 configured to prepare a food product for feeding 15 an infant in accordance with one or more embodiments of the present disclosure. The second exemplary screenshot of GUI 55 may be used in a feeding example. GUI 55 may include indicia such as a nutrient source 130 for the food product (e.g., breast milk), a set state 132 for food temperature controller 35 (e.g., the user may press the graphical object "immediately warm up"), and a set state 134 field that may allow user 45 to enter a warm up time.

Fig. 2C is a third exemplary screen shot of a graphical user interface 55 of the mobile device 50 reminding the user 45 to feed the infant 15 in accordance with one or more embodiments of the present disclosure. A third exemplary screenshot of GUI 55 may be used in a feeding example. The GUI may include indicia such as time 140 when food (e.g., breast milk) will be ready to be fed to the infant 15, buttons to record feeding 142 after feeding is completed, and food preparation status indicator 144 fields (e.g., bottle preparation status-off/warming).

Fig. 2D is a fourth exemplary screen shot of a graphical user interface 55 of a mobile device 50 in a system 5 for educating an infant 15 by projecting 22 an image 25 in accordance with one or more embodiments of the present disclosure. A fourth exemplary screenshot of GUI 55 may be used in an educational use case. GUI 55 may include indicia such as a rabbit icon 145 for informing user 45 that BMID is projecting a rabbit to infant 15, image change buttons 146, BMID for changing images, an audio waveform indicator 147 of audio signal 18 associated with rabbit image 25 being played by speaker 16 of BMID, and other graphical objects 148 to be activated by user 45 (e.g., icons to be pressed by the user). The GUI may also include a real-time video feed of the infant 15.

Fig. 3 is a table 150 of input data sources and output peripherals/IoT devices used by the system 5 to implement the following three use cases in accordance with one or more embodiments of the present disclosure: feeding examples 152, pacifying examples 154, and educational examples 156.

In some embodiments, although the system 5 may include the same input data source from the same plurality of sensors 80, the MLM/algorithm 65 may be different (e.g., trained differently) for implementing three different use cases: feeding examples 152, pacifying examples 154, and educational examples 156. In other embodiments, the MLM/algorithm 65 may be trained on the system 5 to implement all use cases.

In some embodiments, for feeding example 152 and pacifying example 154, mlm/algorithm 65 may include a baby-specific behavioral state detection machine learning model that receives as input audio signal data from microphone 82, image data generated from imaging camera 81 and sensor data from thermal imager 83, lidar device 84 and RF device 85, and baby-specific personal data unique to baby 15. The sensor data may be algorithmically processed to, for example, respiratory waveform signal data of the infant 15, spatial body temperature distribution data of the infant 15, heart rate waveform signal data, and/or any suitable signal data derived from output data of any of the plurality of sensors 80.

In feeding example 152, the infant specific behavioral state detection machine learning model may be trained to output an indication of infant hunger, transmit a reminder of infant hunger to user 45 via mobile device 50 (see fig. 2B and 2C), and transmit instructions to food temperature controller 35 to set a predefined temperature to change the food to feed 15 the infant.

In pacifying use case 154, the infant-specific behavioral state detection machine learning model may be trained to output an indication of infant hunger, transmit a reminder to user 45 via mobile device 50 indicating infant anxiety, about to wake up, or both (see fig. 2A), and transmit instructions to the audio system, optical subsystem, or both to perform actions to reduce infant anxiety, such as infant crying. These actions may include, but are not limited to: (1) causing the audio system to generate and play a pacifying sound for the infant 15 that reduces the level of anxiety or irritation of the infant 15, (2) causing the audio system to generate and play a sleep-aiding sound for the infant 15 that causes the infant 15 to fall asleep and/or prevent the infant from waking up, and/or (3) causing the optical subsystem to project a relaxed image to the infant that reduces the level of anxiety or irritation of the infant 15 when the infant 15 views the relaxed image for pacifying use cases.

In some embodiments, for educational use case 156, mlm/algorithm 65 may include an infant-specific educational machine learning model that receives as input audio signal data from microphone 82, image data generated from imaging camera 81, and sensor data from thermal imager 83, lidar device 84, and RF device 85, as well as infant-specific personal data unique to infant 15. The sensor data may include, for example, respiratory waveform signal data of the infant 15, spatial body temperature distribution data of the infant 15, heart rate waveform signal data, and/or any suitable signal data derived from any of the plurality of sensors 80.

In educational use case 156, an infant-specific educational machine learning model may be trained to output (1) an indication that an infant understands or does not understand a visual image and an audio output associated with the visual image (see fig. 2D), and (2) an infant-specific educational recommendation based on the understanding indication. In response to the output of the infant-specific educational machine learning model, for example, the processor 60 may then perform at least one of the following based on the infant-specific educational recommendation: (1) Modifying the at least one infant-specific education program when the at least one indication indicates that the at least one infant is not understood, or (2) continuing to execute the infant-specific education program for the at least one infant.

Fig. 4A is a first exemplary list 160 of input-output data features in a dataset for training an infant-specific behavioral state detection machine learning model and/or an infant-specific educational machine learning model, in accordance with one or more embodiments of the present disclosure. The input features 163 and output features 165 of the input feature class 162 and output feature class 164 in the table shown in fig. 4A are for pacifying examples 154 and educational examples 156. Although the first exemplary input-output dataset 160 may show, for example, 30 different features, this is for conceptual clarity only and not to limit the embodiments disclosed herein. The input-output training set may include 1000 features, 10,000 features, or 100,000 features.

Fig. 4B is a second exemplary list 170 of input-output data features in a dataset for training a baby-specific behavioral state detection machine learning model, in accordance with one or more embodiments of the present disclosure. The input features 174 and the output features 176 of the input feature class 173 and the output feature class 175 in the table shown in fig. 4B are for the feeding example 152. Although the second exemplary list 160 may show, for example, 20 different features, this is for conceptual clarity only and is not limiting of the embodiments disclosed herein. The input-output training set may include 1000 features, 10,000 features, or 100,000 features.

Regarding the list of input (stimulus) feature data and output (response) feature data shown in the tables of fig. 4A through 4B, inputs 163 and 174 show, but are not limited to, examples of: infant specific stimulation data including, but not limited to, environmental data such as weather, time of day, indoor light intensity, temperature, humidity, etc.; infant specific physiological data such as, but not limited to, respiration rate, motion heat map, nipple presence, etc. These inputs may include infant specific personal data such as, for example, but not limited to, age, height, weight, gestation length, etc.

In some embodiments, the baby-specific response data or output features 165 and 176 may include, for example, nothing, notifying the user, stopping the current action, playing a video file on the projector 18, increasing/decreasing light, increasing/decreasing temperature, changing scent, changing vibration cadence, etc.

In some implementations, any electronic resource from the plurality of electronic resources 110 can be stored in any of the N databases 112: (1) a plurality of infant-specific educational programs for a plurality of infants, (2) infant-specific stimulus data for a plurality of infant-specific stimuli provided to a plurality of infants, (3) infant-specific response data for a plurality of infant-specific responses acquired in response to a plurality of infant-specific stimuli provided to a plurality of infants, and (4) infant-specific personal data for each of a plurality of infants.

It should be noted that the system 5 may query the electronic resource to retrieve data from multiple infants. The infant specific stimulus data and infant specific response data from multiple infants may also be referred to herein as crowd-sourced data or infant crowd-sourced data, which may be data associated with other infants using the system 5 via the server100, data from laboratory experiments, or any combination thereof.

In some embodiments, data from the electronic resource 110 may be used to create a dataset for training any of the machine learning models and/or algorithms discussed herein. Thus, when data associated with a particular infant is entered into each of the machine learning models discussed herein, the output data from each of the machine learning models incorporates data (e.g., learning) from approximately 100 infants, 1,000 infants, 10,000 infants, or a large number of infants in the 100,000 infant amount.

In some embodiments, the infant-specific stimulation data from the plurality of infants may further include simulated infant-specific physiological data for the plurality of infants. For example, image data from an image camera imaging the infant and audio data from a microphone or sound transducer that samples the infant's voice and/or sound emanating from the infant may be processed to simulate or calculate infant-specific physiological data, such as, for example, heartbeat data (see, for example, algorithms associated with fig. 10-14) and/or respiration data (see, for example, algorithms associated with fig. 15-19). The analog data may also be included in the data set.

In some embodiments, to generate a training data set, infant-specific stimulation data, and infant-specific response data for each infant from the plurality of infants, the data store database in the electronic resource may include the same data as shown in the exemplary listing of fig. 4A-4B from which image data and/or audio data features for each infant may be extracted to train the machine learning model.

In some embodiments, the respiratory rate signal data may be simulated using algorithms associated with fig. 15-19. In summary, image (video) data and/or audio data of the infant may be used to identify pixels in the image associated with the infant's chest region and to monitor the characteristic periodic respiratory motion of those identified pixels in time. From this pixel variation analysis, the respiration rate can be calculated. If characteristic periodic respiratory motion is not detected, the system 5 may generate a notification, such as a reminder, on the GUI 55 that the infant is not breathing.

In some embodiments, the spatial body temperature distribution data may be acquired from an IR array thermal imaging camera, such as, for example, an MLX90640 IR array thermal imaging camera from southern microsatellite electronics limited, post code 518033(Waveshare Electronics World Trade Plaza south side,Fuhong Rd,Futian District,Shenzhen,518033,China), 32 x 24 pixels, 110 ° FOV, from the fossa region, fossa, city, china. When the thermal imaging camera is positioned to image a baby and may also capture objects in the immediate environment of the baby (e.g., a blanket of the baby), the output data from the camera may be a spatial map of the baby's external body temperature at different locations along the baby's body contour and the temperature at certain spatial points along the object contour in the baby's environment.

In some embodiments, the heartbeat signal data may be calculated by applying the algorithm described below in connection with fig. 10-14 to image (video) data of the infant. The infant heartbeat signal data may be, for example, a calculated heartbeat signal h _i (t) 3465, as mentioned below in fig. 11E.

In some embodiments, infant movement data (e.g., movement heat maps and/or movement capabilities of the infant's movement) may be derived from image (video) data and/or spatial body temperature distribution data. The pixel change algorithm may be applied to video data, for example, which may be used to track pixels in time to identify motion capabilities, for example, where low-capability motion is defined as small-amplitude motion of an infant and high-capability motion is defined as large-amplitude motion of an infant, such as infant turn-over. In other embodiments, the anatomical motion of the infant (head, shoulders, etc.) also known as the infant's skeletal detection algorithm may be used to detect the motion of the infant.

In some embodiments, the infant voice classification data may include the infant's voice rate, voice size, voice type (crying, crying upon hunger, crying seeking attention, craving, choking, coughing, swallowing, etc.), and voice pitch, which may be determined by applying a classification model and/or genetic classification to the raw audio data, for example.

It should be noted that the input-output data characteristics of the infant-specific characteristics and the environment-specific characteristics for each use case in fig. 4A and 4B are related to infant-specific stimulus data and infant-specific response data stored in the database of electronic resources. For example, the baby-specific stimulation data (e.g., baby-specific features 225) may include baby-specific personal data, breath detection data, heart rate detection data, baby voice data, baby temperature data that may represent the behavioral state of a baby, and environment-specific features 230 that may be entered into the machine learning model at any time. Thus, the infant-specific response data may be indicative of an infant's response or action to change the infant's behavioral state. When using this data to train the machine learning model, the machine learning model may incorporate environmental stimuli and infant behavior states in infant-specific stimulus data for each of the plurality of infants in order to output actions, recommendations, notifications, and/or behavior state indications for changing the infant behavior state of each of the plurality of infants.

In some embodiments, output data generated from the plurality of sensors 80 may be preprocessed using different preprocessing algorithms to generate input data features for training the machine learning model. For example, the output data of the image camera 81, the thermal imager 83, the lidar device 84 and/or the RF device 85 may be used in an algorithm disclosed below that may determine the respiration rate and heart rate of the infant 15 over time. The processor 60 may use output data from the plurality of sensors 80 in the 3D map generation module 61 to generate a time-varying 3D map of the infant 15 in order to generate a motion heat map and/or motion capabilities. The machine learning model may be trained to output different actions and transmit instructions to different peripheral/IoT devices. These actions may include, but are not limited to, playing an audio file (e.g., equalizer) with time-varying parameters such as changing volume, frequency of audio signals, etc., playing a video file, and changing projected brightness, e.g., applying vibration intensity and vibration pattern to the infant 15 via the vibration unit 40, changing the indoor light 28 (e.g., controlling the illumination lamp 27 to change light color and illumination intensity), and/or changing different IoT device states through BMID to control thermostats, humidifiers, actuators for opening/closing windows, activating fans, food temperature controller 35, and/or fragrance devices within the infant's room to change indoor temperature, humidity, air pressure, scent changes in the sensing area 12.

In some embodiments, the system 5 may collect data from the infant through the plurality of sensors 80, such as, for example, behavioral data, and may provide stimulation or pacifying through the peripheral/IoT devices after processing the behavioral data by the MLM/algorithm module 65. For example, a change in the breathing rate of the infant or a crying of the infant, a change in the sleep and wake patterns of the infant may be determined. The system may then react to the infant's behavioral patterns to provide stimulation when appropriate and pacifying when appropriate (e.g., reducing the level of anxiety or irritation of the infant). In some embodiments, the system 5 may detect infant hunger. The system 5 may prepare the food (e.g., by the temperature controller 35 plus Wen Yinger the feeding bottle 32) and then may alert the user 45 via the mobile device 50 so that when the user 45 arrives at the infant 15, the food is ready to be fed to the infant 15 (e.g., at the correct predefined temperature).

In some embodiments, if the system 5 detects infant fatigue or discomfort based on both video and audio analysis, the system 5 may be configured to pacify the infant using the AI-based pacifying system by playing pacifying music, providing vibrations, and/or projecting images to the infant. In other embodiments, the system 5 may determine whether the level of discomfort to the infant may be reduced by pacifying alone. In other embodiments, the system 5 may use an AI-based algorithm to determine to send a reminder to the user 45 via the GUI 55 that attention to the infant is needed.

In some embodiments, if the system 5 evaluates that only pacifying is needed, the system 5 may automatically begin playing the pacifying sound. The best comfort sound for the infant may be selected, i.e. tune, volume and/or play duration may be AI-adjusted for the infant, for example. In other embodiments, the comfort sound and volume may be adjusted as desired based on the infant's response. The sound may be a tune, white noise, hum, prerecorded sound of a parent, grandparent or caretaker, or any combination thereof. In other embodiments, the machine learning model may output pacifying sounds to the infant 15 to be generated by the speakers.

In some embodiments, the system 5 may implement an infant-specific pacifying program (e.g., tailored to a particular infant), which may include the steps of:

1. crying (via audio and/or video) can be detected.

2. At least one of sound, video, image or vibration may be transmitted to calm the infant. In some embodiments, the sound, video, image, or vibration may continue to be played until the infant is asleep.

3. The behaviour of the infant may be sampled by the system 5, such as for example at predefined time intervals.

4. If the system 5 detects that the infant has not been calm (or is not asleep), the system 5 may change at least one of: sound played to the infant, video or images projected for viewing by the infant, and/or vibration applied via a vibration unit, such as a vibration pad. For example, one or more additional sounds, videos, images, and/or vibrations may be added. The volume of sound may be made larger or softer. The type of sound may be changed (changing tune, changing sounds recorded by the caretaker, changing what the caretaker speaks of, changing to white noise, changing to hum, etc.). Similarly, the vibration applied to the infant may be made stronger or weaker, or the type of vibration may be changed. The video or image displayed to the infant may be changed. The display of the video or image may be made brighter, lighter, stronger or weaker. The color image may be recoloured.

5. The system can recheck or resample the infant's behaviour (crying noise/changes in respiration rate etc.).

6. Repeating steps 4 to 6 until the infant is calm (or falls asleep).

In some embodiments, the system 5 may refine the machine learning models/algorithms 65 by retraining them in order to model the real-time behavior of a particular infant. Thus, when it is desired to pacify the infant again, the system 5 may begin the future pacifying process from a known starting point, as which actions (such as sound, illumination, projected image, vibration, etc.) have been captured by the retrained model were previously effective for pacifying the particular infant.

In some embodiments, the system 5 may query data from at least one electronic resource store 110, i.e., infant-specific stimulus data for a plurality of infant-specific stimuli provided to a plurality of infants and infant-specific response data for a plurality of infant-specific responses acquired in response to the infant-specific stimuli provided to the plurality of infants. Thus, the system may use data from multiple infants to train machine learning models that may capture more efficient data, for example, for infants with similar characteristics to a particular infant. The system may be configured to dynamically adapt the solution to each child.

In some embodiments, the system 5 may be configured to predict an increase in stress level of the infant and automatically execute the solutions output by the machine learning model to relieve the infant of stress.

In some embodiments, the monitoring subsystem (e.g., which may include a bi-directional audio system) may be separate from the host system (e.g., BMID a) and may be portable such that the pacifying system may be used when the infant is not in place of the host system, such as, for example, when the infant is in a stroller or in a vehicle but is communicatively coupled via a wired or wireless communication protocol. In other embodiments, the monitoring subsystem may include a battery or any suitable power storage device. In other embodiments, the monitoring subsystem may be recharged for connection to the host system and/or may be connected to at least one conventional power supply system.

In some embodiments, the system 5 may include a two-way video system that may enable the user 45 (such as a parent, family member, and/or caretaker associated with the infant 15) to communicate bi-directionally with the infant 15. The bi-directional video system may project the video image of user 45 on any convenient flat surface in the vicinity of the baby. The video image of the infant may be displayed to the user via any convenient screen such as, for example, a computer display (e.g., a notebook, desktop, and/or tablet) or GUI 55 of mobile phone 50.

In some embodiments, the system 5 may be configured to provide stimulation and education to the infant. The image may be shown by telling the baby what the image is, for example, by voice via the speaker 16. The monitoring system (e.g., BMID, 10) may be configured to determine from the infant's response (such as, for example, its eye movement, body movement, and/or the infant's voice) whether the infant 15 is focusing on the displayed images or sounds and whether the infant 15 understands what the infant 15 sees. The system 5 may also be configured to determine whether the infant is boring, for example, by eye movements and/or body movements of the infant, and BMID may interrupt the infant-specific educational program for the infant 15 until the infant 15 again shows interest in the projected image 25. In some embodiments, the user 45 may be alerted via the GUI 55 of the mobile device 50 that, for example, the baby has learned a new word or new concept. Thus, based on AI, a personalized infant-specific educational program can be provided for the infant 15. In other embodiments, the user 45 may be able to change or adjust personalized baby-specific educational programs in real-time via the GUI 55.

In some embodiments, personalized infant-specific educational programs may be used for many years, for example, where educational and pacifying use cases change as children develop. These phases may cooperate with, for example, a service provider, a subscriber, an interactive game, a user, or any combination thereof.

In some embodiments, educational, developmental, and entertainment systems based on personalized infant-specific educational programs may be considered to be the infant's "university". The use of personalized infant-specific educational programs for infants 15 may be free or subscription-based. Subscription-based systems may use enterprise-to-enterprise models or enterprise-to-consumer models of entertainment and mass media markets.

Fig. 5A is an algorithm flow diagram 200 of a computer-based system for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure. The algorithm flow chart may be used to generate a training dataset of the machine learning model disclosed herein for each use case. Once trained, the machine learning model may incorporate the functionality of one or more of the algorithm blocks in the algorithm flow diagram 200 into the machine learning model.

In some embodiments, with respect to the physical algorithm 220 data pipeline, the processor 60 may be configured to process raw sensor data 215 (e.g., video, audio, temperature, light, RF, etc.) related to the infant 205 and/or the environment 210 in which the infant 205 is located. It should be noted that the physical algorithm module 66 of fig. 1 is equivalent to the physical algorithm 220 block of fig. 5A. The sensor data 215 may be input into a physical algorithm 220. The physical algorithm 220 is configured to output infant metadata and environmental metadata from which features characterizing the infant 205 and environmental conditions 210 may be extracted. The baby-specific features may include, for example, baby-specific physiological data (such as baby heart rate, baby heart rate variability, baby temperature at different locations on the baby's body), baby voice classification data (such as baby's voice rate, voice size, and voice pitch), baby movement rate, movement location, and the like. The environment-specific features may include light intensity, sound, room temperature, etc.

In some implementations, the physical algorithm 220 may pre-process the sensor data to generate a temporal waveform and/or image (video) by, for example, applying filtering, decimation, and calibration operations to the sensor data. The signal-to-noise ratio of the time waveform and/or image may be improved by filtering the relevant frequency bands of the infant for each sensor. The distinction between environmental data and infant data may be implemented using a sensor fusion algorithm (e.g., cross-correlation of sound transducers directed at different locations). Then, the signal transformation is generated, for example, by applying a 2D/3D fast fourier transform using a 1D fourier transform, a 2D spectrogram, a wavelet, for images and videos, a hilbert transform (for envelope signals), a cepstrum (for example, autocorrelation signal balance), or the like.

In some implementations, feature extraction algorithms can be applied to the correlation vectors to diagnose to extract features (such as, for example, infant-specific features 225 and environmental features 230) using basic mathematical operations (such as, for example, sound and light mean and root mean square, probability density functions, waveform moments, etc.). Other infant-specific features may also be extracted using advanced algorithms, such as for example infant heart rate may be extracted using euler amplification of FFT phases to identify repeated changes and its heart rate quantified using the (smart heart beat) heart beat detection algorithm disclosed below.

In some implementations, the baby smile rate may be extracted from video image data from the imaging camera 81 using a pattern recognition algorithm that identifies smile segmentations for various face sizes and quantifies their ratio. Other features may also be extracted using an unsupervised deep learning method (such as a convolutional automatic encoder) that can extract potential features describing the state of the baby at the time of measurement. These automatic encoders may be pre-trained on multiple infant data sets, such as multiple infant data sets queried from one or more databases 112 stored in one or more electronic resources 110, data from multiple infants with specific meta-features characterizing each infant in the multiple infants.

In some embodiments with respect to time-driven data pipeline 250, processor 60 may collect features (e.g., general environmental features 255) using features generated by a physical algorithm pipeline, such as infant-specific meta-features 225 and environment-specific meta-features 230, and environment-related actions. Time-driven data pipeline 250 may generate a time sequence X_ (i-n), X_ (i-n+1), … …, X_ (i) for each feature X_ (i), where n represents the nth measurement prior to measurement i. (note that the time-driven pipeline module 67 of fig. 1 is equivalent to the time-driven data pipeline 250 box of fig. 5A.) for each feature's time series, the time-driven pipeline 250 may calculate a time-dependent (TD) feature 270 that characterizes feature statistics and feature progression over time. The time-dependent infant-specific features may include infant-specific features that exhibit time-dependence, such as respiration rate, heart rate, etc., with the exception of infant-static feature 245 in fig. 5A. Similarly, time-dependent environmental specific features may include environmental specific features that exhibit time-dependence (e.g., time of day, weather, humidity, air pressure, etc.). The TD features, such as the mean, variance, maximum, minimum, or percentile (x) of the features over a window of k records, may be calculated using basic arithmetic operations. The TD feature 270 may be calculated using advanced signal processing functions, such as calculating the probability gradient for n time windows and taking the maximum (or minimum) gradient. The TD characteristics may also be calculated using a sensor fusion algorithm, for example, baby motion characteristics relative to environmental specific characteristics (such as environmental sounds, light, etc.).

In some embodiments, a baseline 265 may be generated that characterizes the infant status given available information including actions applied to the infant's environment (e.g., a set of features 240 and 245 reflecting a particular time window, TD features 270, actions 235, and environment-specific features 255). The baseline 265 may be calculated using a parametric configuration (such as using k features and TD features 270 for a particular window), or the baseline 265 may be calculated using an advanced ML method (such as a depth auto-encoder that encodes k features and TD features to a particular state for a particular window or a variance auto-encoder that learns a baseline probability density function per particular window). The output of the time-driven pipeline 250 may then be relayed to the behavior model pipeline 275 and additionally stored in a history bucket (e.g., feature history 260).

In some embodiments, the behavioral model 275 pipeline may collect output data (features, TD features 270, baseline 265 signatures, metadata, and actions taken on the environment) of previous data pipelines to model the behavior of the infant 15. It should be noted that the behavior model 68 module of FIG. 1 is equivalent to the behavior model 275 block of FIG. 5A. The behavioral model 275 may generate a model describing the mental, emotional, and behavioral states of the infant 15 given external conditions (such as environmental states, movies, sounds, lights, or other external states, such as food consumed, breast feeding time, etc.).

In some embodiments, the behavioral model may include sub-models such as stress level models (based on infant motion, heart Rate (HR), heart Rate Variability (HRV), crying, etc.), happiness characteristic models (based on smile mode, HR, HRV, and sound), sleep characteristic models (total sleep time, time during each sleep mode, number of wakeups, etc.), and the like. The behavioral model 275 may be trained using pipelined input data and data from multiple infants with similar infant-specific characteristics (such as similar age, gender, similar environmental characteristics, etc.). The behavioral model 275 may be constructed using a supervised approach in which data is trained with respect to behavioral characteristics or an unsupervised approach in which all features are encoded as a single state defining infant behavior.

Fig. 5B illustrates an exemplary width 340 and depth 330 neural network model 315 for modeling a smile rate (node 317) of an infant in accordance with one or more embodiments of the present disclosure. An example of the development of such a model is to employ a breadth and depth neural network 315, where the dataset used to train the network is the features, time-dependent features, environmental meta-features, and actions taken, and then train the breadth and depth model in a supervised approach to the smile rate of the baby (e.g., with a Mean Square Error (MSE) loss function). In this way, the model may correlate between the environmental states (nodes 332, 334, and 336) and the happiness characteristic (e.g., smile rate of node 317), so the environmental states may be tuned later to improve happiness. The importance of environmental parameters to infant behavioral status may be captured using the breadth and depth architecture 315. As the infant 15 grows and changes continuously, a portion of the training data may include specific infant data and data from similar infants from multiple infants, which changes over time to maximize the infant's behavioral and/or emotional state.

In some embodiments regarding the feedback environment generator 300, the feedback environment generator 300 may compile a user configuration 309 with the digital twin 290 that may be input (e.g., set up use cases, such as pacifying, feeding, and/or educational) by the user 45 via the GUI 55 of the mobile device, for example. The digital twin infant 290 may capture all behavioral models 275 and infant environment configurations to provide actions based on the infant's environment. The digital twin infant 290 may be a set of parameters representing different responses of the infant to different stimuli when the infant is in different behavioral states taken at different time intervals. The digital twin infant and user configuration 309 may be input to the feedback environment generator 300

In some embodiments, the feedback environment generator 300 may output notifications, actions, recommendations, or any combination thereof, to change the behavioral state of the infant to a desired behavioral state (predefined behavioral state) over time based on the use case. Behavioral states may include, for example, anxiety states, resting states, sleep states, hunger states, satiety states, stress states, playful states, unobserved states for a particular subject, educational states for a particular subject, and the like. For pacifying use cases, the behavioral state of the infant may change from an anxiety state to a resting state (desired behavioral state). For feeding examples, the behavioral state of the infant may change from a hunger state to a satiety state (desired behavioral state) via actions of the user when feeding the infant in response to the system automatically preparing a food item (e.g., warming infant milk). For educational examples, the behavioral state of an infant may change from an unobserved state to a specific subject to an educated state (desired behavioral state) to the specific subject.

In some embodiments, examples of actions output from feedback environment generator 300 may be, for example, lowering/raising light intensity, changing light color, playing sound (such as that of a mother, family, female, male, or song, or any other digital sound source), projecting a movie on a ceiling, doing nothing, stopping a current action, and the like. The outputted notification may include sending a notification, such as a reminder, to the user 45, such as a notification of infant hunger, on the GUI 55. The output recommendations may include nothing, stopping the current action, or changing the educational course of the infant. This may be implemented by using behavior model functionality and optimizing it to the state requested by the user (e.g., pacifying the infant). The minimization of the loss function for the crying feature of the infant, or the movement feature of the infant, or any other behavioral feature generated by the behavioral model 275, may be used for optimization. This process of performing actions to change the state of the infant may be repeated and iterated until the loss function reaches equilibrium (as defined by the model). The iterative process may be applied using arithmetic functions, ML models, or reinforcement learning techniques with the addition of random parameters to produce two paths for minimization.

In some embodiments, the algorithm flow diagram 200 of the computer-based system for monitoring and interacting with infants may be adjusted for each of the use cases described herein, such as feeding, pacifying or educating infants based on training different models using training data sets having input-output data characteristics of each of the use cases, for example, as shown in the exemplary embodiments of fig. 4A-4B. The input data features may be the infant-specific features 225 and the environment-specific features 230 of fig. 5A, which map to actions 302 and/or IoT activations 308 based on the output of the feedback environment generator 302. Examples of infant-specific features 225 as shown in fig. 5A may include, but are not limited to, breath detection (respiration rate), heart rate (heartbeat signal data), infant voice (infant voice classification data), infant temperature as shown in inputs 163 and 174 of fig. 4A and 4B.

In some embodiments, the infant-specific behavioral state detection machine learning model and/or the infant-specific educational machine learning model may be any suitable machine learning model that may be trained for a particular use case using a data set as shown, for example, in fig. 4A and 4B.

In some embodiments, the infant-specific behavioral state detection machine learning model and/or the infant-specific educational machine learning model may incorporate the functionality of the time-driven pipeline 250, the behavioral model 275, and the feedback environment generator 300 such that the infant-specific features 225 and the environment-specific features 230 map to each use case-specific actions 302. In other words, for each use case (e.g., pacify, feed, and educate), the blocks shown in algorithm flow 200 may be used to generate a use case specific training dataset for training an infant specific behavioral state detector machine learning model and/or an infant specific educational machine learning model.

In some embodiments, the baby-specific behavior state detector machine learning model and/or the baby-specific educational machine learning model may be implemented using any suitable machine learning model, such as a classification neural network model, a long-short term memory (LSTM) model, a convolutional neural network, and/or a multi-class neural network. These machine learning models may be trained for specific use cases using training data sets.

In some embodiments, an infant specific behavioral state detector machine learning model may be used to pacify and feed the feeding examples. The infant-specific behavioral state detection machine learning model in pacifying use cases may map infant-specific and environmental-specific features to actions such as, for example and without limitation, generating pacifying sounds for an infant by speaker 16 when an infant is determined to be anxious, generating sleep-aiding sounds by speaker 16 when the system detects that an infant is about to wake up (so as to cause the infant to fall asleep again), and/or projecting a relaxed image for viewing by the infant via a projection device (e.g., projector 18). For feeding examples, the infant-specific behavioral state detection machine learning model in the feeding examples may map infant-specific features and environmental-specific features to actions such as, for example and without limitation, changing the temperature of the food temperature controller 35 when starvation of the infant 15 is detected in order to prepare food (such as infant milk in the feeding bottle 32) for feeding to the infant.

In some embodiments, determining that the infant is hungry may include considering the period of time that the infant is feeding (e.g., every three hours), the time that the infant is last feeding, and observing that the infant may exhibit some movement during the time interval prior to the infant being fed. This data may be acquired, for example, among a plurality of infants and for infants 15 that may be used to train the AI model such that when the infant 15 exhibits similar movement within a predefined time after the infant's last meal, the act 176 of the AI model may be, for example, sending a reminder 140 that the infant is in preparation for food (fig. 2B and 2C) or just that the infant is hungry (user notification) on the GUI 55 on the mobile device 50 of the user 45. For example, determining infant hunger may include considering movement rate, respiration rate, time of day, user configuration (user feedback), such as user indicating on GUI 55 that the infant has been fed or that the infant needs to gain weight before 2 hours, infant age, weight, height.

In some embodiments, the identification by the system 5 that the infant is sleeping and/or may be about to wake up and/or has wake up may include acquiring image and/or audio data of the infant 15 and/or the motion and/or bone detection and/or voice classification of each infant of the plurality of infants. This data can be used to train an AI model to identify the infant anxiety and/or about to wake up.

In some embodiments, the feedback environment generator 69 may be trained with data from multiple infants regarding which audio stream (tune), audio volume, equalizer parameters successfully pacified the infant. Pacifying an infant as referred to herein may be an iterative process of playing a pacifying sound or audio stream to the infant and/or projecting a relaxation-causing image (e.g., a visual image 25 of a rabbit) to the infant, which quieters the infant and/or re-falls asleep. For example, this may be evaluated by the AI model from motion data detected from image data of an image camera indicating smaller motion and/or audio data indicating no crying and/or that it may be determined that the infant has fallen asleep. In other embodiments, as such, the sleep-aiding sound may be any audio stream that determines to cause the infant to fall back asleep when the infant wakes or is about to wake as predicted by any AI model. The AI model may be trained to output any projection features shown in output feature 165.

In some embodiments, the AI model may be trained to output the type of vibration (vibration intensity and/or vibration cadence) that the vibration unit 40 applies to the infant in order to pacify the infant, as described above.

In some embodiments, the infant-specific educational machine learning model for educational use cases when subjecting an infant to an infant-specific educational program may map infant-specific features and environment-specific features to actions, such as outputting an indication that the infant understands or does not understand a visual image based on the infant-specific educational program and sounds associated with the visual image, as shown for example for the rabbit image shown in fig. 2D, or may output an infant-specific educational recommendation, such as to modify the infant-specific educational program in the event that the infant-specific educational program is too difficult for the infant or the infant feels boring (when the infant may be detected boring).

In some embodiments, act 302 of detecting a machine learning model and/or an infant-specific educational machine learning model predicted by an infant-specific behavioral state after training with a training data set as generated in the algorithm flow of fig. 5A (e.g., environmental changes experienced by infant 15 shown in outputs 165 and 176 of fig. 4A and 4B) may be performed iteratively in order to change the infant's behavioral state according to the goals of a particular use case. For example, for a pacifying use case, the infant may experience one or more environmental changes within a predefined time interval caused by system 5 performing any actions as shown, for example, in the list of outputs 165 in fig. 4A, until the infant's behavioral state changes from an anxiety state to a resting state (or falls asleep again).

In some embodiments, the machine learning model may dynamically change parameters in motion until the behavioral state of the infant changes from an anxiety state to a resting state (or falls asleep again). For example, detecting a changing output of a machine learning model based on a baby-specific behavioral state, parameters of an action (e.g., pacifying a sound) such as changing an audio file, changing BPM in audio, changing volume, changing audio flow, and/or changing equalizer parameters may occur multiple times within a predefined interval.

Fig. 6A is a flow diagram of a method 350 for pacifying an infant 15 in accordance with one or more embodiments of the present disclosure. Method 350 may be performed by processor 60 and elements BMID in system 5.

The method 350 can include receiving 355 image data, audio signal data, and sensor data associated with at least one infant.

The method 350 may include inputting 360 image data, audio signal data, sensor data, and infant-specific personal data associated with at least one infant to at least one infant-specific behavioral state detection machine learning model, wherein the at least one infant-specific behavioral state detection machine learning model is trained using a dataset based at least in part on infant-specific stimulus data, infant-specific response data, infant-specific personal data associated with a plurality of infants.

The method 350 may include receiving 365 output of at least one infant anxiety, about to wake up, or both from at least one infant-specific behavioral state-detecting machine learning model.

The method 350 may include transmitting 370 sensor data, at least one infant anxiety reminder, at least one infant about to wake reminder, or any combination thereof, to at least one communication device of at least one user over a communication network.

The method 350 may include transmitting 375 instructions based on the output that cause the audio system, the optical subsystem, or both to perform at least one of:

(i) When at least one infant is anxious, a pacifying sound is generated by the speaker,

(Ii) When at least one infant is about to wake up, a sleep-aiding sound is generated by the speaker, or

Fig. 6B is a flow diagram of a method 380 for feeding an infant 15 in accordance with one or more embodiments of the present disclosure. Method 380 may be performed by processor 60 and elements of BMID in system 5.

The method 380 may include receiving 385 image data, audio signal data, and sensor data associated with at least one infant.

The method 380 may include inputting 390 image data, audio signal data, sensor data, and infant-specific personal data associated with at least one infant to at least one infant-specific behavioral state detection machine learning model, wherein the at least one infant-specific behavioral state detection machine learning model is trained using a dataset based at least in part on infant-specific stimulus data, infant-specific response data, infant-specific personal data associated with a plurality of infants.

The method 380 may include receiving 395 at least one indication of at least one infant hunger from at least one infant specific behavioral state detection machine learning model.

The method 380 may include transmitting 400, to at least one communication device 50 (e.g., mobile communication device 50) of at least one user, a reminder to feed at least one infant, sensor data, or both, over the communication network 30.

The method 380 may include transmitting 405 an instruction that causes the temperature control system 35 to change a predefined temperature of at least one food item in preparation for feeding at least one infant.

Fig. 6C is a flow diagram of a method 410 for educating an infant 15 according to one or more embodiments of the present disclosure. Method 410 may be performed by processor 60 and elements BMID in system 5.

The method 410 may include projecting 415, by the projection device, at least one visual image to the at least one infant based on an infant-specific educational plan from a plurality of infant-specific educational plans for the at least one infant.

The method 410 may include generating 420, by the audio system, at least one sound associated with the at least one visual image.

The method 410 may include receiving 425 image data, audio signal data, and sensor data associated with at least one infant.

The method 410 may include inputting 430 image data, audio signal data, sensor data, at least one visual image, at least one sound, and baby-specific personal data associated with at least one baby to at least one baby-specific educational machine learning model, wherein the at least one baby-specific educational machine learning model is trained using a dataset based at least in part on baby-specific stimulus data, baby-specific response data, baby-specific personal data, a plurality of baby-specific educational programs associated with a plurality of babies. The at least one sound generated by the speaker may be a pacifying sound, a sleep-aiding sound or a sound associated with the projected visual image, such as a cat call for the visual image of a cat, a person's voice speaking the word "cat".

The method 410 may include receiving 435 an output from at least one infant-specific educational machine learning model, wherein the output includes:

(i) At least one infant understands or does not understand at least one visual image and at least one indication of at least one sound associated with the at least one visual image according to at least one infant-specific educational program for the at least one infant, and

(Ii) At least one infant-specific educational recommendation based at least in part on the at least one indication.

The method 410 may include transmitting 440 at least one indication, at least one baby-specific educational recommendation, sensor data, or any combination thereof, to at least one communication device of at least one user over a communication network.

The method 410 may include performing at least one of the following based on at least one infant-specific educational recommendation:

(i) Modifying at least one infant-specific educational program, or, when at least one indication indicates that at least one infant is not understood

(Ii) The execution of the infant-specific educational program for the at least one infant continues.

Fig. 7A illustrates a second exemplary embodiment of a computer-based system 480 for monitoring and interacting with an infant in accordance with one or more embodiments of the present disclosure. The system 480 may include a monitor 482, a processing unit 481, a mobile application (user interface) 486, and a temperature control subsystem 485. The monitor 482 may be mounted on a wall or ceiling, or it may be mounted on furniture such as a crib, for example. It should be noted that the elements of system 480 may be used interchangeably with the elements of system 5 of fig. 2

In some embodiments, monitor 482 may include a base station portal for connecting (490, double-headed arrow) to accessories. In other embodiments, there may be no base station entry. Typically, the monitoring unit comprises a processing unit. The communication connection may be wired or wireless.

In some embodiments, monitor 482 may be connected to devices such as an optical subsystem 482a, a bi-directional audio system 484, a projector 483, a microphone and acoustic/audio detector 484a, a breath detector 482d, a temperature detector 482b, a heart rate detector 482c, a user interface 486, a temperature controller 485, a processing unit 481, a weight scale (not shown), a lidar sensor (not shown), or any combination thereof.

In some implementations, the processing unit 481 may be located within the monitor 482, in a separate unit from the monitor 482, in the cloud, or any combination thereof. The processing unit may include software, hardware, or any combination thereof configured to store data, convert sensor signals to sensor signal data, extract visual data from visual patterns, extract audio data from auditory patterns, execute a machine learning model, or any combination thereof, using visual and/or audio data as input.

Fig. 7B shows a diagram of data elements of an Infant Condition Description Vector (ICDV) in accordance with one or more embodiments of the present disclosure. ICDV may be the output of the behavioral model 275, as shown in FIG. 5A.

In some embodiments, the AI-based system may generate an infant status description vector (ICDV). ICDV may include historic and current performance of the infant's emotional state. ICDV may include current sensor data and history (IH) of the infant. The current sensor data may include data from a number of sensors, may be processed or may be raw data, may be a continuation of the instant history data, or any combination thereof. Typically, the sensor data may be processed into different signals of interest, such as, for example, heartbeat signals. New data may be concatenated to previous data, thereby forming a continuation of the instant history data.

Infant History (IH) may include 3 parts:

a. biological History (IBH) data, which may include infant-specific personal data, such as the infant's age, physical characteristics (such as, for example, height, weight, head circumference, etc.), and developmental stages (such as, for example, athletic ability, communication level, sleep ability, melatonin state estimation, etc.)

B. the infant personal history data (IPH) may include an infant-specific history response to the reaction loop, which is generated from the infant-specific history data.

C. The infant crowdsourcing historical data (ICH) may include a historical weighted average of other infant responses (e.g., data from multiple infants) that take into account IBH and IPH of other infants. The infant crowdsourcing data may be from other infants using the system, from laboratory experiments, and any combination thereof.

In some embodiments, the ICH may include infant response data for a plurality of infant-specific responses acquired in response to a plurality of infant-specific stimuli provided to a plurality of infants. The ICH may include infant specific stimulus data for a plurality of infant specific stimuli provided to a plurality of infants. The ICH may include infant specific personal data for each of a plurality of infants.

Fig. 7B illustrates an exemplary embodiment of ICDV in which current sensor data may be algorithmically processed to generate a respiration waveform 521, a temperature map 522, a sleep state 523 (e.g., asleep, wake, drowsy, awake, etc.), a crying level 524 (not crying, tinnitus, always crying, intermittent crying, loud crying, light crying, etc.), a heart rate 525, and other infant-specific historical responses 526 as disclosed herein.

In some embodiments, the IBH may include infant-specific data for the movement level 531, communication level 532, and more 533 of the infant 15.

In some embodiments, the IPH may include an audio response history 541, a projector response history 542 that may include the baby's response to projected images and/or sounds, and more 543 baby-specific data for the baby 15.

In some embodiments, the ICH may include the same types of numbers as IPH, i.e., audio response history 551, projector response history 552 that may include the baby's response to projected images and/or sounds, and more 553 baby-specific data (e.g., crowd-sourced baby-specific data) for each of a plurality of babies.

Fig. 7C and 7D illustrate exemplary embodiments of a detection reaction loop for pacifying an infant in accordance with one or more embodiments of the present disclosure. The detection reaction loop may be performed by the system 480 or BMID 10.

The detection reaction loop for pacifying the infant may comprise the steps of:

I detection

A. An embodiment of the system 480 including a pacifying method decision component (AI/machine learning) is established. The pacifying method decision part may be preloaded into the system and used only locally, or may include at least one parameter from the crowdsourcing pacifying method server 611. In other words, system 480 may use at least one machine learning model that has been trained for pacifying use cases (such as in, for example, algorithm flow diagram 200 of fig. 5A).

B. data is collected from a plurality of sensors 612 (such as, for example, optical subsystems, microphones, acoustic/audio detectors, temperature detectors, user interfaces, temperature controllers, and/or lidar). It should be noted that breath detection and heart rate detection may be determined from sensor data from an image camera, RF sensor, and/or thermal sensor to use algorithms to evaluate whether the infant is breathing and/or to detect the heart rate of the infant. Breathing and heart rate algorithms are discussed below.

C. the data is transmitted to a computing unit 613, which may be part of a camera computing processor, may be part of a separate processor, and any combination thereof, to analyze the data and determine if the infant is not pacified or about to wake up.

D. The computing unit may then perform the following steps (see fig. 7D):

i. Historical data is received from a priori data gate 620.

Sensor data is received from sensor data gate 622 and raw data is analyzed.

In ICDV calculator 621, inputs to all response systems (audio, video, reminder, etc.) are generated from the analyzed data

The results are transmitted ICDV to the response agent pool 624.ICDV the response agent pool 624 may include a computing agent, which may be one agent per response system, one agent for multiple response systems, multiple agents for at least one response system, or any combination thereof. ICDV the response agent pool 624 may generate a plurality of ICDV values from the analyzed sensor data, which ICDV values may be weighted using predefined or AI-generated weights to determine the relative importance of the generated ICDV values.

The results from each agent may be relayed to a response scheme calculator 625 to generate an action list for each response system and transmit the action list through a response API pool 626, and update actions, IPHs, and ICH through a priori data gate 620.

II reaction/pacifying

E. Activating at least one of the following functions:

a. Activating pacifying method 614

I. pacifying by sound from a speaker such as, for example, music, white/pink noise, user's voice, or any combination thereof. Although recorded voice messages are typically available, the voice may be recorded, in real-time, or both. The volume, type of sound and length of time to sound may vary depending on the reaction of the infant.

Activating the projector and displaying the pacifying video.

Activating the vibration unit.

B. Activating the functional IoT device 615

I. the food temperature controller 35 is activated to heat, for example, a food product, typically a bottle of milk.

Control ambient light (e.g., illumination lamp 27), such as, for example, turning on or off a lamp, adjusting an illumination level, and/or changing a color of the light.

Activating another IoT device.

C. Sending a notification 616 to the user

I. The notification is sent via a mobile application (e.g., displayed on GUI 55 of mobile device 50) (e.g., an application, a built-in OS notification, or any combination thereof).

A notification is sent over a telephone call (e.g., to mobile device 50).

The notification is sent by sound from a speaker system (e.g., speaker 16).

F. Data is collected from a plurality of sensors 612 (such as, for example, an optical subsystem, microphone, acoustic/audio detector, respiratory detector, temperature detector, heart rate detector, user interface, temperature controller, lidar, or any combination thereof)

G. based on the sensor data from step E, pacifying method 617 is automatically modified using AI.

H. repeating step D based on the modified parameters of step (E) and the result of step (F).

I. The modified parameters are sent to crowd-sourced pacifying method server 611 for modification of the generic pacifying method decision part (ai\machine learning).

In some embodiments, optical subsystem 482a may include a visible spectrum imaging detector (for use in daytime) and an infrared imaging detector for night vision. In other embodiments, computer vision motion detection may also be used integrally with the optical system, as part of the processing unit, or any combination thereof.

In some embodiments, the bi-directional audio system 484 may be integrated into the monitor, removably connected to the monitor, or may be a unit separate from the monitor.

In some embodiments, projector 483 may be integrated into the monitor, removably connected to the monitor, or may be a unit separate from the monitor.

In some implementations, the removable speaker and the removable projector may be charged (e.g., to charge a battery) or may be charged separately when connected to the monitor.

In some embodiments, the user interface 486 may be integrated with the monitor, may be a separate unit, or any combination thereof. In other embodiments, at least one user interface may be mobile. The user interface may be, for example, a smart phone application, a telephone messaging system, a computer messaging system, GUI 55, or any combination thereof. Examples of computer messaging systems may include email messaging systems, audible alerts, visual alerts, and/or computer-based interfaces. The smart phone application and the telephone messaging system may use audible alerts, visual alerts, tactile alerts, or any combination thereof to inform the user of incoming messages, provide messages, or any combination thereof.

In some embodiments, temperature controller 485 (e.g., food temperature controller 35 in fig. 1) may be configured to store and maintain the feeding at a predefined temperature. The feed may be milk, breast milk or prepared infant milk (formula) or milk substitutes, water and/or food for preparing the formula or milk substitutes. The feed may be kept at a low temperature until use and may be heated to an appropriate temperature at the time of use.

In some embodiments, the temperature controller 485 or processing unit 481 may determine that a feeding time is imminent based on the behavior of the monitored person (e.g., infant), and may provide a mechanism for warming milk to an appropriate temperature for use. In other embodiments, the system may include a mechanism for alerting the user 45 (e.g., a parent or other caretaker) that feeding time is upcoming, such as via a GUI 55 on the mobile device 50. In other embodiments, the system may include a mechanism for determining that a parent or other caretaker may be approaching a monitored person (e.g., an infant) to feed the monitored person. In other words, the system 5 may monitor the distance D that the user 45 may be approaching the infant 15, as shown in fig. 1.

In some embodiments, the temperature controller 485 may include both heating and cooling mechanisms, such as, for example, a peltier device for both heating and cooling, but separate heating and cooling devices may be used. The heating device may include, for example, a peltier system, an infrared heater, a heating mantle, a heating band, microwaves, hot air, or any combination thereof. The cooling device may comprise, for example, a peltier system, a cooling hood, cold air, circulating cold liquid, evaporative cooling, or any combination thereof.

In some embodiments, the processing unit 481 may be integrated with the monitor, may be a separate component from the unit, may be part of the cloud, or any combination thereof. The processing unit may collect, process, and provide feedback of the collected data to subsystems, such as temperature controllers (e.g., from IoT activation 308 in fig. 5A) and users (e.g., from user 310 in fig. 5A).

In some embodiments, the system 480 may provide personalized feedback to the user regarding the infant by collecting various data from the infant and its environment. Data collection performed by monitor 482 and/or by other sources (e.g., speaker 484, microphone 484a, thermometer 482b, and user input) may include, for example, monitoring respiration, acoustic and visual monitoring, heart rate and temperature monitoring, feeding schedules, sleep routines, crying signals, and the like. From these, the processor may determine, for example, breathing patterns and changes in breathing patterns, acoustic patterns and changes in acoustic patterns, movement patterns and changes in patterns and movement patterns, body temperature and changes therein, heart rate and changes therein, or any combination thereof.

In some embodiments, the processing unit 481 may continuously collect and process data to obtain feedback in order to generate an automated personalized solution. The information collected from the user may be used to activate various system functions to provide a customized solution for implementing various purposes of feeding, pacifying, inducing sleep and/or educational use cases. The system 480 may be connected to a user-friendly user interface 486.

In some implementations, information collected from a user via a monitor may be processed via a machine learning model/algorithm (e.g., MLM/algorithm 65 of fig. 1). The processed information may be used to activate various system functions that may provide a customized solution for feeding, pacifying, facilitating sleep and/or educational use cases, for example.

Fig. 7E schematically illustrates an embodiment of an audio generator response agent of a reaction pacifying loop according to one or more embodiments of the present disclosure.

In some embodiments, the audio generator response agent may generate sound to be played to the infant. The system may perform the following steps:

A. Data is received from ICDV.

B. A representation of ICDV is constructed from the data from the transformers 651, 652, 653, 654 and based at least in part on:

1. a response history of the infant and an appropriate waveform found in the response history of the infant.

2. Response histories for other infants in the plurality of infants, the response histories having approximately the same IBH and having similar audio waveforms found in the response histories of the other infants.

3. The current response history of the infant from the start of the pacifying session.

4. Responses from a predetermined number of pacifying session histories. The predetermined number may be in the range from 0 to 100, in the range from 5 to 20, or the last 10 pacifying sessions.

5. Sensor data for a predetermined time prior to the current time. The predetermined time may be in the range of from 0 hours to 2 hours, from 1 minute to 20 minutes, or the previous 5 minutes of data. Typically, the infant-specific parameters derived from the output sensor data may include any combination of the following:

1. Respiration rate

2. Heart rate of heart

3. Temperature (temperature)

4. Exercise diagram

5. Audio waveform, or

6. Crying and crying state

C. a sound waveform is generated from ICDV representation 655 by:

a. Based on the sensor data and the IPH data, the basic waveform that will have the best pacifying effect is selected.

B. The base waveform is compared to the pacifying waveform in the ICH data and small changes to the waveform are performed according to the waveform history in the ICH, if necessary.

D. The waveform generated in step C is sent to a speaker 656 which can play the waveform to the baby, for example, if the baby has not fallen asleep after a predetermined time (in which case a reminder may be sent to the user) or if a manual "stop" command is issued.

E. The response of the infant was measured (repeat steps a and B).

F. if the infant is in a steady sleep state, the process is stopped.

G. if the infant is approaching a steady sleep state but has not reached that state, small changes are made to waveform 657 based on changes in sensor data and how the changes in sensor data affect the Infant (IPH) and other Infants (ICH) in the plurality of infants.

H. Steps D to G are performed iteratively until the process stops.

It should be noted that the above process is exemplary and not limiting, such that steps may be performed in a different order and/or more or fewer steps may be used.

Feeding/bottle preparation

In some embodiments, the behavioral patterns and parameters of the infant's feeding routine may be determined from the infant's breathing and sleep patterns, heart rate, and other factors. These may be determined from monitoring data communicated to the processing unit and data provided by the user via the user interface 486. In the processing unit, the system (implementing machine learning) can learn how to detect feeding time, and alert the user that feeding time is reached, or independently activate the heating feature of the temperature controller 485 to prepare the feeding bottle to feed the infant.

Pacifying/sleeping support

In some embodiments, once the system detects that the infant is about to wake up or crying via sound detection analysis by detecting an increased respiration rate 482d, altered vital signs 482b and 482c, or a pre-learned signal, the system may simulate the sound that the infant hears in the uterus to pacify the infant and calm it and/or re-sleep the infant, thereby reducing the need for the parent to wake up at night. The pacifying system may be personalized to improve the effectiveness of pacifying sounds to infants. The pacifying sounds may include, for example, tones, rhythms, sound effects, and the like. The speaker system may be connected to the monitor or may be a separate personalized pacifier. The speaker system may be located near the monitor or may be remote from the monitor but close to the infant for daytime use. In addition to speakers, projectors may also increase the visual dimension of the pacifying system, where the collected personalized data may display the image that is most pacifying to the infant. This may be provided manually or automatically. Both the speaker and projector may be charged when connected to the monitor or charged separately (e.g., the speaker and/or projector may include a battery).

In some embodiments, vital signs may include an average heart rate, a maximum heart rate, a minimum heart rate, a heart rate inconsistency, an indication that the heart has stopped, an average respiration rate, a maximum respiration rate, a minimum respiration rate, a respiration rate inconsistency, an indication that respiration has stopped, a rate of change in heart rate greater than a predetermined amount, a rate of change in respiration rate greater than a predetermined amount, or any combination thereof. An inconsistent heart rate may be the case, for example, where the heart periodically misses a beat or where the heart beats at one rate and then switches to a second rate over a period of time. An inconsistent breathing rate may be the case, for example, where there is a short period of apnea or where the infant breathes at one rate for a period of time and then switches to a second breathing rate.

In some embodiments, the reactive sign may include data from vital signs, movement of the eye, movement of a body part, sound emitted by the infant, reaction to a scheduled event, reaction to a pacifying method, change in behavior over time, or any combination thereof. The sounds made by the infant may include crying, lanking, pterygoid, speaking, breathing, screaming, or any combination thereof. The response to the pacifying method may include lying down, closing the eyes, blinking, stopping crying, entering a predetermined sleeping position, or any combination thereof. The predetermined sleeping pose may be determined from a machine learning algorithm trained from previous event data while the infant is asleep.

Education/communication

In some embodiments, the system may allow the infant to communicate with a remote home and develop educational skills such as, for example, understanding shape, animals, objects, etc., using a 2D or 3D (e.g., hologram) projector 483 and based on a machine learning model trained to evaluate the infant's attention and interests as a function of the infant's eye movements, body movements and body position, type of eye movements (voluntary or involuntary), vital signs (such as, but not limited to, heart rate variation, respiration rate variation, and temperature), reactions (such as, but not limited to, startle reactions, reactions to discomfort or pain, directing vision toward someone or something, moving vision away from someone or something, happy reactions, and any combination thereof), and developmental stage.

In some embodiments, one type of eye movement may be, for example, saccadic, smooth tracking, eye steering, and/or vestibular eye movement.

In some embodiments, projector 483, monitor 482, and speakers (acoustic audio detector/microphone 484 a) may also be configured to function as an active training system. Based on the received vital signs, responses, video (motion and eye motion detection) and audio information, machine learning models and algorithms can be used to collect and analyze data to personalize different training programs and provide feedback to the user (parent or caretaker) regarding the ongoing development of the infant.

Non-contact heart rate and blood oxygen measurement

In some embodiments, blood oxygen levels and heart rate may be determined by photoplethysmography (PPG), which uses light and optics to measure properties of blood. Hemoglobin may absorb varying amounts of light as blood enters and leaves capillaries of the skin. The processor may use the amount of change in reflected light from the infant's skin to determine blood oxygen levels and use the period of change in hemoglobin levels to measure the rate at which the heart pumps blood. The reflection pattern PPG can be measured over a large distance. The basic optical detection device for this type of measurement may be a simple webcam or an imaging device, such as a camera on a mobile device. The light source may be ambient light in a room.

In some embodiments, the heart rate may be determined from skin pixels of image data of the infant's face. In other embodiments, PPG measurements from other parts of the body may be used. In still other embodiments, a gradient descent algorithm with machine learning may be used to find heart rate.

In some embodiments, the heart rate determination algorithm may first distinguish between skin pixels and non-skin pixels, which may be, for example, hair, eyes, clothing, and accessories, or background objects such as, for example, bedding, floors, walls, doors, or furniture. The output from the tracker may be analyzed and the background signal may be estimated.

In some embodiments, in the next processing stage, the signal may pass through a zero-phase butterworth band-pass filter having a band-pass of 40 to 180 times/min. A fourier transform may be applied to the filtered signal to determine the heart rate. Typically, this process may be repeated about once per second. The determined frequency may be in the range of once every 0.1s to once every 20 s. In other embodiments, the state machine may suspend operation of the algorithm when motion occurs that may not be adequately compensated by the tracker or background signal removal, and then the state machine may resume operation when the undesired motion has stopped.

In some embodiments, for example, when the system 480 can detect a dangerous condition (such as a respiratory abnormality), the system can determine the severity of the condition and can alert the user 45 via the GUI 55 accordingly, for example, transmitting an alert triggered by a severity level decision when the system evaluates. The image processing technique can enable accurate and timely detection of heart rate and respiration from the image. These detections may enable a strong monitoring and alerting in infant sleep situations. Negative false positives can be kept as low as possible while positive false positives (false alarms) are avoided.

In some embodiments, the system may detect the posture of the infant.

In some embodiments, the system may alert the user if the respiration or heartbeat rate is outside of a predetermined range.

In some implementations, the camera may include additional features such as night vision, storage of video and photographs, and digital zoom. The night vision feature may allow a user (such as a caretaker) to obtain a clear view of the infant even in the dark, which may be seen in the area 57 of the GUI 55 of the mobile device 50. Digital zoom may allow close-up viewing of the infant and its vicinity in region 57 in GUI 55.

In some embodiments, the system may further include a thermal camera capable of detecting whether the infant's head is covered and sending an alert to the user's mobile device 50 (e.g., a caretaker mobile device).

In some embodiments, the system may further comprise sensors for monitoring the infant and the environment of the infant. These sensors may include thermal sensors, motion sensors, and acoustic sensors. The data acquired from these sensors may be transferred to the user's mobile device 50 (e.g., a caregiver mobile device).

Fig. 8A illustrates a second embodiment of an infant monitoring and interaction (BMID) device 1000 in accordance with one or more embodiments of the present disclosure. BMID 1000 may include a monitor 1010, a smart object 1020, and a stand 1030 for mounting the device. The monitor may include an optical subsystem, a bi-directional speaker or bi-directional audio system (which may include a microphone and an acoustic/audio detector, or any other bi-directional audio communication system), a projector, a microphone, an acoustic/audio detector, a respiratory detector, a temperature detector, a heart rate detector, a user interface, a processing unit, an electromagnetic detector, or any combination thereof.

Fig. 8B illustrates a third embodiment of an infant monitoring and interaction device (BMID) 2000 in accordance with one or more embodiments of the present disclosure. BMID 2000 may include a camera/projector 2020, at least one sensor 2030, and a microphone/speaker 2040. The support for the device is not shown.

Fig. 9A is a flowchart 2200 of a method for educating an infant in accordance with one or more embodiments of the present disclosure. The user may select a course 2201 from the suggested courses. This course may also be referred to herein as an infant-specific educational program. The appropriate course may be selected from possible courses provided by the system via GUI 55 of mobile device 50 associated with the user. The selection of a course may be based on the age of the infant and the cognitive ability as estimated by the user (e.g., caretaker). Based on the selected course, the user may attach an associated smart object (e.g., smart object 1020) to an arm of BMID device 2202. Courses may be selected based on the age of the infant. Thus, for example, black and white objects may be selected for newborns, and colored objects may be selected for larger infants.

In some embodiments, examples of infant-specific educational programs or courses may include projecting specific visual images to an infant in different categories, such as different animals (dogs, cats, rabbits 25), different figures of the person (pictures of mother, father, siblings, friends, etc.), different objects (cars, microwave ovens, cake blenders, etc.), natural events (e.g., wind, water, and fire), and projecting visual images along with playing sounds associated with the projected visual objects. Then, the baby may be caused to view the two projected visual images, i.e., the first visual image and the second visual image, while the sound associated with the first image is played from the speaker. The system 5 may evaluate the correlation between the infant understanding sound and the first visual image by evaluating image data of eye movements and/or head movements of the infant towards the first visual image. Similarly, sound associated with the second image may be played.

In some embodiments, after the infant experiences each visual image and associated sound, the system 5 may project the visual image of the car and determine whether the infant sounds the car to determine whether the infant understands.

In some embodiments, an AI model (e.g., an infant-specific educational machine learning model) may output an indication or notification on GUI 55 that an infant understands sounds associated with the projected visual image by tracking eye and/or head movements of the infant. The system 5 may make a recommendation to the user 45 on the GUI 55 to continue the current course or to modify the course to a higher level stage. If the system evaluates that the infant is not understood, the infant-specific educational machine learning model may output a recommendation to continue the current course to the user 45 on the GUI 55.

In some embodiments, the infant-specific educational machine learning model may automatically modify an infant-specific educational program or course. For example, the modification may include the AI model (e.g., an infant-specific educational machine learning model) outputting an image-sound pair that the infant understands to be presented to the infant and may change the category of the image (e.g., change from animal to household) and/or the modification may be to insert video instead of a static visual image, which may be to speak words associated with the projected images to see if the infant understands the word "cat" when the system 5 projects images of cats and dogs.

In some embodiments, smart objects 1020 to be attached to a device may include objects of different colors, shapes, animals, and other suitably shaped objects. In other embodiments, smart object 1020 may include a processor for processing data, storing data, and communicating. The attached smart object may then be identified 2203 by BMID device 1000 or by an application on the user's mobile device 50 (e.g., the caretaker's mobile device). Via an application on the mobile device, the user can select 2204 one of the attached objects to introduce to the infant. Based on the selection, the device may send 2205 visual and audible signals to clearly identify the subject to the infant. The visual signal may include movement of the object and/or lighting the object. The audio signal may include sound (e.g., animal sound of an animal) and a speech signal associated with the subject, thereby identifying the subject to the infant. The system may repeat 2206 the visual and/or audible signal to the infant at least once. After the learning course, the infant is tested to evaluate the infant's learning condition.

Fig. 9B is a flowchart 2300 of a method for testing an infant to assess infant learning in accordance with one or more embodiments of the present disclosure. A first step may include activating a sound signal by the device 2301. No visual signal may be provided. The infant may be captured by the camera of the device 2302 in response to movement of the audio signal. The obtained data may then be analyzed by video analysis using the monitoring software 2303. The monitoring software may use machine learning to evaluate the mental and cognitive development of the infant 2304. The comparison data for analysis may be obtained from a remote platform. Based on this analysis, the rating-based custom lesson 2305, which may be sent to the user's mobile device 50, may be adjusted to suit the infant's ability.

In some embodiments, the system may include a projector for projecting an image onto a surface remote from the infant. The projector may include a two-way video system having a microphone and a speaker for audio communication with the infant. Using the cameras and video analysis software described above, the ability of the infant to recognize images can be evaluated in order to provide courses tailored to the infant's developmental ability (e.g., modifying an infant-specific educational program based on machine learning evaluation of the infant's cognitive development).

In some embodiments, the system may include respiration monitoring functionality. The respiratory monitor may include computer vision technology, motion sensors, sound sensors, or any combination thereof for detecting movement or sound made by the infant. If the movement of the infant is not detected within a predetermined period of time, a reminder will be sent to the user. The respiration monitor may include radar or sonar components for detecting movement of the infant. The radar or sonar system may record heart rate and respiration rate and may send a reminder to the user if the respiration or heartbeat rate is outside a predetermined range. The radar system may be a Continuous Wave (CW) radar, a roi radar, a monopulse radar, or a passive radar. A radar or sonar detection system may be provided to provide a second alarm when motion cessation or respiratory rate reduction is detected. The purpose of the second alarm is to avoid false alarms, as computerized vision systems and infant breathing monitors may often be prone to false alarms.

Heart rate of heart

Fig. 10A is a diagram illustrating a heart rate estimation system 3100 according to one or more embodiments of the present disclosure. The heart rate estimation system 3100 shown herein may be used, for example, to determine the heart rate of infant-specific features 225 as used in the algorithm flow diagram 200 of a computer-based system for monitoring and interacting with infants or to determine the heart rate required for any other infant monitoring and interaction process as disclosed herein. System 3100 can include one or more electronic devices 3105A, 3105B and one or more bodies 3115. The electronic devices 3105A, 3105B may be mobile phones, tablet computers, laptop computers, computer workstations, cameras, and the like. The electronic devices 3105A, 3105B may use cameras to capture 3110 video clips of video data of the body 3115.

In some implementations, video data may be captured by one or more of a variety of cameras (e.g., a 3-color channel camera, a multispectral n-channel camera, an infrared camera, a depth camera, a 1-pixel sensor, a servo-controlled camera, or any combination thereof). The 3-color channel camera may be a red/green/blue (RGB) 3-color channel camera. For simplicity, the electronic devices 3105A, 3105B and cameras may be referred to or referred to in the singular hereinafter as imaging devices or imaging cameras, but any number of electronic devices 3105 and cameras may be used.

In the past, it was impractical to calculate heart rate from video data of the body 3115 due to movement of the electronic device 3105, movement of the body 3115, and/or changes in illumination. Implementations described herein may generate a super-pixel model from video data and may calculate heartbeat signals and heart characteristics, as will be described below. Accordingly, the electronic devices 3105A, 3105B may accurately estimate the heart rate of the subject 3115.

In some embodiments, the body 3115 may be a human or animal. The heart rate of the one or more subjects 3115 may be estimated from the video data. Video data may be captured 3110 from a face or other body part of the subject 3115. Video data may be captured 3110 from one or more of reflected natural light, reflected electrical illumination in the environment, reflected illumination provided by the system through a laser or infrared Light Emitting Diode (LED), and/or long wave thermal infrared emissions, for example.

In some embodiments, the video data may be a motion stable region of interest (ROI). The ROI may be any exposed portion of skin, such as at least a portion of the forehead of the body 3115, at least a portion of the neck of the body 3115, at least a portion of the arms of the body 3115, at least a portion of the back of the body 3115, at least a portion of the chest of the body 3115, at least a portion of the abdomen of the body 3115, at least a portion of the buttocks of the body 3115, at least a portion of the legs of the body 3115, at least a portion of the hands of the body 3115, at least a portion of the feet of the body 3115, or any combination thereof. In other implementations, the pixels may include ROIs to form a continuous portion of the body 3115.

Fig. 10B is a diagram illustrating an object of interest (OOI) 3285 and a region of interest (ROI) 3250 on a subject 3115 according to one or more embodiments of the present disclosure. In some implementations, the electronic device 3105 may receive video data and detect OOI 3285 from the video data. The electronic device 3105 may detect the face, a portion of the face (such as the forehead, neck, arms, and/or other body parts) as OOI 3285.

In some embodiments, the electronic device 3105 may also detect and/or track the OOI 3285. In other implementations, the OOI 3285 may be detected using cascaded object detection of RGB pixels of video data. OOI 3285 can also be tracked at sub-pixel resolution using a spatial correlation based method. Alternatively, the OOI 3285 may be detected and tracked using infrared band information. For example, the forehead OOI 3285 of the subject 3115 may be identified from an infrared hotspot. The OOI 3285 may also be detected and tracked using multispectral information.

In some implementations, the facial markers may be used to track the OOI 3285 from RGB pixels of the video data. For example, the electronic device 3105 may identify the eyes and mouth of the subject 3115 from the RGB pixels and may detect the OOI 3285 relative to the eyes and mouth, typically via AI, but other optical analysis methods may also be used. Alternatively, the OOI 3285 may be tracked from RGB pixels of the video data using a spatial correlation filter.

In some implementations, the OOI 3285 may be detected and tracked using information from a depth camera. For example, the depth camera electronic device 3105 may recognize the outline of the body 3115 and may detect the face OOI 3285 from the outline.

In some implementations, the ROI 3250 may be identified within the OOI 3285. ROI 3250 may be a designated region within OOI 3285. For example, ROI 3250 may be the forehead or cheek of head OOI 3285. In some implementations, the OOI 3285 and/or the ROI 3250 can be identified from the image segmentation. For example, the electronic device 3105 may segment the video data into a plurality of image segments and may identify the OOI 3285 and/or the ROI 3250 from the image segments.

In some implementations, the OOI 3285 and/or ROI 3250 can be detected using a bounding box. The bounding box may include a luminance component, a blue differential chromaticity, a red differential chromaticity (YCbCr) color space. For example, OOI 3285 and/or ROI 3250 may be identified as a region bounded by a YCbCr bounding box. In some embodiments, the electronic device 3115 can detect and track one or more OOIs 3285 and can detect and track one or more ROIs 3250 within each OOI 3285.

Fig. 10C is a schematic block diagram illustrating video data 3120 in accordance with one or more embodiments of the present disclosure. The video data 3120 may include a plurality of pixels 3225 of the temporal sequence 3125. The pixels 3225 of the time series 3125 may form an image. The video data 3120 may organize data structures in memory. The time series 3125 may be sequential. The time series 3125 may be randomly sampled from the video data. The data structure may be a random sample of the image or any combination thereof. The pixels 3225 may be RGB, YCbCr, any other means of determining color, thermal pixels, UV pixels, or any combination thereof.

Fig. 10D is a schematic block diagram illustrating a super pixel time series 3195 in accordance with one or more embodiments of the present disclosure. The superpixel temporal sequence 3195 may be organized as a data structure in memory. In the depicted embodiment, pixel group 3225 as shown in FIG. 10C may be organized into super-pixels 3240. The generation of superpixel 3240 is described below in fig. 14. A plurality of temporal sequences 3125 may be generated from each super-pixel 3240 of the video data 3120.

Fig. 11A is a schematic block diagram illustrating video data 3120 in accordance with one or more embodiments of the present disclosure. The video data 3120 may be organized as data structures in memory. In the depicted implementation, the video data 3120 may include a plurality of pixel data 3205. The pixel data 3205 may be organized into an array and may store brightness data, contrast data, color data, and the like. Additionally, each instance of pixel data 3205 may include a pixel identifier. The pixel identifier may be a memory address, a matrix index, etc.

Fig. 11B is a schematic block diagram illustrating data 3440 in accordance with one or more embodiments of the present disclosure. OOI data 3440 may be organized as data structures in memory. OOI data 3440 may describe OOI 285. In the depicted implementation, the OOI data 3440 may include an OOI identifier 3430 and/or a plurality of pixel identifiers 3435.OOI identifier 3430 may uniquely identify OOI 3285. Pixel identifier 3435 may reference pixel data 3205 that may include pixels 3225 of OOI 3285.

Fig. 11C is a schematic block diagram illustrating ROI data 3425 in accordance with one or more embodiments of the present disclosure. The ROI data 3425 may be organized as a data structure in memory. ROI data 3425 may describe ROI 3250. In the depicted embodiment, the ROI data 3425 may include an ROI identifier 3445 and a plurality of pixel identifiers 3435.ROI identifier 3445 may uniquely identify ROI 3250. Pixel identifier 3435 may reference pixel data 3205 that may include pixels 3225 of ROI 3250.

Fig. 11D is a schematic block diagram illustrating superpixel data 3255, in accordance with one or more embodiments of the present disclosure. Superpixel data 3255 may describe superpixel 3240. In the depicted embodiment, the superpixel data 3255 may include a superpixel identifier 3215, a time series identifier 3220, measured pixel values 3265, and a plurality of pixel identifiers 3435.

In some implementations, the superpixel data 3255 may be organized as a data structure in memory, and may include pixels from any sensor data, as described herein.

In some implementations, the superpixel identifier 3215 may uniquely identify the superpixel 3240. Time sequence identifier 3220 may identify time sequence 3125 of superpixel 3240. In some embodiments, time sequence identifier 3220 indicates a position in the sequence. Alternatively, the time series identifier 3220 may indicate absolute and/or relative time. Pixel identifier 3435 may reference pixel data 3205 that may include pixels 3225 of superpixel 3240.

In some embodiments, the measured pixel values 3265 may include one or more values representing an average of pixels in the ROI 3250. These values may be one or more color values, such as RGB values. In addition, these values may include brightness values, contrast values, and the like.

Fig. 11E is a schematic block diagram illustrating a superpixel model 3270 in accordance with one or more embodiments of the present disclosure. The superpixel model 3270 may be organized as a data structure in memory. In the depicted embodiment, model 3270 may include a super-pixel identifier 3215, a time-series identifier 3220, measured pixel values 3265, a background signal 3460, a heartbeat signal 3465, a respiration rate signal 3467, and a sensor noise signal 3470. The sensor signals that may be included are signals that are the result of machine learning, such as, but not limited to, a class probability map, a feature map, or any combination thereof. Other sensor signals may also be included in the superpixel model, as described herein. The superpixel model may also include signals that are the result of machine learning, such as, but not limited to, class probability maps, feature maps, heat maps, or any combination thereof.

In some implementations, the superpixel identifier 3215 may identify one or more superpixels 3240 represented by the model 3270. The time series identifier 3220 may identify one or more time series t 3125 represented by the model 3270. For example, the time series identifier 3220 may identify 48 time series 3125 captured during a two second video clip. The measured pixel value y _i (t) 3265 may include the pixel value of each pixel 3225i in each time series t 3125. For each pixel 3225i in each time series t 3125, the background signal u _i (t) 3460 may estimate the contribution to the measured pixel value 3265 due to motion and illumination changes captured by the electronic device 3105.

In some embodiments, for each pixel 3225i in each time series t 3125, the heartbeat signal h _i (t) 3465 can estimate the contribution of the measured pixel value 3265 due to the heartbeat. For each pixel i 3225 in each time series t 3125, the sensor noise signal n _i (t) 3470 may estimate the contribution to the measured pixel value 3265 due to the sensor noise in the electronic device 3105. Accordingly, the super pixel model 3270 of the time series t 3125 may be modeled using equation 1.

1. Y _i(t)＝u _i(t)+h _i(t)+n _i (t) equation 1

In some embodiments, it may be assumed that the sensor noise signal 3470 is independent, identically distributed gaussian noise. In addition, it may be assumed that the background signal 3460 is smooth. For example, it may be assumed that the variance of the background signal 3460 between the time series 3125 is less than a background threshold. In some embodiments, the background signal 3460 may be modeled as a first order markov random process. Background signal 3460 may be modeled using an autoregressive model of a first order markov random process. In some embodiments, the heartbeat signal 3465 may be assumed to be the same in each super pixel 3240. For example, h _i (t) =h (t) may be assumed to be true for all i.

Fig. 12 is a flow diagram illustrating a heart rate estimation process 3101 in accordance with one or more embodiments of the present disclosure. Process 3101 may be performed by an electronic device 3105. Process 3101 is described in more detail in fig. 14. In the depicted embodiment, OOI module 3320, ROI module 3325, superpixel calculator 3330, preprocessor 3335, modeler 3340, optimizer 3345, and/or heart rate detector 3350 may perform process 3101.OOI module 3320, ROI module 3325, superpixel calculator 3330, preprocessor 3335, modeler 3340, optimizer 3345, heart rate detector 3350, respiratory rate detector 3355, and/or any other detector may be implemented in semiconductor hardware and/or code executed by a processor.

In some implementations, the OOI module 3320 may receive video data 3120 from a camera of the electronic device 3105 and detect OOI 3285.OOI module 3320 may use a camera to track OOI 3285 and generate OOI data 3440 describing OOI 3285. ROI module 3325 may receive OOI data 3440 and identify ROI 3250 within OOI 3285. ROI module 3325 may generate ROI data 3425 describing ROI 3250.

In some implementations, the superpixel calculator 3330 may receive the ROI data 3425 and generate superpixels 3240 in the superpixel time sequence 3195. The preprocessor 3335 may preprocess the superpixel time series 3195 to remove interfering signals from the superpixel time series 3195 and generate a preprocessed superpixel time series 3290.

In some implementations, modeler 3340 may generate superpixel model 3270 from superpixel time series 3195 and/or pre-processed superpixel time series 3290. The optimizer 3345 may calculate the heartbeat signal 3255 according to the superpixel model 3270. In some embodiments, the optimizer 3345 may calculate the heartbeat signal 3465 from the superpixel model 3270 and the pre-processed superpixel time sequence 3290. The heart rate detector 3350 may calculate heart characteristics such as, but not limited to, heart rate 3480, inter-beat intervals 3475, and/or heart rate variability 3490 from the heart beat signal 3465. The respiratory rate detector 3355 may calculate respiratory characteristics from the respiratory signal such as, but not limited to, respiratory rate 3367, respiratory rate variability, and any combination thereof.

Fig. 13 is a schematic block diagram illustrating a computer 3400 in accordance with one or more embodiments of the present disclosure. The computer 3400 may be embodied in an electronic device 3105. The computer 3400 may include a processor 3405, memory 3410, and communication hardware 3415. The memory 3410 may be a computer readable storage medium such as a semiconductor memory device, a hard disk drive, a holographic memory device, a micromechanical memory device, or a combination thereof. The memory 3410 may store codes. The processor 3405 may execute code. The communication hardware 3415 may communicate with other devices.

Fig. 14 is a flow diagram of a method 3500 of heart characteristic estimation in accordance with one or more embodiments of the present disclosure. Method 3500 may remotely estimate heart characteristics such as, but not limited to, heart rate, inter-beat intervals, and heart rate variability. Method 3500 may be performed by processor 3405 and/or OOI module 3320, ROI module 3325, superpixel calculator 3330, preprocessor 3335, modeler 3340, optimizer 3345, and heart rate detector 3350 in electronic device 3105.

In some embodiments, the method 3500 may begin in some embodiments when the electronic device 3105 receives 3505 video data 3120 from a camera of the electronic device. In some implementations, the video data 3120 may be received as one or more time sequences 3125 of pixels 3225.

In some implementations, the electronic device 3105 may also detect 3510 OOI 3285 in each image of the video data 3120. The image may include a temporal sequence 3125 of pixels 3225.OOI 3285 may be a body and/or body part of body 3115 such as a head, neck, and arms, legs, etc. In some implementations, the OOI 3285 may be detected using cascaded object detection of RGB pixels of the video data 3120.

In some implementations, the electronic device 3105 may also track 3515 the OOI 3285 in each image of the video data 3120. In some implementations, the OOI 3285 can be tracked using infrared band information from an infrared camera and/or a multispectral camera. The electronic device 3105 may generate OOI data 3440 representing OOI 3285

In some embodiments, the electronic device 3105 may identify 3520ooi 3285 one or more ROIs 3250.ROI 3250 may be a region of a body part, such as the forehead, wrist, etc. In some implementations, the ROI 3250 is identified using image segmentation. The electronic device 3105 can generate ROI data 3425 representing the ROI 3250.

In some implementations, the electronic device 3105 can generate 3525 superpixels 3240 in each ROI 3250 from the video data 3120 and the ROI data 3425. In some implementations, each super pixel 3240 can include a specified number of pixels 3225. Alternatively, each superpixel 3240 may be formed by an adjacent pixel 3225 having a measured pixel value 3265 that is within a range of values.

In some embodiments, the electronic device 3105 may also generate 3530 a temporal sequence 3195 of superpixels of the plurality of superpixels 3240 in each image of the video data 3120. In some implementations, one or more sequential superpixels 3240 may be cascaded to form a superpixel temporal sequence 3195. Alternatively, one or more non-sequential superpixels 3240 may be selected and concatenated to form a superpixel temporal sequence 3195.

In some embodiments, the electronic device 3105 may remove 3535 the interfering signal from the super-pixel time series 3195. The removal of the interfering signal may be a preprocessing. In some embodiments, a detrack may be used to remove 3535 the interfering signal. Trending may be performed by modeling background signal 3460 as a gaussian process. Alternatively, trending may be performed by decorrelating the super-pixel time series 3195 with auxiliary signals derived from the position of the face frame defining the face of the subject 3115 and from other regions in the video data 3120. In some embodiments, removing 3535 the interference signal from the superpixel temporal sequence 3195 may include bandpass filtering to remove signals that are out of the frequency band of the normal heart rate. For example, signals having a frequency below 40 times per minute (bpm) and above 170bpm may be filtered out of the super pixel time series 3195.

In some embodiments, the electronic device 3105 may model 3540 the superpixel temporal sequence 3195 as a superpixel model 3270. In some embodiments, the superpixel temporal sequence 3195 is modeled in the form of equation 1.

In some embodiments, the electronic device 3105 may calculate 3545 the heartbeat signal 3465 using the super pixel model 3270. In some embodiments, the 3545 heartbeat signal 3465 and the background signal 3460 may be calculated by optimizing equation 2 subject to equations 3 and 4. In some embodiments, in time series 3125, the sum of i may be over a plurality of superpixels 3240, and the sum of T may be over a plurality of superpixels 3240, λ ₁ and λ ₂ may be user parameters, H may be a (m+1) × (2l+1) tepriz matrix with the (i, j) th element H (2l+i-j), where i=1, 2, … …, m+1 and j=1, 2, … …, 2l+1, H is a (m+1) x 1 vector with the i-th element H (l+t+i-1), where i=1, 2, … …, m+1, and · _* may be a kernel number.

[H] _(i,j) = h (2l+i-j equation 3

[H] _j = h (l+t+i-1) equation 4

In some embodiments, alternatively, the 3545 heartbeat signal 3465 and the background signal 3460 may be calculated by optimizing equation 5, where D is given by equation 6 and P is given by equation 7, and α and β are user selectable constants that may generate smoothness of the background signal 3460 and/or predictability of the heartbeat signal 3465. Vector u may comprise samples of the background signal and vector h may comprise samples of the heartbeat signal. The prediction coefficients P _L, P-L may be interpolation coefficients derived from the hypothetical period of the heartbeat signal 3465, and the position of-1 in the P matrix may also depend on the hypothetical period of the heartbeat signal 3465. The optimization may be repeated for a series of different heart beat cycles and a first heart beat cycle, giving a minimum target value that may be selected as a cycle of the heart beat signal 3465.

In some embodiments, the electronic device 3105 may calculate 3550 the heartbeat characteristic from the heartbeat signal 3465 and the method 3500 ends. The beat characteristics may include heart rate 3480, inter-beat intervals 3475, and/or heart rate variability 3490. The electronic device 3105 may calculate 3550 the heart rate 3480 using one or more of a machine learning analysis of the heart beat signal 3465, a peak of a fourier transform of the heart beat signal 3465, a power spectral density of the heart beat signal 3465, a zero crossing rate of the heart beat signal 3465, and/or a sliding correlation analysis of the heart beat signal 3465.

Implementations disclosed herein may detect OOI 3285 from video data, may track OOI 3285, and may identify ROI 3250 in OOI 3285. Embodiments disclosed herein may also generate superpixel 3240 from pixel 3225 within ROI 3250. Additionally, embodiments herein may be used to generate the superpixel temporal sequence 3195 and model the superpixel temporal sequence 3195 as the superpixel model 3270. The super pixel model 3270 may be used to calculate the heartbeat signal 3265 and other cardiac characteristics. Thus, embodiments may remotely estimate heart rate 3480 of one or more subjects 3115. Embodiments may allow, for example, remote estimation of an animal's heart rate 3480, estimation of the subject's 3115 heart rate 3480 in the event that the human subject 3115 may be active, and rapid determination of heart rate 3480. Thus, the embodiments described herein may provide an actual and efficient way for remotely estimating heart rate 3480.

Respiration rate

Fig. 15A is a schematic block diagram of a respiratory event recognition system 4100 according to one or more embodiments of the disclosure. The respiratory event recognition system 3100 shown herein may be used, for example, for determining respiratory detection of infant specific features 225 as used in the algorithm flow diagram 200 of a computer-based system for monitoring and interacting with infants or for detecting infant respiratory rate required for any of the other infant monitoring and interaction processes as disclosed herein. The system 4100 may identify respiratory events and/or wide range of motion of the subject 4110 from the video stream of the subject 4110 captured by the camera 4105. In addition, if a respiratory event is not identified and a wide range of motion is not identified, the system 4100 may generate a reminder. In some embodiments, the system 4100 may also include a monitoring device 4123. The monitoring device 4123 may include a camera 4105, a microphone 4115, a hardware module 4116, and/or a motion detector 4117. In addition, the monitoring device 4123 may include a display 4119 and/or a speaker 4121. In some implementations, the hardware module 4116 can include dedicated semiconductor circuits. The dedicated semiconductor circuit may include a memory. In addition, the hardware module 4116 may include a computer.

In some implementations, the camera 4105 can capture a video stream 4120 of the subject 4110. The camera 4105 may employ a bandpass filter in the range of 0.8 microns to 2.5 microns. In addition, the camera 4105 may employ a Charge Coupled Device (CCD) tuned to 1.5 microns. The camera 4105 may capture the video stream as infrared image frames.

In some embodiments, when the subject 4110 may be at risk of stopping breathing, the subject 4110 may be monitored to identify respiratory rate and/or detect respiratory stopping so that timely assistance may be given. Further, non-invasive monitoring may be used to identify respiratory events, so the body 4110 is not disturbed.

In some embodiments, respiratory events may be detected optically and audibly. For example, a baby monitor may be used to monitor the baby's breath by video (image sequence) of the baby captured by camera 4105 and/or sound of the baby's breath captured by microphone 4115. Unfortunately, the impact of positive and negative false positives may be significant in identifying respiratory events, so monitoring may require extremely accurate detection of respiratory events.

Implementations described herein identify respiratory events and/or wide range of motion based on video streams, as will be described below. Embodiments also generate reminders, presentation displays, and presentation statistics based on respiratory events.

Fig. 15B is a schematic block diagram of a breath report 4165 according to one or more embodiments of the present disclosure. The breath report 4165 may be organized as a data structure in memory. In the depicted embodiment, the breath report 4165 may include a respiration rate 4231, a maximum inter-breath interval 4233, a minimum inter-breath interval 4235, inter-breath interval statistics 4237, an inter-breath interval histogram 4239, and/or apnea event data 4241.

In some embodiments, respiration rate 4231 may represent the respiration rate. The maximum inter-breath interval 4233 may specify a longest interval between respiratory events. The minimum inter-breath interval 4235 may specify the shortest interval between respiratory events. Inter-breath interval statistics 4237 may specify one or more of a mean, average, and pattern of intervals between respiratory events. Inter-breath interval histogram 4239 may describe the relative frequency of breath intervals between respiratory events. The breathing interval may be organized into one or more ranges.

In some embodiments, the apnea event data 4241 may be calculated from a breathing rate 4231, a maximum inter-breath interval 4233, a minimum inter-breath interval 4235, inter-breath interval statistics 4237, and an inter-breath interval histogram 4239. The apnea event data 4241 may be used to identify sleep apnea events.

Fig. 15C is a schematic block diagram of a motion report 4160 according to one or more embodiments of the present disclosure. The motion report 4160 may be organized as a data structure in memory. In the depicted embodiment, the motion report 4160 may include a motion frequency 4243, a motion amplitude 4245, a motion duration 4247, a sleep length 4249, a sleep quality 4251, and a sleep interval 4253.

In some implementations, the movement frequency 4243 may describe the frequency of the wide range of movement of the body 4110. The motion amplitude 4245 may describe the number of pixels affected by each motion. The movement duration 4247 may describe the duration of the body 4110 from the beginning to the end of each wide range of movement.

In some embodiments, the sleep length 4249 may describe the length of the time interval during which the subject 4110 sleeps. Sleep quality 4251 may estimate the restlessness of the sleep of the subject 4110. Sleep interval 4253 may describe each interval during which body 4110 sleeps.

Fig. 15D is a schematic block diagram of respiratory data according to one or more embodiments of the present disclosure. The breathing data may be organized as a data structure in memory. In the depicted embodiment, the respiratory data may include a field of view strategy 4261, an event time interval 4263, a respiratory event 4265, and a wide range of motion 4267.

In some implementations, the field of view policy 4261 may specify when the subject 4110 may be satisfactorily viewed by the camera 4105. The event time interval 4263 may specify a time interval during which respiratory events 4265 and/or wide range movements 4267 of the subject 4110 may be identified from generating a reminder. The respiratory event 4265 may be an identified breath of the subject 4110. The wide range of motion 4267 may be indicative of the motion of the body 4110. The movement may make it impossible to determine a respiratory event 4265.

Fig. 16A is a schematic diagram illustrating a region 4003 in an image frame 4005 in accordance with one or more embodiments of the present disclosure. In the depicted embodiment, the image frame 4005 is divided into a rectangular grid of one or more regions 4003. In some embodiments, all of the regions 4003 may have a rectangular shape and may be equal in size.

Fig. 16B is a schematic diagram illustrating a region 4003 in an image frame 4005 in accordance with one or more embodiments of the present disclosure. In the depicted embodiment, the image frame 4005 may be divided into circular areas 4003. The regions 4003 do not overlap and some pixels of the image frame 4005 are not included in any of the regions 4003.

Fig. 16C is a schematic diagram illustrating a region 4003 in an image frame 4005 in accordance with one or more embodiments of the present disclosure. In the depicted embodiment, the image frame 4005 may be divided into circular areas 4003. The regions 4003 may overlap such that some pixels of the image frame 4005 are included in two or more regions 4003, while other pixels are not included in any region 4003.

Fig. 16D is a schematic diagram illustrating a user selected region 4003 in an image frame 4005 in accordance with one or more embodiments of the present disclosure. Using the interface, a user may identify pixels in the image frame 4005 that are likely to be occupied by the subject 4110 with user input defining a user selection area 4217. When a user input is provided, the region 4003 can be obtained by dividing the user selection region 4217.

Fig. 16E is a schematic block diagram of a respiratory event identification state according to one or more embodiments of the present disclosure. The depicted embodiment may include a normal breathing state 4201, an alarm state 4203, and a motion state 4205 of the state machine. When normal breathing is detected by the monitoring device 4123 in the alert state 4203 or the motion state 4205, the state machine transitions to the normal breathing state 4201. Normal breathing may be indicated by respiratory events 4265 within event time interval 4263. When a wide range of motion 4267 is detected by the monitoring device 4123 in the alert state 4203 or normal breathing state 4201, the state machine transitions to the motion state 4205. In some embodiments, the wide range motion 4267 is detected within the event time interval 4263. If the monitoring device 4123 does not detect normal breathing and motion in either the normal breathing state 4201 or the motion state 4205, the state machine transitions to the alert state 4203. In some embodiments, it may be desirable to detect normal breathing and/or extensive movement 4267 within event time interval 4263, otherwise the state machine transitions to alarm state 4203. The use of respiratory event recognition states is described in more detail in fig. 17A.

Fig. 17A is a schematic block diagram of a respiratory event identification process 4103 according to one or more embodiments of the disclosure. The video stream 4120 of the subject may include image frames 4005 that may be captured by the camera 4105. The camera 4105 may output an image frame 4005. The image frame 4005 may be produced in many different formats including color images with red, green and blue channels, gray scale images, infrared images, depth images. Some formats may be derived from other formats. For example, the grayscale image frame 4005 may be calculated from red, green, and/or blue images.

In some embodiments, a series of operations may be applied to extract the respiratory signal 4183. The order in which the operations may be performed may vary. Region sum operation 4170 may calculate region sum 4002. Region sum operation 4170 may add all pixel intensities located in region 4003 of image frame 4005 together. The region 4003 may have different shapes. For example, the image frame 4005 may be divided into a rectangular grid of regions, as shown in fig. 16A. The region may be circular as shown in fig. 16B to 16D. The region may be a vertical line or a horizontal line across the image. Other contours than straight lines may also be used. The regions may overlap as shown in fig. 16C or may not overlap as shown in fig. 16B. Not every pixel in the image frame 4005 needs to be included in the area as shown in fig. 16B to 16C.

In some embodiments, the selection of the region 4003 may be guided by user input through a graphical user interface, as shown in fig. 16D. Once the region selection with user input is performed, the region 4003 may remain fixed during the remainder of the process. In some embodiments, the computing area sum 4002 may not be needed. In some implementations, subsequent operations may be performed directly on the pixel intensities.

In some embodiments, the output of the region and operation 4170 may be a vector time series 4006, where there is one element in the vector for each region. When the region sum 4002 is not calculated, the vector may include one element for each pixel in the image frame.

In the depicted embodiment, the detrening operation 4175 may be applied after computing the regions and 4002. However, the region and operation 4170 and the trending operation 4175 may be performed in the reverse order, as shown in fig. 17B. The trending operation 4175 may remove the signal mean in the region 4003. The trending operation 4175 may also normalize the amplitude of the signal of the region 4003. Since the region and operation 4170 may spatially combine pixels across one image frame 4005, the detrending operation 4175 may operate on the signal in the time dimension. The trending operation 4175 may not be required. In some embodiments, the untrending operation 4175 may not be performed. The dimension of the vector time series 4006 output by the untrending operation 4175 may be the same as its input to the untrending operation 4175.

In some embodiments, the respiratory signal operation 4185 may estimate the respiratory signal 4183. In some embodiments, sparse plus low rank decomposition may be used, as shown below in fig. 17C. In some implementations, the respiratory signal operation 4185 may use subspace tracking. Respiration may result in small changes in pixel intensities that may be concentrated in a priori unknown regions of the image frame 4005. The breath itself manifests itself as quasi-periodic changes in certain areas and 4002. Not all regions and 4002 are affected by respiration. In areas where respiration is present, the phase of the respiration signal 4183 may be opposite to the phase of the respiration signal 4183 in other areas. The respiratory signal operation 4185 may automatically combine the regions that exhibit respiration with 4002 and ignore those regions that do not exhibit respiration with 4002. The automatic respiratory signal operation 4185 may also consider the phase of the respiratory signal 4183 in the region 4003 where respiration is present.

In some embodiments, one process for combining regions and 4002 may be to use linear combinations. The coefficients in the linear combination may be set to include the region 4003 where respiration is present and exclude the region 4003 where respiration is not present. When the coefficient is set to zero, the corresponding region sum 4002 may not be included in the final estimate of the respiratory signal. When the coefficient is non-zero, the corresponding region sum 4002 may be included in the linear combination. Algebraic sign (positive or negative) of the coefficients may be used to coherently combine the regions and 4002 and consider the phase of the respiratory signal 4183 in each region and 4002. The weights and/or coefficients in the linear combination of regions and 4002 may be automatically selected to include regions 4003 where respiration is present and regions 4003 where no respiration is present may be excluded. The resulting linear combination may be a time series of scalar values, which is the final respiration signal 4183. Subspace tracking can be used to automatically adjust coefficients in linear combinations. In some embodiments, singular value decomposition may be used for subspace tracking.

In some embodiments, adaptive subspace algorithms may be used, such as Projection Approximation Subspace Tracking (PAST) adapted for image processing. The adaptive subspace algorithm may adaptively update an estimate of the orthogonal basis of the pixel vector space spanned by the incoming pixel vector. The principal basis vectors learned by the adaptive subspace algorithm can be used as coefficients in a linear combination. In addition, one of the steps in the adaptive subspace algorithm may be to calculate a linear combination. Thus, the adaptive subspace algorithm may calculate an estimate of the respiration signal 4183.

In some implementations, an adaptive subspace algorithm may be applied to the vector 4002 of the region sums. The adaptive subspace algorithm is also applicable to the original pixels in the original image frame 4005. The adaptive subspace algorithm is also applicable to video streams of moving objects obtained from sparse plus low rank decomposition. Alternative algorithms for subspace tracking and subspace updating may also be used.

In some embodiments, after estimating the respiration signal 4183, a respiration analysis operation 4195 may be performed and a respiration report 4165 may be generated. The respiration report 4165 may include a variety of information, including the current respiration rate 4231. The respiration rate 4231 may be saved in the respiration report 4165 for a period of time so that the respiration history may be analyzed and reported. Respiratory variability may also be reported. Irregular breathing and breathing abnormalities may be of interest. Breath analysis may include minimum inter-breath intervals 4233, maximum inter-breath intervals 4235, and/or inter-breath interval statistics 4237, such as mean, median, pattern, standard deviation, and the like. An inter-breath interval histogram 4239 may be calculated. The apneic events may be counted and time stamped as apneic event data 4241.

In some implementations, the movement operation 4190 may generate a movement report 4160. The motion operation 4190 may employ the respiratory event identification state of fig. 16E. The respiration can be accurately measured only when the subject is relatively stationary. When, for example, a sleeping infant turns over on a crib, the large range of motion 4267 causes high intensity changes in the image pixels, resulting in large areas and values. This masks the respiratory signal 4183. Thus, a large range of motion 4267 may be detected. If the subject's breath ceases, the caregiver will desire to be alerted to respond and wake the child. Information about the motion may be included in the region sum 4002. The motion may be detected using a classical proof of calibration test. The null hypothesis refers to the case where there is no motion. Another option is that there is motion. Motion may result in a significant increase in the variance of the region sums, such that motion is detected. Classical detection methods may be applied. Machine learning techniques may also be applied.

In some embodiments, the absence of both the wide range motion 4267 and the respiratory event 265 may indicate that a reminder should be generated. The areas and 4002 may be analyzed to detect when all movement and respiration has ceased. The estimated respiration signal 4183 may provide additional information. The regional sum 4002 and the respiratory signal 4183 may be fed into the motion operation 4190. The motion operation 4190 may employ the respiratory event identification state of fig. 16E. Transitions between the normal breathing state 4201 and the motion state 4205 may be detected by the hardware module 4116. Alarm state 4203 may be entered when all of the wide range motion 4267 approaches zero and when the respiratory signal 4183 falls below a prescribed level and/or no respiratory event 4265 is detected.

In some embodiments, a motion report 4160 may be generated and provided with information to the user. The movement and respiration information may be used to evaluate sleep length 4249 and sleep quality 4251. The beginning and end of sleep interval 4253 may be time stamped.

Fig. 17B is a schematic block diagram of a respiratory event identification process 4103 according to one or more embodiments of the disclosure. The process 4103 of fig. 17A is shown with a detrending operation 4175 performed prior to the region and operation 4170.

Fig. 17C is a schematic diagram of a video stream of the moving object generation process 4101 according to one or more embodiments of the disclosure. The process 4101 may generate a video stream of moving objects 4150 from the video stream 4120 that may include the image frames 4005 of the objects 4110. The process 4101 may receive a video stream 4120 from the camera 4105. The video stream 4120 may include the image frame sequence I (t) 4005 indexed by the cursor t. In addition, video stream 4120 may include one or more channels, which may include, but are not limited to, optical color image channels, such as red-green-blue (RGB) channels, grayscale image channels, and infrared image channels.

In some implementations, the process 4101 can define a window length N. The process may also extract windowed video sub-sequence 4145 from video stream 4120 as sequences I (t-N+1), I (t-N+2), I (t-N+3), … …, I (t-2), I (t-1), I (t). The process 4101 may organize the windowed video sequence into a matrix X4125. In some embodiments, X4125 may be an mxn matrix, and may include M rows and N columns.

In some implementations, M may be calculated as the product of the height (in pixels) of the image frame 4005 of the video stream 4120, the width (in pixels) of the image frame 4005 of the video stream 4120, and the number of channels in the image frame of the video stream 4120. Thus, matrix X4125 may be organized into a plurality of vectors, one for each image frame I in windowed video sequence 4145. The organization of matrix X4125 may simplify and enhance the computation of respiratory event 4265.

In some implementations, the process 4101 can decompose the matrix X4125 into a sparse matrix S4130 representing moving objects and a low rank matrix L4135 representing non-moving objects, as will be described below. In some implementations, the decomposition of matrix X4125 may include an additive noise matrix N4140.

In some implementations, the process 4101 can reconstruct the video stream of the moving object 4150 from the sparse matrix S130. In some implementations, each pixel of the reconstructed video stream of the moving object 4150 can include a sealer time sequence. Alternatively, each pixel of the reconstructed video stream of moving object 4150 may comprise a vector time series. In some implementations, a sliding sparse sub-sequence 4155 corresponding to windowed video sequence 4145 may be extracted from the video stream of moving object 4150.

Fig. 18A is a diagram illustrating a heat map 4363 in accordance with one or more embodiments of the present disclosure. In the depicted embodiment, the video stream 4120 of the body 4110 is shown with a heat map 4363 superimposed on the video stream 4120. In some embodiments, the respiration rate 4231 from the plurality of respiratory events 4265 may be encoded as a heat map 4363. Heat map 4363 may be superimposed on video stream 4120 for display to a user.

Fig. 18B is a diagram illustrating a heat map 4363 in accordance with one or more embodiments of the present disclosure. In the depicted embodiment, the video stream 4120 of the body 4110 (such as the chest region of the body 4110) is shown with a heat map 4363 superimposed on the video stream 4120. The heat map 4363 may be encoded with a wide range of motion 4267 of the body 4110. In some implementations, the heat map 4363 may encode the motion amplitude 4245 of the body 4110.

Fig. 18C is a schematic block diagram of a computer 4400 according to one or more embodiments of the present disclosure. The computer 4400 may be embodied in a hardware module 4116. In the depicted embodiment, the computer may include a processor 4405, memory 4410, and communication hardware 4415. Memory 4410 may comprise a semiconductor memory device. Memory 4410 may store codes. The processor 4405 may execute code. The communication hardware 4415 may communicate with other elements of the monitoring device 4123 and/or other devices, such as a mobile phone network or Wi-Fi network.

Fig. 18D is a schematic block diagram of a neural network 4475 in accordance with one or more embodiments of the disclosure. In the depicted embodiment, the neural network 4475 can include one or more hidden neurons 4455. The hidden neurons 4455 may receive input from one or more input neurons 4450 and may communicate with one or more output neurons 4460. The output neurons 4460 may indicate predictions, such as normal breathing and/or movement. The neural network 4475 may be trained with one or more video streams 4120 and one or more motion reports 4160 and/or respiration reports 4165 corresponding to the one or more video streams 4120. In addition, the real-time video stream 4120 may be presented to the input neurons 4450 of the trained neural network 4475 and the output neurons 4460 may generate the current motion report 4160 and/or the current respiration report 4165.

Fig. 19A is a flow diagram of a respiratory signal estimation method 4500 according to one or more embodiments of the present disclosure. The method 4500 may generate a breath report 4165, a motion report 4160, and/or generate a reminder. The method 4500 may be performed by the monitoring device 4123.

In some implementations, the method 4500 begins with the camera 4105 capturing a video stream 4120 of the 4505 subject 4110. The 4505 video stream 4120 may be captured without disturbing the body 4110. In some implementations, 4505 video stream 4120 can be captured in the infrared spectrum.

In some implementations, the hardware module 4116 may verify 4515 that the field of view of the camera 4105 satisfies the field of view policy 4261. In some implementations, the field of view policy 4261 may be satisfied if the subject 4110 occupies at least 10% of the field of view of the camera 4105. The hardware module 4515 may adjust the field of view of the camera 4105 until the field of view policy 4261 may be satisfied.

In some implementations, the hardware module 4116 can generate 4525 the vector time sequence 4006 for the video stream 4120. The temporal vector sequence 4006 may comprise a vector for each image frame 4005 of the video stream 4120. The hardware module 4116 may divide each image frame 4005 into regions 4003. In addition, the hardware module 4116 may sum the pixel values in each region 4003 as a pixel sum, as described in fig. 17A to 17B.

In some embodiments, the hardware module 4116 may generate 4525 the vector time-series 4006 by dividing each image frame 4005 into regions 4003 and removing the signal mean in each region 4003, as described in fig. 17A-17B. In some implementations, the signal mean may be removed from the pixel sum.

In some implementations, the hardware module 4116 may estimate 4530 the respiratory signal 4183 from the vector time sequence 4006. The respiratory signal 4183 may be estimated 4530 from the reconstructed video stream, as described below in fig. 19B. In addition, the respiratory signal 4183 may be estimated 4530 by applying an adaptive subspace algorithm to each vector of the vector time series 4006. The hardware module 4116 may also estimate a respiratory event 4265 from the respiratory signal 4183.

In some implementations, the hardware module 4116 may generate 4535 the breath report 4165. A breath report 4165 may be generated 4535 based on the breath signal 4183 and/or the breath event 4265.

In some implementations, the hardware module 4116 may determine 4540 one of the wide range motion 4267 and the respiratory event 4265 of the body 4110 based on the vector time sequence 4006 and/or the respiratory signal 4183. The hardware module 4116 may also generate 4545 the motion report 4160 based on the wide range motion 4267.

In some implementations, the hardware module 4116 can generate 4550 the reminder, and the method 4500 ends. The alert may be delivered through speaker 4121. In addition, alerts may be delivered through the display 4119. Alternatively, the reminder may be delivered by another device, such as a mobile phone.

In some embodiments, a reminder may be generated 4550 if a respiratory event 4265 is not identified and a wide range of motion 4267 of the subject 4110 is not identified within the event time interval 4263. For example, if a respiratory event 4265 is not identified from respiratory signal 4183 during event time interval 4263 and a large range of motion 4267 of body 4110 is not identified within event time interval 4263, the respiratory event identification state may transition to alarm state 4203, thereby generating a reminder.

Fig. 19B is a flow diagram of a respiratory event identification method 4600 in accordance with one or more embodiments of the present disclosure. Method 4600 may identify a respiratory event 4265 from video stream 4120. The method 4600 may be performed by one or more of the camera 4105, the hardware module 4116, and/or the memory 4420 storing code.

In some implementations, the method 4600 begins and the hardware module 4116 may extract 4605 the windowed video sub-sequence 4145 from the video stream 4120. In addition, the hardware module 4116 may organize 4610 the windowed video sub-sequences 4145 into a matrix X4125, as described in fig. 16A-16C.

In some implementations, the hardware module 4116 may decompose 4615 the matrix X4125 into a sparse matrix S4130 and a low rank matrix L4135. In some implementations, equation 1 may be used to decompose 4615 matrix X4125.

X=s+l+n equation 8

In some implementations, the hardware module 4116 may also initialize the matrix S4130 to s=0. In addition, the hardware module 4116 may calculate the low rank matrix L4135 using equation 2.

L=mean (X-S) equation 9

In some embodiments, the "mean" operator may return a matrix L having the same dimensions as the input X-S, where each element on the ith row of L is the mean (average) of the corresponding row of input X-S.

In some implementations, the hardware module 4116 may calculate each pixel of the matrix S4130 using equation 3, where S _{i,j} represents an element on the i-th row of the matrix S and on the j-th column of the matrix S, and T is the sparseness threshold. The sparsity threshold may be set such that a specified percentage of the elements in the specified matrix are filled.

The hardware module 4116 may iteratively repeat the calculations of equations 2 and 3 until the stop condition is met, resulting in a sparse matrix S4130.

In some implementations, the hardware module 4116 may also reconstruct 4620 the video stream of the moving object 4150 according to the sparse matrix S4130. In some implementations, each column vector of the sparse matrix S4130 may be reconstructed 4620 into video frames of a video stream of the moving object 4150.

In some implementations, the hardware module 4116 may reconstruct 4620 the video stream of the moving object 4150 from the plurality of sparse matrices S4130 for the plurality of windowed video sequences 4145, and the method 4600 ends. The hardware module 4116 may combine the corresponding vectors of each sparse matrix S4130 to reconstruct 4620 the video stream of the moving object 4150.

In some implementations, the hardware module 4116 may calculate 625 a normalized autocorrelation of each pixel time sequence of the reconstructed video stream of the moving object 4150. The hardware module 4116 may identify 4630 a respiratory event 4265 in response to the autocorrelation peak of a given pixel of the reconstructed video stream of the moving object 4150 continuing over a specified number of consecutive time periods, and the method 4600 ends. The time lag of the autocorrelation peak may be indicative of the respiratory cycle. The inverse of the time lag may be indicative of respiration rate.

In some implementations, using the reconstructed video stream of the moving object 4150 reduces the computational overhead of computing the respiratory signal 4183. Thus, the respiration signal 4183 may be calculated using limited processing power. Accordingly, the monitoring apparatus 4123 can be manufactured at significantly reduced cost. In addition, the monitoring device 4123 may consume less power and may be powered by a battery. Accordingly, the utility of the monitoring device 4123 can be significantly improved.

Fig. 19C is a flow diagram of a respiratory event communication method 4700 according to one or more embodiments of the present disclosure. Method 4700 may encode information based on respiratory events and communicate the information. The method 4700 may be performed by the monitoring device 4123.

In some embodiments, the method 4700 begins and the hardware module 4116 may encode 4705 the respiration rate 4231 from a plurality of respiratory events. In other embodiments, the respiration rate 4231 may be calculated as a function of the average time interval between each of the plurality of respiratory events 4265. The hardware module 4116 may also encode 4705 the respiration rate 4231 as a heat map 4363. In some implementations, a first heat map value may be assigned to a respiratory rate 4231 that is within a normal range, while a second heat map value may be assigned to a respiratory rate that is outside of the normal range. Alternatively, the heat map value may be equal to a function of the respiration rate 4231.

In some implementations, the hardware module 4116 may also encode 4710 the moving object of the video stream of the moving object 4150 as a heat map 4363. The moving object value for each pixel may be calculated as a function of the number of corresponding pixels in the sliding sparse sub-sequence 4155 that includes the moving object. The heat map value of each pixel may be a moving object value.

In some implementations, the hardware module 4116 may superimpose 4715 the heat map 4363 on the video stream 4120 for display to the user. The heat map may be superimposed in a channel (such as the alpha channel) of the video stream 4120. For example, the alpha channel of an image frame and/or pixel may be modified as a function of the thermal image value of the image frame and/or pixel.

In some implementations, the hardware module 4116 can present 4720 the statistics to the user using the display 4119, and the method 4700 ends. 4620 statistics can be presented via r4 video stream 4120. The statistics may include information of the motion report 4160 and/or the respiration report 4165.

In some embodiments, a system for monitoring an infant may include:

An optical subsystem;

A bi-directional audio system;

A sensor selected from the group consisting of: a breath detector, a heart rate detector, and any combination thereof;

A user interface configured to provide a member of the group consisting of: a reminder to at least one user, a one-way communication with the user, a two-way communication with the at least one user;

a temperature control subsystem for maintaining at least one food item at least one predetermined temperature; and

A processing unit in communication with the imaging device, the bi-directional audio system, the user interface, and the temperature control subsystem, the processing unit configured to determine at least one behavioral state of the infant from the image of the infant generated by the imaging device and generate at least one response from the behavioral state;

Wherein at least one response is selected from the group consisting of: providing audio to the baby, providing video to the baby, providing a reminder to the user, providing two-way communication between the baby and the user, altering the temperature of at least one food item, or any combination thereof.

In some embodiments, the system may include at least one projector for generating the visual signal.

In some embodiments, the system may be configured to receive at least one of an audio signal and a visual signal via the user interface and display the at least one of the audio signal and the visual signal to the infant.

In some embodiments, the system may be configured to display at least one of the pre-recorded audio signal and the visual signal to the infant.

In some embodiments, the system may be configured to display at least one of an interactive audio signal and a visual signal to the infant.

In some embodiments, at least one of the breathing rate of the infant and the at least one heartbeat rate of the infant may be determined without contact between the system and the infant.

In some embodiments, the system may be configured to determine at least one of a respiration rate of the infant and a heartbeat rate of the infant from the image of the infant.

In some embodiments, at least one of the infant's respiratory rate and the infant's heart rate may be measured via photoplethysmography (PPG).

In some embodiments, the system may be configured to generate a reminder when the heartbeat rate is outside of the range of 40 to 180 beats/min.

In some embodiments, the system may be configured to generate the reminder when the determined frequency of at least one of the breathing rate of the infant and the heartbeat rate of the infant is in a range between once every 0.1s and once every 20 s.

In some embodiments, the system may be configured to determine the heart rate from skin pixels of exposed skin of a portion of the body (such as the face, neck, torso, arms, hands, legs, and/or feet).

In some embodiments, the behavioral state of the infant may include a sleep state, an awake state, a wake state, a fatigue state, an alert state, a boring state, a affliction state, a sleep state, and/or a hunger state.

In some embodiments, the system may be configured to enable two-way communication between the infant and the user.

In some embodiments, the system may be configured for multiple users.

In some embodiments, the temperature controller may include a peltier device, an infrared heater, a heating mantle, a heating band, a microwave source, a source of hot air, a cooling mantle, a source of cold air, a source of circulating cold liquid, and/or an evaporative cooling source.

In some embodiments, the temperature controller may control the temperature via a single device that provides both heating and cooling.

In some embodiments, the temperature controller may control the temperature via different devices for heating and cooling.

In some embodiments, the system may further comprise a temperature detector for detecting the temperature of the infant.

In some embodiments, the temperature detector may be a non-contact temperature detector.

In some embodiments, the temperature detector may be selected from the group consisting of: an infrared thermometer, an infrared temperature sensor, a thermal imaging system, an infrared thermal imaging system, a thermal imaging camera, and a bolometer.

In some embodiments, the system may be configured to measure blood oxygen levels.

In some embodiments, the blood oxygen level may be measured via photoplethysmography (PPG).

In some embodiments, the system may be configured to determine the heart beat rate via steps comprising:

Detecting an object of interest (OOI) from the video data;

Tracking the OOI;

identifying a region of interest (ROI) within the OOI;

Generating super-pixels according to pixels in the ROI;

generating a super-pixel time sequence;

modeling the superpixel time series as a superpixel model;

Calculating at least one of a heartbeat signal and another cardiac characteristic using the super pixel model; and

And remotely estimating the heart rate according to the heart beat signals.

In some embodiments, the processor may be configured to determine the respiration rate via steps comprising:

Detecting an object of interest (OOI) from the video data;

Tracking the OOI;

identifying a region of interest (ROI) within the OOI;

Generating super-pixels according to pixels in the ROI;

generating a super-pixel time sequence;

modeling the superpixel time series as a superpixel model;

calculating at least one of a respiratory signal and another respiratory characteristic using a super-pixel model; and

The respiration rate is estimated remotely from the respiration signal.

In some embodiments, at least one behavioral state of the infant may be automatically determined via Artificial Intelligence (AI) (e.g., a machine learning model).

In some embodiments, the response to the at least one behavioral state may be automatically generated via the AI.

In some embodiments, a detection reaction loop for pacifying an infant may be configured to:

(1) Providing a pacifying method decision component;

(2) Collecting data from a plurality of sensors;

(3) Transmitting data to a computing unit to create an action list for each response system;

(4) Using the action list, a pacifying method selected from the group consisting of: pacifying through sound, and displaying pacifying videos by a projector; activating the vibration unit; activating a functional IoT device; activating the temperature control device to alter the temperature of the food product; controlling ambient light; activating the IoT device; sending a notification to a user;

(5) Repeating the collecting of data from the plurality of sensors;

(6) Automatically modifying the pacifying method; and

(7) Repeating the second to sixth steps until the infant is pacified.

In some embodiments, the detection reaction loop for pacifying an infant may further comprise the steps of: the modified pacifying method is sent to a crowdsourcing pacifying method server to modify the generic pacifying method decision part.

In some embodiments, the detection reaction loop for pacifying an infant may further comprise the steps of: storing the modified pacifying method;

in some embodiments, the pacifying method decision component may be preloaded into the system.

In some embodiments, the pacifying method decision feature may be used only locally.

In some embodiments, the pacifying method decision component may include at least one parameter from a crowdsourcing pacifying method server.

In some embodiments, the sensor is selected from the group consisting of: optical subsystems, microphones, acoustic/audio detectors, and lidar.

In some embodiments, sensor output data from the sensors may be used to implement a respiration detector, a temperature detector, and a heart rate detector.

In some implementations, the computing unit may be located in a component that is a camera computing processor and/or a separate processor.

In some embodiments, the computing unit may be configured to perform the steps of:

retrieving historical data from the a priori data gate;

Retrieving sensor data from the sensor data gate;

Analyzing the sensor data;

Generating, in a ICDV calculator, inputs to all response systems from the analyzed data;

transmitting inputs of all response systems to ICDV a pool of response agents, the ICDV-pool of response agents generating ICVD values for each computing agent;

transmitting the ICVD values to a response scheme calculator and creating an action list for each response system;

broadcasting an action list of each response system through the response API pool; and

The actions, IPH and ICH are updated through a priori data gates.

In some embodiments, the response system may include audio, video, and reminders.

In some embodiments, ICDV response agent pools are computing agents.

In some embodiments, the computing agent may comprise a member of the group consisting of: one agent per response system, one agent for multiple response systems, and/or multiple agents for at least one system.

In some embodiments, ICDV response agent pools may generate at least one ICDV value from the analyzed sensor data.

In some embodiments, a plurality of at least one ICDV values may be weighted to determine the relative importance of the ICDV value.

In some embodiments, the weights may be predefined and/or AI-generated.

In some embodiments, the sound used for pacifying may include music, white/pink noise, and/or the user's voice.

In some embodiments, the user's voice may be recorded or in real-time.

In some embodiments, pacifying a sound may be characterized by sound volume, sound type, and/or length of time that the sound is emitted.

In some embodiments, ambient light level changes may be implemented by turning on lights, turning off lights, adjusting illumination levels, and/or changing the color of light.

In some implementations, the alert notification can be sent to the user via a mobile application, via an application, via a built-in OS notification, via a telephone call, via sound from a speaker system, and/or via an optical signal.

In some embodiments, a method of monitoring an infant may comprise the steps of:

a system for monitoring an infant is provided, the system comprising:

i. An optical subsystem;

A bi-directional audio system;

a plurality of sensors outputting sensor data, the plurality of sensors configured to perform breath detection and/or heart rate detection;

A processing unit in communication with the imaging device, the bi-directional audio system, the sensor, the user interface, and the temperature control subsystem, the processing unit configured to determine at least one behavioral state of the infant from the infant signal selected from the group consisting of: an image of the infant generated by the imaging device, signals from the sensor, and any combination thereof;

placing the optical subsystem within the visual range of the infant;

placing the bi-directional speaker within the hearing range of the infant;

placing the bi-directional audio system within the hearing range of the infant;

Placing the sensor within a sensing range of the infant;

Activating a system for monitoring the infant, thereby generating an infant signal;

Determining a behavioral state of the infant from the infant signal;

Generating at least one response based on the behavioral state; and

Selecting at least one response from the group consisting of: providing audio to the baby, providing video to the baby, providing a reminder to the user, providing two-way communication between the baby and the user, and altering the temperature of at least one food item.

In some embodiments, the method may include providing at least one projector for generating the visual signal.

In some embodiments, the method may include receiving at least one of an audio signal and a visual signal via the user interface, and displaying the at least one of the audio signal and the visual signal to the infant.

In some embodiments, the method may include displaying at least one of the pre-recorded audio signal and the visual signal to the infant.

In some embodiments, the method may include displaying at least one of an interactive audio signal and a visual signal to the infant.

In some embodiments, the method may include determining at least one of a respiration rate of the infant and a heartbeat rate of the infant without contact between the system and the infant.

In some embodiments, the method may include determining at least one of a respiration rate of the infant and a heartbeat rate of the infant from the image of the infant.

In some embodiments, the method may include measuring at least one of the infant's respiratory rate and the infant's heartbeat rate via photoplethysmography (PPG).

In some embodiments, the method may include sending a reminder when the heart beat rate is outside of the range of 40 to 180 beats/min.

In some embodiments, the method may include at least one of the following steps:

Determining at least one of the infant's respiration rate and the infant's heartbeat rate at a frequency in the range between once every 0.1s and once every 20 s;

Determining the heart beat rate from skin pixels of exposed skin of a portion of the body selected from the group consisting of: face, neck, torso, arms, hands, legs, and feet;

the behavioral state of the infant is selected from the group consisting of: sleeping state, awake state, fatigue state, alert state, boring state, distress state, sleep-in state, and hunger state.

In some embodiments, the method may include providing two-way communication between the infant and the user.

In some embodiments, the method may include configuring the system for a plurality of users.

In some embodiments, the method may include providing a temperature controller, which may include a peltier device, an infrared heater, a heating mantle, a heating band, a microwave source, a hot air source, a cooling mantle, a cold air source, a circulating cold liquid source, and/or an evaporative cooling source.

In some embodiments, the temperature controller may control the temperature via a single device that provides both heat and cooling.

In some embodiments, the method may include providing a temperature detector for detecting the temperature of the infant.

In some embodiments, the method may include providing a temperature detector as the non-contact temperature detector.

In some embodiments, the method may include selecting the temperature detector from the group consisting of: an infrared thermometer, an infrared temperature sensor, a thermal imaging system, an infrared thermal imaging system, a thermal imaging camera, and a bolometer.

In some embodiments, the method may include measuring blood oxygen levels.

In some embodiments, the method may include measuring blood oxygen levels via photoplethysmography (PPG).

In some embodiments, the method may include determining the heart beat rate via:

Detecting an object of interest (OOI) from the video data;

Tracking the OOI;

identifying a region of interest (ROI) within the OOI;

Generating super-pixels according to pixels in the ROI;

generating a super-pixel time sequence;

modeling the superpixel time series as a superpixel model;

And remotely estimating the heart rate according to the heart beat signals.

In some embodiments, the method may include determining the respiration rate via:

Detecting an object of interest (OOI) from the video data;

Tracking the OOI;

identifying a region of interest (ROI) within the OOI;

Generating super-pixels according to pixels in the ROI;

generating a super-pixel time sequence;

modeling the superpixel time series as a superpixel model;

The respiration rate is estimated remotely from the respiration signal.

In some embodiments, the method may include automatically determining, via the AI, at least one behavioral state of the infant.

In some embodiments, the method may include automatically generating, via the AI, a response to the behavioral state.

In some embodiments, the method may include providing a detection reaction loop for pacifying an infant, comprising the steps of:

(1) Providing a pacifying method decision component;

(2) Collecting data from a plurality of sensors;

(5) Repeating the collecting of data from the plurality of sensors;

(6) Automatically modifying the pacifying method; and

(7) Repeating the second to sixth steps until the infant is pacified.

In some embodiments, the method may include sending the modified pacifying method to a crowdsourcing pacifying method server to modify the generic pacifying method decision part.

In some embodiments, the method may comprise the steps of:

Storing the modified pacifying method;

Preloading a pacifying method decision component into a system;

Only the pacifying method decision-making component is used locally;

Wherein the pacifying method decision part may comprise at least one parameter from a crowdsourcing pacifying method server;

Selecting a sensor from the group consisting of: an optical subsystem, a microphone, an acoustic/audio detector, a temperature detector, and a lidar;

using the sensor output data to perform breath detection and heart rate detection;

The computing unit is located in a component selected from the camera computing processor or a separate processor.

In some embodiments, the method may include the computing unit additionally performing the steps of:

retrieving historical data from the a priori data gate;

Retrieving sensor data from the sensor data gate;

Analyzing the sensor data;

The actions, IPH and ICH are updated through a priori data gates.

In some embodiments, the method may include selecting a response system having audio, video, and/or alerts.

In some embodiments, the ICDV response agent pool may include computing agents.

In some embodiments, the computing agent may include one agent per response system, one agent for multiple response systems, and/or multiple agents for at least one method.

In some embodiments, the method may include ICDV generating at least one ICDV value from the analyzed sensor data in response to the proxy pool.

In some embodiments, the method may include weighting a plurality of at least one ICDV values to determine the relative importance of ICDV values.

In some embodiments, the method may include selecting weights from the group consisting of: predefined weights and AI-generated weights.

In some embodiments, the method may include selecting the pacifying sound from the group consisting of: music, white/pink noise, and user speech.

In some embodiments, the method may include providing at least one of recorded speech and real-time speech.

In some embodiments, the method may include changing a characteristic of the pacifying sound, wherein the characteristic of the pacifying sound may include sound volume, sound type, and/or length of time the sound is emitted.

In some embodiments, the method may include selecting the ambient light level change from the group consisting of: turning on, turning off, adjusting the illumination level, and changing the color of the light.

In some embodiments, the method may include sending the alert notification via a mobile application, an application, a built-in OS notification, a telephone call, sound from a speaker system, and/or an optical signal.

In some embodiments, a system may include:

A nonvolatile memory;

At least one electronic resource, the at least one electronic resource comprising a database;

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

An optical subsystem including an imaging device, a projection device, or both.

Wherein the optical subsystem is configured to perform at least one of:

(i) Acquiring image data of an image of at least one infant from a plurality of infants from an imaging device, or

(Ii) Projecting, by a projection device, at least one visual image for viewing by at least one infant;

an audio system comprising a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(i) Receiving audio signal data from at least one infant by a microphone, or

(Ii) Generating at least one sound for at least one infant by a speaker;

a plurality of sensors that output sensor data;

A communication circuit configured to communicate with at least one communication device of at least one user associated with at least one baby over a communication network; and

Receiving image data, audio signal data, and sensor data associated with at least one infant;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

Inputting image data, audio signal data, sensor data, baby-specific physiological data, and baby-specific personal data indicative of a current behavioral state of the at least one baby into the at least one baby-specific behavioral state detection machine learning model;

wherein the at least one infant-specific behavioral state detection machine learning model is trained using a dataset based at least in part on infant-specific stimulus data, infant-specific response data, and infant-specific personal data associated with the plurality of infants;

receiving an output having at least one environmental action from at least one infant specific behavioral state detection machine learning model;

Wherein the output comprises at least one environmental action, at least one behavioral recommendation, or both that changes the current behavioral state of the at least one infant to a predefined behavioral state; and

At least one of the following is performed based on the output:

(i) Transmitting instructions to a projection device, a speaker, at least one peripheral device, at least one internet of things (IoT) device, or any combination thereof to implement at least one environmental action, and

(Ii) At least one behavioral recommendation is displayed on a graphical user interface of at least one communication device of at least one user associated with at least one infant.

In some embodiments, a method may comprise:

Receiving, by a processor, from at least one electronic resource, the at least one electronic resource comprising a database;

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

receiving, by the processor, image data from an image of at least one infant of the plurality of images from the imaging device;

receiving, by the processor, audio signal data of at least one infant from the microphone;

Receiving, by the processor, sensor data associated with at least one infant from a plurality of sensors;

Determining, by the processor, infant-specific physiological data of the at least one infant based on the image data, the sensor data, and the audio signal data;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

Inputting, by the processor, image data, audio signal data, sensor data, baby-specific physiological data, and baby-specific personal data associated with the at least one baby to the at least one baby-specific behavioral state detection machine learning model;

receiving, by the processor, an output having at least one environmental action from at least one infant-specific behavioral state-detecting machine learning model;

Executing, by the processor, at least one of the following based on the output:

(i) Transmitting, by the processor, instructions to the projection device, the speaker, the at least one peripheral device, the at least one internet of things (IoT) device, or any combination thereof to implement at least one environmental action, and

(Ii) At least one behavioral recommendation is displayed by the processor on a graphical user interface of at least one communication device of at least one user associated with the at least one infant.

In some embodiments, a system may include:

A nonvolatile memory;

At least one electronic resource, which may include a database;

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Wherein the optical subsystem is configured to perform at least one of:

An audio system, which may include a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(i) Receiving audio signal data from at least one infant by a microphone, or

(Ii) Generating at least one sound for at least one infant by a speaker;

a plurality of sensors that output sensor data;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

Inputting image data, audio signal data, sensor data, baby-specific physiological data, and baby-specific personal data associated with the at least one baby to the at least one baby-specific behavioral state detection machine learning model;

Receiving output of at least one infant anxiety, about to wake up, or both from at least one infant-specific behavioral state-detecting machine learning model;

Transmitting, over a communication network, sensor data to at least one communication device of at least one user, at least one reminder of anxiety in the infant, at least one reminder of the infant about to wake up, or any combination thereof;

In some embodiments, the at least one processor may be further configured to transmit the instructions iteratively until the infant is pacified or falls asleep again.

In some embodiments, the at least one processor may be further configured to receive user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

In some embodiments, the at least one processor may be further configured to transmit instructions based on the output and user instructions from the graphical user interface.

In some embodiments, the sensor of the plurality of sensors may be selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

In some embodiments, the system may further comprise a vibration unit operably coupled to the at least one infant, and wherein the at least one processor may be further configured to transmit instructions to the vibration unit that cause the vibration unit to apply vibrations to the at least one infant when the infant is anxious.

In some embodiments, the sensor of the plurality of sensors may be selected from the group consisting of: thermal imaging devices, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

In some embodiments, a method may comprise:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

receiving, by the processor, output of at least one infant anxiety, about to wake up, or both from at least one infant-specific behavioral state-detecting machine learning model;

Transmitting, by the processor, sensor data, at least one reminder of anxiety about the infant, at least one reminder of about to wake up the infant, or any combination thereof, over a communication network to at least one communication device of at least one user associated with the at least one infant; and

Transmitting, by the processor, instructions based on the output, the instructions causing the speaker, the projection device, or both to perform at least one of:

In some embodiments, the method includes iteratively transmitting instructions by the processor until the infant is pacified or falls asleep again.

In some embodiments, the method includes receiving, by the processor, user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

In some embodiments, the method includes transmitting, by the processor, instructions based on the output and user instructions from the graphical user interface.

In some embodiments, the sensor of the plurality of sensors is selected from the group consisting of: video cameras, thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

In some embodiments, the method includes transmitting, by the processor, instructions to the vibratory unit that cause the vibratory unit to apply vibration to the infant when the at least one infant is anxious, wherein the vibratory unit is operatively coupled to the at least one infant.

In some embodiments, a method comprises:

Wherein the database comprises:

(i) Infant specific stimulus data for a plurality of infant specific stimuli provided to each of a plurality of infants,

(Ii) Infant-specific response data of a plurality of infant-specific responses of each infant acquired in response to infant-specific stimulus data provided to each infant, and

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Determining, by the processor, sensor data from the plurality of sensors, image data from the image device, and audio data from the microphone acquired while monitoring each baby of the plurality of babies based on the baby-specific stimulus data and the baby-specific response data for each baby;

Determining, by the processor, baby-specific physiological data for each baby based on the image data, the sensor data, and the audio signal data;

wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environmental-specific features for each of a plurality of infants unique to an infant pacifying use case from infant-specific physiological data, image data, audio signal data, sensor data, and infant-specific personal data for each infant;

Executing, by the processor, a time-driven pipeline software module configured to calculate, from infant-specific features and context-specific features of each infant that are unique to the infant pacifying use case, time-dependent infant-specific features and time-dependent context-specific features of each infant that characterize the progress of the features over time based at least in part on a time sequence generated from the infant-specific features and context-specific features;

executing, by the processor, a behavioral model configured to generate a digital twin model for each infant of the plurality of infants based at least in part on:

(1) Correlation between infant-specific characteristics from each infant and infant-specific behavioral characteristics of environmental-specific characteristics, and

(2) Baby-specific stimulus data, baby-specific response data and baby-specific personal data for each baby,

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant according to an infant pacifying use case for each infant; and

Generating, by the processor, a training data set of infant pacifying use cases based at least in part on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Time-dependent infant-specific characteristics of each infant, and

(D) Time-dependent environmental specific characteristics of each infant, and

(2) An output characteristic for each infant from the plurality of infants, comprising:

(A) At least one recommendation for each infant,

(B) At least one action for each infant, or

(C) Both of which are located in the same plane.

In some embodiments, the method includes training, by the processor, at least one infant-specific behavioral state-detection machine learning model with a training data set unique to the infant pacifying use case.

In some embodiments, the at least one action for pacifying each infant is selected from the group consisting of: playing pacifying music, applying vibration via a vibration unit, projecting pacifying images, and playing sleep-aiding music.

In some embodiments, a system may include:

A nonvolatile memory;

at least one electronic resource comprising at least one database;

wherein the at least one database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

An imaging device configured to acquire image data of an image of at least one infant from a plurality of infants;

a microphone configured to receive audio signal data from at least one infant;

a plurality of sensors that output sensor data;

a communication circuit configured to communicate with at least one communication device of at least one user associated with at least one baby over a communication network;

A temperature controller; and

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

receiving at least one indication of at least one infant hunger from at least one infant specific behavioral state detection machine learning model;

Transmitting, over a communication network, a reminder, sensor data, or both, to at least one communication device of at least one user to feed at least one infant; and

Transmitting instructions that cause the temperature controller to change a predefined temperature of the at least one food item in preparation for feeding the at least one infant.

In some embodiments, the at least one processor is further configured to receive user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

In some embodiments, the at least one processor is further configured to transmit instructions based on the at least one indication and the user instructions from the graphical user interface.

In some embodiments, the sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

In some embodiments, the vibration unit is operably coupled to the at least one infant, and wherein the at least one processor is further configured to transmit instructions to the vibration unit that cause the vibration unit to apply vibration to the at least one infant when the infant is hungry.

A method may include:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

receiving, by the processor, an output from the at least one infant-specific behavioral state-detecting machine learning model indicative of at least one indication of at least one infant hunger;

transmitting, by the processor, sensor data, a reminder of anxiety about the at least one infant, a reminder of feeding the at least one infant, the sensor data, or both, over a communication network to at least one communication device of at least one user associated with the at least one infant; and

Instructions are transmitted by the processor that cause the temperature controller to change a predefined temperature of the at least one food item in preparation for feeding the at least one infant.

In some embodiments, the method includes transmitting, by the processor, instructions based on the at least one indication and user instructions from the graphical user interface.

In some embodiments, a method may comprise:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environmental-specific features for each of a plurality of infants unique to an infant feeding instance based on the infant-specific physiological data, image data, audio signal data, sensor data, and infant-specific personal data for each infant;

executing, by the processor, a time-driven pipeline software module configured to calculate, from infant-specific features and context-specific features of each infant that are unique to the infant feeding instance, time-dependent infant-specific features and time-dependent context-specific features of each infant that characterize the feature progression over time based at least in part on a time sequence generated from the infant-specific features and context-specific features;

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant according to an infant feeding profile for each infant; and

Generating, by the processor, a training data set of infant feeding at least partially based on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Time-dependent infant-specific characteristics of each infant, and

(D) Time-dependent environmental specific characteristics of each infant, and

(A) At least one recommendation for each infant,

(B) At least one action for each infant, or

(C) Both of which are located in the same plane.

In some embodiments, the method includes training, by the processor, at least one infant specific behavioral state detection machine learning model with a training data set unique to the infant feeding instance.

In some embodiments, the at least one action for the at least one infant comprises warming at least one food item.

In some embodiments, a system comprises:

A nonvolatile memory;

at least one electronic resource comprising at least one database;

wherein the at least one database comprises:

(Ii) Infant-specific stimulus data for a plurality of infant-specific stimuli provided to a plurality of infants based on a plurality of infant-specific education programs,

(Iv) Baby-specific personal data for each baby in the plurality of babies;

an optical subsystem comprising an imaging device, a projection device, or both;

Wherein the optical subsystem is configured to perform at least one of:

an audio system comprising a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(i) Receiving audio signal data from at least one infant by a microphone, or

(Ii) Generating at least one sound for at least one infant by a speaker;

a plurality of sensors that output sensor data;

projecting, by the projection device, at least one visual image to the at least one infant based on an infant-specific education program from a plurality of infant-specific education programs for the at least one infant;

generating, by the audio system, at least one sound associated with the at least one visual image;

Receiving image data, audio signal data, sensor data, or any combination thereof associated with at least one infant;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

Inputting image data, audio signal data, sensor data, at least one visual image, at least one sound, baby-specific physiological data, baby-specific personal data associated with at least one baby to at least one baby-specific educational machine learning model;

Wherein the at least one infant-specific educational machine learning model is trained using a dataset based at least in part on infant-specific stimulus data, infant-specific response data, infant-specific personal data, a plurality of infant-specific educational programs, or any combination thereof associated with the plurality of infants;

Receiving output from at least one infant-specific educational machine learning model;

wherein the outputting comprises:

Transmitting at least one indication, at least one baby-specific educational recommendation, sensor data, or any combination thereof, to at least one communication device of at least one user over a communication network; and

At least one of the following is performed based on at least one infant-specific educational recommendation:

In some embodiments, the at least one processor is further configured to perform at least one of the following based on the at least one indication, the at least one infant-specific educational recommendation, and the user instructions:

In some embodiments, a method may comprise:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Transmitting, by the processor, instructions to the projection device to project at least one visual image to the at least one infant based on an infant-specific educational program from a plurality of infant-specific educational programs for the at least one infant;

Transmitting, by the processor, instructions to the audio system to generate at least one sound associated with the at least one visual image;

wherein the infant-specific physiological data comprises:

(i) At least one of the infant's respiratory rate signal data,

(Ii) Spatial body temperature distribution data of at least one infant,

(Iii) At least one infant's heartbeat signal data,

(Iv) Baby movement data of at least one baby, and

(V) Infant speech classification data for at least one infant;

Inputting, by the processor, image data, audio signal data, sensor data, at least one visual image, at least one sound, baby-specific physiological data, baby-specific personal data associated with the at least one baby to the at least one baby-specific educational machine learning model;

Wherein the at least one infant-specific educational machine learning model is trained using a dataset based at least in part on infant-specific stimulus data, infant-specific response data, and infant-specific personal data associated with the plurality of infants;

receiving, by the processor, output from at least one infant-specific educational machine learning model;

wherein the outputting comprises:

Transmitting, by the processor, at least one indication, at least one baby-specific educational recommendation, sensor data, or any combination thereof, over a communication network to at least one communication device of at least one user associated with at least one baby; and

Performing, by the processor, at least one of the following based on the at least one infant-specific educational recommendation:

In some embodiments, the method includes performing, by the processor, at least one of the following based on the at least one indication, the at least one infant-specific educational recommendation, and the user instructions:

Modifying at least one infant-specific educational program, or, when at least one indication indicates that at least one infant is not understood

The execution of the infant-specific educational program for the at least one infant continues.

In some embodiments, a method comprises:

Wherein the database comprises:

(i) An infant-specific education program from a plurality of infant-specific education programs for each of the plurality of infants,

(Ii) Infant specific stimulus data for a plurality of infant specific stimuli provided to each of a plurality of infants,

(Iii) Infant-specific response data of a plurality of infant-specific responses of each infant acquired in response to infant-specific stimulus data provided to each infant, and

(Iv) Baby-specific personal data for each baby in the plurality of babies;

determining, by the processor, from the baby-specific stimulation data and the baby-specific response data for each baby:

(i) Sensor data from a plurality of sensors acquired while monitoring each infant of a plurality of infants,

(Ii) Image data from the image device acquired while each baby is monitored,

(Iii) At least one visual image presented to each infant based on the infant-specific educational program,

(Iv) At least one sound associated with at least one visual image played to each baby, and

(V) Audio data from the microphone acquired while monitoring each baby;

wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environment-specific features of each of a plurality of infants unique to the infant education use-case from the infant-specific physiological data, the image data, the audio data, the sensor data, the at least one visual image, the at least one sound, and the infant-specific personal data of each infant;

Executing, by the processor, a time-driven pipeline software module configured to calculate, from infant-specific features and context-specific features of each infant that are unique to the infant education use-case, time-dependent infant-specific features and time-dependent context-specific features of each infant that characterize the feature progression over time based at least in part on a time sequence generated from the infant-specific features and context-specific features;

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant in accordance with the infant educational use case for each infant; and

Generating, by the processor, a training data set of the infant education use-case based at least in part on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Time-dependent infant-specific characteristics of each infant, and

(D) Time-dependent environmental specific characteristics of each infant, and

(A) At least one recommendation for each infant,

(B) At least one action for each infant, or

(C) Both of which are located in the same plane.

In some embodiments, the method includes training, by the processor, at least one infant-specific behavioral state-detection machine learning model with a training data set unique to the infant education use case.

In some embodiments, the at least one recommendation for the at least one infant is selected from the group consisting of: changing at least one visual image, changing at least one sound, modifying at least one infant-specific education program, and continuing to execute the infant-specific education program.

The materials disclosed herein may be implemented in software or firmware or a combination thereof or as instructions stored on a machine-readable medium readable and executable by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include: read Only Memory (ROM); random Access Memory (RAM); a magnetic disk storage medium; an optical storage medium; a flash memory device; electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.

As used herein, the terms "computer engine" and "engine" identify at least one software component and/or a combination of at least one software component and at least one hardware component that is designed/programmed/configured to manage/control other software and/or hardware components (such as libraries, software Development Kits (SDKs), objects, etc.).

Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application Specific Integrated Circuits (ASIC), programmable Logic Devices (PLD), digital Signal Processors (DSP), field Programmable Gate Array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. In some implementations, the one or more processors may be implemented as Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors; the x86 instruction set is compatible with a processor, multi-core, or any other microprocessor or Central Processing Unit (CPU). In various implementations, the one or more processors may be dual-core processors, dual-core mobile processors, or the like.

Computer-related systems, computer systems, and systems as used herein include any combination of hardware and software. Examples of software may include software components, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application Program Interfaces (APIs), instruction sets, computer code segments, words, values, symbols, or any combination thereof. Determining whether to implement an embodiment using hardware elements and/or software elements may vary depending on any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.

One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represent various logic within a processor, which when read by a machine, cause the machine to fabricate logic to perform the techniques described herein. Such representations, referred to as "IP cores," may be stored on a tangible machine-readable medium and supplied to various customers or manufacturing facilities for loading into the manufacturing machines that manufacture the logic or processor. It is noted that the various embodiments described herein may of course be implemented using any suitable hardware and/or computational software language (e.g., C++, objective-C, swift, java, javaScript, python, perl, QT, etc.).

In some embodiments, one or more of the example inventive computer-based systems/platforms of the present disclosure, the example inventive computer-based devices, and/or the example inventive computer-based components may include or incorporate, in part or in whole, at least one Personal Computer (PC), laptop computer, ultra-laptop computer, tablet, touchpad, portable computer, handheld computer, palm-top computer, personal Digital Assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile Internet Device (MID), messaging device, data communication device, and the like.

As used herein, the term "server" should be understood to refer to a service point that provides processing, databases, and communication facilities. By way of example, and not limitation, the term "server" may refer to a single physical processor with associated communication and data storage and database facilities, or it may refer to a networked or clustered complex of processors and associated networks and storage devices, as well as operating software and one or more database systems and application software supporting services provided by the server. Cloud servers are examples.

In some embodiments, as detailed herein, one or more of the computer-based systems/platforms of the present disclosure, the computer-based devices of the present disclosure, and/or the computer-based components of the present disclosure may obtain, manipulate, transmit, store, transform, generate, and/or output any digital object and/or data unit (e.g., from within and/or outside of a particular application), which may be in any suitable form, such as, but not limited to, a file, a contact, a task, an email, a social media post, a map, an entire application (e.g., a calculator), and the like. In some embodiments, as detailed herein, the exemplary inventive computer-based system/platform of the present disclosure, the exemplary inventive computer-based device, and/or the exemplary inventive computer-based component may be implemented across one or more of a variety of computer platforms, such as, but not limited to ：(1)FreeBSD、NetBSD、OpenBSD;(2)Linux;(3)Microsoft Windows;(4)OS X(MacOS);(5)MacOS11;(6)Solaris;(7)Android;(8)iOS;(9) embedded Linux; (10) Tizen; (11) WebOS; (12) IBM i; (13) IBM AIX; (14) a binary run time environment for wireless (BREW); (15) Cocoa (API); (16) Cocoa Touch; (17) a Java platform; (18) JavaFX; (19) JavaFX Mobile; (20) Microsoft DirectX; (21) NET framework; (22) Silverlight; (23) an open Web platform; (24) an Oracle database; (25) Qt; (26) Eclipse rich client platform; (27) SAP NETWEAVER; (28) SMARTFACE; and/or (29) Windows runtime.

In some embodiments, the exemplary inventive computer-based systems/platforms of the present disclosure, the exemplary inventive computer-based devices, and/or the exemplary inventive computer-based components may be configured to utilize hardwired circuitry that may be used in place of or in combination with software instructions to implement features consistent with the principles of the present disclosure. Thus, implementations consistent with the principles of the disclosure are not limited to any specific combination of hardware circuitry and software. For example, the various embodiments may be embodied as software components in many different ways, such as but not limited to stand-alone software packages, combinations of software packages, or it may be a software package incorporated as a "tool" into a larger software product.

For example, exemplary software specifically programmed according to one or more principles of the present disclosure may be downloaded from a network (e.g., website) as a stand-alone product or as a plug-in package for installation in an existing software application. For example, exemplary software specifically programmed according to one or more principles of the present disclosure may also be implemented as a client-server software application or as a web-enabled software application. For example, exemplary software specifically programmed according to one or more principles of the present disclosure may also be embodied as a software package installed on a hardware device.

In some embodiments, the exemplary inventive computer-based systems/platforms, exemplary inventive computer-based devices, and/or exemplary inventive computer-based components of the present disclosure may be configured to handle numerous concurrent users, which may be, but are not limited to, at least 100 (e.g., but not limited to, 100 to 999), at least 1,000 (e.g., but not limited to, 1,000 to 9,999), at least 10,000 (e.g., but not limited to, 10,000 to 99,999), at least 100,000 (e.g., but not limited to, 100,000 to 999,999), at least 1,000,000 (e.g., but not limited to, 1,000,000 to 9,999,999), at least 10,000,000 (e.g., but not limited to, 10,000 to, 999), at least 100,000,000 (e.g., but not limited to, 100,000 to 999,999), at least 1,000,000 (e.g., but not limited to, 999,000,999,000 to, 999,000 to, etc.

In some embodiments, the exemplary inventive computer-based system/platform, the exemplary inventive computer-based device, and/or the exemplary inventive computer-based component of the present disclosure may be configured to output to different, specially programmed graphical user interface implementations (e.g., desktops, web applications, etc.) of the present disclosure. In various implementations of the present disclosure, the final output may be displayed on a display screen, which may be, but is not limited to, a screen of a computer, a screen of a mobile device, or the like. In various implementations, the display may be a holographic display. In various implementations, the display may be a transparent surface that may receive a visual projection. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be visual overlays for Mobile Augmented Reality (MAR) applications.

As used herein, the term "mobile electronic device" or the like may refer to any portable electronic device that may or may not enable location tracking functionality (e.g., MAC address, internet Protocol (IP) address, etc.). For example, the mobile electronic device may include, but is not limited to, a mobile phone, a Personal Digital Assistant (PDA), blackberry ^TM, a pager, a smart phone, or any other reasonable mobile electronic device.

As used herein, the terms "cloud," "internet cloud," "cloud computing," "cloud architecture," and similar terms correspond to at least one of: (1) A large number of computers connected via a real-time communication network (e.g., the internet); (2) Providing the ability to run programs or applications on many connected computers (e.g., physical machines, virtual Machines (VMs)) simultaneously; (3) Network-based services, which appear to be provided by real server hardware and are actually provided by virtual hardware (e.g., virtual servers) that is emulated by software running on one or more real machines (e.g., allowing for immediate movement around and expansion up (or expansion down) without affecting the end user).

Of course, the foregoing examples are illustrative and not limiting.

As used herein, the term "user" shall have the meaning of at least one user. In some embodiments, the terms "user," "subscriber," "consumer," or "client" should be understood to refer to a user of one or more applications and/or a consumer of data supplied by a data provider as described herein. By way of example and not limitation, the term "user" or "subscriber" may refer to a person receiving data provided by a data or service provider over the internet in a browser session, or may refer to an automated software application that receives data and stores or processes the data.

In some embodiments, the exemplary inventive computer-based systems/platforms of the present disclosure, the exemplary inventive computer-based devices, and/or the exemplary inventive computer-based components may be configured to utilize one or more exemplary AI/machine learning techniques selected from, but not limited to, decision trees, boosting, support vector machines, neural networks, nearest neighbor algorithms, naive bayes, bagging (bagging), random forests, and the like. In some embodiments, and optionally in combination with any of the embodiments described above or below, the exemplary neural network technique may be, but is not limited to, one of a feed forward neural network, a radial basis function network, a recurrent neural network, a convolutional network (e.g., U-net), or other suitable network. In some embodiments, and optionally, in combination with any of the embodiments described above or below, an exemplary implementation of the neural network may be performed as follows:

i) A neural network architecture/model is defined,

Ii) transmitting the input data to an exemplary neural network model,

Iii) The exemplary model is incrementally trained and,

Iv) determining the accuracy of a specific number of time steps,

V) applying an exemplary trained model to process the newly received input data,

Vi) optionally and in parallel, continuing to train the exemplary trained model with a predetermined periodicity.

In some embodiments, and optionally in combination with any of the embodiments described above or below, the exemplary trained neural network model may specify the neural network by at least a neural network topology, a series of activation functions, and connection weights. For example, the topology of a neural network may include the configuration of nodes of the neural network and the connections between such nodes. In some embodiments, and optionally in combination with any of the embodiments described above or below, the exemplary trained neural network model may also be specified to include other parameters, including, but not limited to, bias values/functions and/or aggregation functions. For example, the activation function of a node may be a step function, a sine function, a continuous or piecewise linear function, an sigmoid function, a hyperbolic tangent function, or other type of mathematical function that represents a threshold value for activating the node. In some embodiments, and optionally in combination with any of the embodiments described above or below, the exemplary aggregate function may be a mathematical function that combines (e.g., sums, products, etc.) the input signals to the nodes. In some embodiments, and optionally in combination with any of the embodiments described above or below, the output of the exemplary aggregation function may be used as an input to the exemplary activation function. In some embodiments, and optionally in combination with any of the embodiments described above or below, the bias may be a constant value or function that may be used by the aggregation function and/or the activation function to make the node more or less likely to be activated.

Publications cited throughout this document are incorporated by reference in their entirety. While one or more embodiments of the present disclosure have been described, it should be understood that these embodiments are merely illustrative and not limiting, and that many modifications may become apparent to those of ordinary skill in the art, including various embodiments of the present methods, present systems/platforms, and the present apparatus described herein may be utilized in any combination with one another. Furthermore, the steps may be performed in any desired order (and any desired steps may be added and/or any desired steps may be eliminated).

Claims

1. A system, the system comprising:

A nonvolatile memory;

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

an optical subsystem including an imaging device, a projection device, or both.

Wherein the optical subsystem is configured to perform at least one of:

an audio system comprising a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(Ii) Generating at least one sound for the at least one infant by the speaker;

A plurality of sensors that output sensor data;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

2. The system of claim 1, wherein the at least one processor is further configured to iteratively transmit the instructions until the infant is pacified or falls asleep again.

3. The system of claim 1, wherein the at least one processor is further configured to receive user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

4. The system of claim 3, wherein the at least one processor is further configured to transmit the instructions based on the output and the user instructions from the graphical user interface.

5. The system of claim 1, wherein the sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

6. The system of claim 1, further comprising a vibration unit operatively coupled to the at least one infant, and wherein the at least one processor is further configured to transmit instructions to the vibration unit that cause the vibration unit to apply vibrations to the infant when the at least one infant is anxious.

7. A method, the method comprising:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Receiving, by the processor, image data from an image of at least one infant of the plurality of images from an imaging device;

Receiving, by the processor, audio signal data of the at least one infant from a microphone;

Receiving, by the processor, sensor data associated with the at least one infant from a plurality of sensors;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Inputting, by the processor, the image data, the audio signal data, the sensor data, the infant-specific physiological data, and the infant-specific personal data associated with at least one infant to at least one infant-specific behavioral state detection machine learning model;

receiving, by the processor, output of the at least one infant anxiety, about to wake up, or both from the at least one infant-specific behavioral state-detection machine learning model;

Transmitting, by the processor, the sensor data, a reminder of anxiety disorder of the at least one infant, a reminder of about to wake up of the at least one infant, or any combination thereof, over a communication network to at least one communication device of at least one user associated with the at least one infant; and

Transmitting, by the processor, instructions based on the output, the instructions causing a speaker, a projection apparatus, or both to perform at least one of:

8. The method of claim 12, further comprising iteratively transmitting, by the processor, the instructions until the infant is pacified or falls asleep again.

9. The method of claim 13, further comprising receiving, by the processor, user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

10. The method of claim 14, further comprising transmitting, by the processor, the instruction based on the output and the user instruction from the graphical user interface.

11. The method of claim 12, wherein a sensor from the plurality of sensors is selected from the group consisting of: video cameras, thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

12. The method of claim 12, further comprising transmitting, by the processor, instructions to a vibratory unit that cause the vibratory unit to apply vibrations to the infant when the at least one infant is anxious;

wherein the vibration unit is operatively coupled to the at least one infant.

13. A method, the method comprising:

Wherein the database comprises:

(Ii) Infant-specific response data of a plurality of infant-specific responses of each infant acquired in response to the infant-specific stimulus data provided to each infant, and

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Determining, by the processor, sensor data from a plurality of sensors, image data from an image device, and audio data from a microphone acquired while monitoring each baby of the plurality of babies from the baby-specific stimulus data and the baby-specific response data for each baby;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environment-specific features of each infant of the plurality of infants unique to an infant pacifying use case from the infant-specific physiological data, the image data, the audio signal data, the sensor data, and the infant-specific personal data of each infant;

Executing, by the processor, a time-driven pipeline software module configured to calculate, from the infant-specific features and the environment-specific features for each infant that are unique to the infant pacifying use case, a time-dependent infant-specific feature and a time-dependent environment-specific feature for each infant that characterize feature progression over time based at least in part on a time sequence generated from the infant-specific features and the environment-specific features;

(1) Correlation between the infant-specific characteristics from each infant and the infant-specific behavioral characteristics of the environmental-specific characteristics, and

(2) The baby-specific stimulation data, the baby-specific response data and the baby-specific personal data for each baby,

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant in accordance with the infant pacifying use case for each infant; and

Generating, by the processor, a training data set for the infant pacifying use case based at least in part on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Said time-dependent infant-specific characteristics of each infant, and

(D) Said time-dependent environmental specific characteristics of each infant, and

(A) At least one recommendation for each infant,

(B) At least one action for each infant, or

(C) Both of which are located in the same plane.

14. The method of claim 13, further comprising training, by the processor, at least one infant-specific behavioral state-detection machine learning model with the training data set unique to the infant pacifying use case.

15. The method of claim 13, wherein the at least one action for pacifying each infant is selected from the group consisting of: playing pacifying music, applying vibration via a vibration unit, projecting pacifying images, and playing sleep-aiding music.

16. A system, the system comprising:

A nonvolatile memory;

Wherein the at least one database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

A plurality of sensors that output sensor data;

A temperature controller; and

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

17. The system of claim 16, wherein the at least one processor is further configured to receive user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

18. The system of claim 17, wherein the at least one processor is further configured to transmit the instructions based on the at least one indication and the user instructions from the graphical user interface.

19. The system of claim 16, wherein a sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

20. The system of claim 16, further comprising a vibration unit operably coupled to the at least one infant, and wherein the at least one processor is further configured to transmit instructions to the vibration unit that cause the vibration unit to apply vibrations to the infant when the at least one infant is hungry.

21. A method, the method comprising:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Receiving, by the processor, from the at least one infant-specific behavioral state-detection machine learning model, an output from the at least one infant-specific behavioral state-detection machine learning model indicative of at least one indication of hunger of the at least one infant;

Transmitting, by the processor, the sensor data, a reminder of anxiety about the at least one infant, a reminder of feeding the at least one infant, the sensor data, or both, over a communication network to at least one communication device of at least one user associated with the at least one infant; and

Transmitting, by the processor, instructions that cause a temperature controller to change a predefined temperature of at least one food item in preparation for feeding the at least one infant.

22. The method of claim 21, further comprising receiving, by the processor, user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

23. The method of claim 22, further comprising transmitting, by the processor, the instruction based on the at least one indication and the user instruction from the graphical user interface.

24. The method of claim 21, wherein a sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

25. A method, the method comprising:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environment-specific features of each infant of the plurality of infants unique to an infant feeding instance from the infant-specific physiological data, the image data, the audio signal data, the sensor data, and the infant-specific personal data for each infant;

Executing, by the processor, a time-driven pipeline software module configured to calculate, from the infant-specific features and the environment-specific features of each infant that are unique to the infant feeding instance, a time-dependent infant-specific feature and a time-dependent environment-specific feature of each infant that characterize the progress of the feature over time based at least in part on a time sequence generated from the infant-specific features and the environment-specific features;

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant in accordance with the infant feeding profile for each infant; and

Generating, by the processor, a training data set for the infant feeding regime based at least in part on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Said time-dependent infant-specific characteristics of each infant, and

(A) The at least one recommendation for each infant,

(B) The at least one action for each infant, or

(C) Both of which are located in the same plane.

26. The method of claim 25, further comprising training, by the processor, at least one infant-specific behavioral state-detection machine learning model with the training data set unique to the infant feeding instance.

27. The method of claim 38, wherein the at least one action for the at least one infant comprises warming at least one food item.

28. A system, the system comprising:

A nonvolatile memory;

Wherein the at least one database comprises:

(Iv) Baby-specific personal data for each baby in the plurality of babies;

wherein the optical subsystem is configured to perform at least one of:

an audio system comprising a microphone, a speaker, or both;

wherein the audio system is configured to perform at least one of:

(Ii) Generating at least one sound for the at least one infant by the speaker;

A plurality of sensors that output sensor data;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Wherein the at least one infant-specific educational machine learning model is trained using a dataset that is based at least in part on the infant-specific stimulus data, the infant-specific response data, the infant-specific personal data, the plurality of infant-specific educational programs, or any combination thereof associated with the plurality of infants;

wherein the outputting comprises:

(i) The at least one infant understands or does not understand at least one indication of the at least one visual image and the at least one sound associated with the at least one visual image according to the at least one infant-specific educational plan for the at least one infant, and

29. The system of claim 28, wherein the at least one processor is further configured to receive user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

30. The system of claim 29, wherein the at least one processor is further configured to perform at least one of the following based on the at least one indication, the at least one infant-specific educational recommendation, and the user instructions:

31. The system of claim 28, wherein a sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

32. A method, the method comprising:

Wherein the database comprises:

(Iii) Baby-specific personal data for each baby in the plurality of babies;

Transmitting, by the processor, instructions to the projection device to project at least one visual image to the at least one infant based on an infant-specific education program from the plurality of infant-specific education programs for the at least one infant;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data of the at least one infant,

(Ii) Spatial body temperature distribution data of the at least one infant,

(Iii) The heartbeat signal data of the at least one infant,

(Iv) Baby movement data of the at least one baby, and

(V) Infant speech classification data for the at least one infant;

Inputting, by the processor, the image data, the audio signal data, the sensor data, the at least one visual image, the at least one sound, the infant-specific physiological data, the infant-specific personal data associated with the at least one infant to at least one infant-specific educational machine learning model;

Wherein the at least one infant-specific educational machine learning model is trained using a dataset based at least in part on the infant-specific stimulus data, the infant-specific response data, and the infant-specific personal data associated with the plurality of infants;

receiving, by the processor, output from the at least one baby-specific educational machine learning model;

wherein the outputting comprises:

Transmitting, by the processor, the at least one indication, the at least one baby-specific educational recommendation, the sensor data, or any combination thereof, over a communication network to at least one communication device of at least one user associated with the at least one baby; and

33. The method of claim 32, further comprising receiving, by the processor, user instructions from the at least one user through a graphical user interface on the at least one communication device associated with the at least one user.

34. The method of claim 33, further comprising performing, by the processor, at least one of the following based on the at least one indication, the at least one infant-specific educational recommendation, and the user instructions:

35. The method of claim 32, wherein a sensor from the plurality of sensors is selected from the group consisting of: thermal imagers, infrared (IR) cameras, lidar devices, and Radio Frequency (RF) devices.

36. A method, the method comprising:

Wherein the database comprises:

(Ii) Infant specific stimulus data for a plurality of infant specific stimuli provided to each infant of the plurality of infants,

(Iii) Infant-specific response data of a plurality of infant-specific responses of each infant acquired in response to the infant-specific stimulus data provided to each infant, and

(Iv) Baby-specific personal data for each baby in the plurality of babies;

Determining, by the processor, from the infant-specific stimulation data and the infant-specific response data for each infant:

(i) Sensor data from a plurality of sensors acquired while monitoring each infant of the plurality of infants,

(Ii) Image data from the image device acquired while each baby is monitored,

(Iv) At least one sound associated with the at least one visual image played to each infant, and

(V) Audio data from the microphone acquired while monitoring each baby;

Wherein the infant-specific physiological data comprises:

(i) The respiration rate signal data for each infant,

(Ii) Spatial body temperature distribution data for each infant,

(Iii) The heartbeat signal data for each infant,

(Iv) Baby movement data for each baby, and

(V) Infant speech classification data for each infant;

Executing, by the processor, a physical algorithm module that generates infant-specific features and environment-specific features of each infant of the plurality of infants unique to an infant education use-case from the infant-specific physiological data, the image data, the audio data, the sensor data, the at least one visual image, the at least one sound, and the infant-specific personal data of each infant;

Executing, by the processor, a time-driven pipeline software module configured to calculate, from the baby-specific features and the environment-specific features of each baby unique to the baby education use case, a time-dependent baby-specific feature and a time-dependent environment-specific feature of each baby characterizing feature progression over time based at least in part on a time sequence generated from the baby-specific features and the environment-specific features;

Executing, by the processor, a feedback environment generator using the digital twinning model for each infant to output at least one recommendation, at least one action, or both to change a behavioral state of each infant in accordance with the infant educational use-case for each infant; and

Generating, by the processor, a training data set for the infant education use-case based at least in part on:

(1) Input features from each infant of the plurality of infants, comprising:

(A) The infant-specific characteristics of each infant,

(B) The environmental specific characteristics of each infant,

(C) Said time-dependent infant-specific characteristics of each infant, and

(A) The at least one recommendation for each infant,

(B) The at least one action for each infant, or

(C) Both of which are located in the same plane.

37. The method of claim 36, further comprising training, by the processor, at least one infant-specific behavioral state-detection machine learning model with the training data set unique to the infant education use case.

38. The method of claim 36, wherein the at least one recommendation for the at least one infant is selected from the group consisting of: changing the at least one visual image, changing the at least one sound, modifying the at least one infant-specific education program, and continuing to execute the infant-specific education program.