JP2023537681A - Systems and methods for extended neurological rehabilitation - Google Patents

Abstract

A system and method for enhanced neurological rehabilitation (ANR) of a patient is disclosed. The ANR system generates rhythmic auditory stimulation (RAS) and visual augmented reality (AR) scenes that are synchronized according to a common beat tempo and output to the patient during the treatment session. Sensors worn by the patient capture biomechanical data related to repetitive movements performed by the patient in synchronization with the AR visual content and RAS. A critical thinking algorithm analyzes the sensor data to determine the spatial and temporal relationship of the patient's movements to visual and audio components to determine the patient's level of attunement and progress toward clinical/training goals. Further, a 3D AR modeling module configures the processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the determined level of entrainment and whether the training goal has been achieved. [Selection diagram] Figure 32

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to U.S. Provisional Patent Application No. 63/054,599 to McCarthy et al. U.S. Provisional Patent, entitled "SYSTEM AND METHOD FOR NEUROLOGICAL REHABILITATION," filed Apr. 14, 2016, which claims the benefit and priority of said provisional application on the basis of the specification; U.S. Patent No. 10,448, entitled "System and Method for Neurological Rehabilitation," issued Oct. 22, 2019, claiming priority from Application No. 62/322,504; U.S. patent application Ser. No. 16/569,388 to McCarthy et al., "Systems and Methods for Neurological Rehabilitation," which is a continuation of U.S. Pat. Each application and patent specification is hereby incorporated by reference as if set forth herein in its entirety.

The present disclosure relates generally to systems and methods for rehabilitation of users with physical disabilities by administering music therapy.

Many controlled studies over the last decade have emphasized the clinical role of music in neurological rehabilitation. For example, rigorously controlled music therapy is known to be able to directly enhance cognition, movement and language. The process of listening to music enhances brain activity in many ways, stimulating extensive bidirectional networks between brain regions associated with attention, semantic processing, memory, cognition, motor function, and emotional processing.

Clinical data support that music therapy enhances memory, attention, executive function, and mood. A PET scan study of neural mechanisms with background music revealed that pleasant music was associated with the ventral striatum, nucleus accumbens, amygdala, insula, hippocampus, hypothalamus, ventral tegmental area, anterior cingulate area, orbitofrontal cortex, and the ventromedial prefrontal cortex. The ventral tegmental area produces dopamine and connects directly to the amygdala, hippocampus, anterior cingulate area, and prefrontal cortex. The mesolimbic system, which can be activated by music, plays an important role in mediating arousal, emotion, reward, memory attention, and executive function.

Neuroscience research has revealed how the fundamental organizational processes that form musical memories share mechanisms with non-musical memory processes. The basis for phrase classification, hierarchical abstraction, and musical patterns bears direct parallels with the principle of temporal chunking in non-musical memory processes. This suggests that music-activated memory processes can replace and enhance non-musical memory processes.

Accordingly, there remains a need for improved devices, systems, and methods for protecting the use of user identities and securely providing personal information.

In one aspect of the disclosed subject matter, a system is provided for extended neurological rehabilitation of a patient. The system comprises a computer system having a processor configured with software modules containing machine-readable instructions stored on a non-transitory storage medium.

AA/AR modeling software module, when executed by the processor, configures the processor to generate augmented reality (AR) visual content and rhythmic auditory stimuli (RAS) for output to the patient during the treatment session Contains modules. Specifically, RAS includes beat signals output at the beat tempo, and AR visual content includes visual elements that move in a defined spatial and temporal order based on the beat tempo.

The system further comprises an input interface in communication with the processor for receiving patient data in real-time including AR visual content and patient time-stamped biomechanical data associated with repetitive movements performed by the patient in time with the RAS. . Specifically, biomechanical data are measured using sensors associated with the patient.

The software module further analyzes the time-stamped biomechanical data to determine the temporal relationship of the patient's repetitive motion to the beat signal output at the visual component and beat tempo, and to determine the level of entrainment to the target parameter. It includes a Critical Thinking Algorithm (CTA) module that configures the processor to make decisions. Also, the AA/AR modeling module further configures the processor to synchronize the AR vision and RAS output to the patient and dynamically adjust based on the determined entrainment level.

According to a further aspect, a method is provided for extended neurological rehabilitation of patients with physical disabilities. The method is implemented on a computer system having a physical processor configured with machine-readable instructions that, when executed, perform the method.

The method includes providing rhythmic auditory stimuli (RAS) that are output to the patient via an audio output device during a therapy session. Specifically, the RAS contains beat signals that are output at the beat tempo.

The method also includes generating augmented reality (AR) visual content that is output to the patient via the AR display device. Specifically, the AR visual content includes visual elements that move in a defined spatial and temporal order based on the beat tempo and are output in sync with the RAS. The method further includes instructing the patient to perform repetitive movements in time with the beat signal of the RAS and the corresponding movement of the visual elements of the AR visual content.

The method further includes receiving, in real-time, patient data including time-stamped biomechanical data of the patient associated with repetitive movements performed by the patient in concert with the AR visual content and RAS. Specifically, biomechanical data are measured using sensors associated with the patient.

The method further analyzes the time-stamped biomechanical data to determine the temporal relationship of the patient's repetitive motion to the beat signal output in response to the visual component and the beat signal to determine entrainment margin. including the step of Further, the method includes dynamically adjusting to synchronize the AR visual content and the RAS output to the patient and based on the determined entrainment margin that does not meet the predetermined entrainment margin; continuing the therapy session with the visual content and the RAS.

The foregoing and other objects, features, and advantages of the apparatus, systems, and methods described herein will become apparent from the following description of specific embodiments thereof that are illustrated in the accompanying drawings. . The drawings are not necessarily to scale, emphasis rather being placed upon illustrating the principles of the apparatus, systems and methods described herein.

1 illustrates a system for treating a user with music therapy, according to an exemplary embodiment of the disclosed subject matter; FIG. 1 illustrates some components of a system for rehabilitating a user by administering music therapy, according to an exemplary embodiment of the disclosed subject matter; FIG. 1 is a circuit diagram of a sensor for measuring biomechanical motion of a patient, according to an exemplary embodiment of the disclosed subject matter; FIG. FIG. 1 illustrates some components of a system in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 11 illustrates an exemplary display of components of a system for rehabilitating a user by administering music therapy, according to an exemplary embodiment of the disclosed subject matter; FIG. 1 is a flow diagram of one implementation of an analysis process, according to an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 is a flow diagram of one implementation of a process in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 4 is a flow diagram of one implementation of a process in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 4 is a flow diagram of one implementation of a process in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 4 is a flow diagram of one implementation of a process in accordance with an exemplary embodiment of the disclosed subject matter; 4 is a time graph showing music and patient body movements, according to an exemplary embodiment of the disclosed subject matter; FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. FIG. 4 illustrates patient response in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 1 illustrates one implementation of a patient gait training technique, according to an exemplary embodiment of the disclosed subject matter; 4 illustrates one implementation of a patient neglect training technique, according to an exemplary embodiment of the disclosed subject matter; 4 illustrates one implementation of a technique for patient intonation training, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a patient musical stimulation training technique, according to an exemplary embodiment of the disclosed subject matter; 4 illustrates one implementation of a patient gross motor training technique, according to an exemplary embodiment of the disclosed subject matter. 4 illustrates one implementation of a patient grip strength training technique, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a technique for patient speech cue training, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a minimally conscious patient training technique, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a patient attention training technique, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a patient attention training technique, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a patient attention training technique, according to an exemplary embodiment of the disclosed subject matter; 4 illustrates one implementation of a patient dexterity training technique, according to an exemplary embodiment of the disclosed subject matter; 4 illustrates one implementation of an oral movement training technique for a patient, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates one implementation of a patient breathing exercise technique, according to an exemplary embodiment of the disclosed subject matter; 1 illustrates a system for augmented neurological rehabilitation, recovery, or maintenance (“ANR”) for administering therapy to a patient, according to an exemplary embodiment of the disclosed subject matter; FIG. FIG. 33 is a graphical visualization of measured parameters, system responses, and target/target parameters during a treatment session conducted using the ANR system of FIG. 32 in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 33 is a graph depicting exemplary results related to metabolic changes produced by a training session conducted using the ANR system of FIG. 32 in accordance with an exemplary embodiment of the disclosed subject matter; 4 is an exemplary Augmented Reality (AR) display generated by an ANR system for display to a patient during a treatment session in accordance with an exemplary embodiment of the disclosed subject matter; 4 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session in accordance with an exemplary embodiment of the disclosed subject matter; 4 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 11 illustrates one implementation of a technique for gait training by providing enhanced audio and visual stimuli to a patient, according to an exemplary embodiment of the disclosed subject matter; FIG. 1 is a mixture of systems and processes conceptually illustrating an ANR system configured to perform a gait training method, according to an exemplary embodiment of the disclosed subject matter; FIG. 39 is a mixed system and process diagram conceptually illustrating components of an augmented audio (AA) unit of the ANR system of FIG. 38 in accordance with an exemplary embodiment of the disclosed subject matter; FIG. 4 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session in accordance with an exemplary embodiment of the disclosed subject matter; 4 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session in accordance with an exemplary embodiment of the disclosed subject matter;

The present invention relates generally to systems, methods, and apparatus for implementing a dynamic closed-loop rehabilitation platform system that observes and directs human behavioral and functional changes. Such changes are made in language, movement, and cognition temporally triggered by musical rhythm, harmony, melody, and intensity cues.

In various embodiments of the present invention, a dynamic closed-loop rehabilitation platform music therapy system 100, shown in FIG. It comprises a system 108 and a music therapy center 110 . As described in more detail below, sensor elements, edge processing elements, collection element machine learning processes, and music therapy centers can be provided on various hardware elements. For example, in one embodiment, the sensor elements and edge processing elements may be attached to or worn by the patient. In such embodiments, the collection element and music therapy center may be provided on a portable device. In such embodiments, the analysis system may be located on a remote server.

Sensor System Throughout the description herein, the term "patient" is used to refer to an individual undergoing music therapy treatment. The term "therapist" is used to refer to an individual who administers music therapy treatment. In some embodiments, a patient can interact with the system described herein without a treating therapist present.

Sensor element 102 provides sensed biomechanical data about the patient. In some embodiments, the sensor elements include: (1) a wearable wireless real-time motion detection device or IMU (Inertial Measurement Unit); (3) a wearable wireless real-time electromyography (EMG) device such as sensor 208; and (4) a real-time wireless near-infrared (NIR) video capture device such as imaging device 206 (FIG. 4). ).

As shown in FIG. 2, the systems and methods described herein are used in connection with the treatment of gait disorders in patients. Thus, the exemplary sensor 200 can be a combined multi-zone plantar pressure/six degrees of freedom motion capture device. Sensor 200 records the patient's foot pressure and 6-DOF motion profile while the patient is walking during a music therapy session. In some embodiments, the foot pressure/6 DOF motion capture device has a variable recording time interval at a sampling rate of 100 Hz for a foot pressure profile containing 1-4 zones, 100-400 pressure data points are obtained.

Sensor 200 is a foot pressure pad that includes a heel pad (to measure one-zone pressure, e.g., heel-contact pressure) to an insole-wide pad (to measure four-zone pressure). 202. Pressure measurements are made by detecting resistance changes in the material of the transducer as a result of compression by the patient's weight transmitted to the foot. Such foot pressure maps are obtained at sampling intervals or at specific time points during the music therapy session.

The sensor 200 is a six-degree-of-freedom micro-electromechanical system (MEMS) that measures three-dimensional linear accelerations _Ax , _Ay , _Az , and rotational motion such as pitch (pitch), yaw, and roll (roll). ) may include a 6D motion capture device 204 that detects changes in motion via sensors in the base. Sampling at 100 Hz produces 600 motion data points per second. These foot motion captures are obtained at sampling intervals or at specific points in time during the music therapy session.

Multizone pressure detection using a 6-DOF motion capture device enables the tracking of spatial and temporal locomotion dynamics that can be mapped during locomotion. A circuit diagram of sensor 200 is shown in FIG.

As shown in FIG. 4, in the system view, patient P uses two foot sensors 200, one for each of the feet designated right 200R and left 200L. In the exemplary embodiment, the right foot sensor 200R receives time-stamped internal measurement device data and heel-stamped data on a first channel, e.g., channel 5 in the IEEE 802.15.4 Direct Spread Spectrum (DSSS) RF band. Wireless transmission of ground pressure data. Left foot sensor 200L wirelessly transmits time-stamped internal measurement device data and heel pressure data on a second channel, e.g., channel 6 in the IEEE 802.15.4 Direct Spread Spectrum (DSSS) RF band. introduce. As described below, a tablet or laptop computer 220 optionally used by therapist T uses a first channel and a second channel, e.g. It has a wireless USB hub containing two IEEE 802.15.4 DSSS RF transceivers tuned to channel 6. Portable wireless trigger 250 is used to start and stop video and/or to mark and index the time stream as discussed in more detail below.

The video analysis domain can be used to extract patient semantic and event information about a treatment session. Patient movements and interactions are components in therapy that influence the context and management of therapy. In some embodiments, one or more imaging devices 206 (see FIG. 4), such as video cameras, are used with time-synchronized video feeds. Suitable video can be incorporated into the system to capture patient movement, but near-infrared (NIR) video capture is beneficial to protect patient privacy and reduce video data to process. A NIR video capture device captures NIR video images of a patient's body, such as the patient's torso or limbs. Additionally, the NIR video capture device captures real-time dynamic gait characteristics of the patient as a function of the music therapy session. In some embodiments, the video is shot with a stationary camera, the background is subtracted and the foreground pixels are split in the video.

As shown in FIG. 4, one or more video cameras 206 are activated by a tablet or laptop application when a treatment session begins. The video camera 206 can be stopped or activated with a portable wireless trigger unit 250 by the therapist. This allows the creation of labeled and time-stamped indices in the captured biomechanical sensor data and video data streams.

In some embodiments, a wearable wireless real-time electromyography (EMG) device 208 can be worn by the patient. EMG sensors provide a profile of the entire bipedal gait for the firing of key muscles for locomotion. Such sensors provide data on the exact time when a muscle fires.

Edge Processing In some embodiments, edge processing is performed at the sensor 200 and sensor data is captured from the IMU and pressure sensors. The sensor data is filtered and grouped into various array sizes, further processed into frames reflecting the extracted attributes and features, and these frames are collected, e.g., wirelessly, on a tablet or laptop. is sent to device 106. Alternatively, it is understood that raw biomechanical sensor data obtained from sensor 200 can be transferred to a remote processor for collection to perform edge processing functions.

The wearable sensors 200, 208 and video capture device 206 produce sensor data streams that are synthetically processed to facilitate biomechanical feature extraction and classification. Sensor fusion combines outputs from multiple sensors that capture common events and captures results better than any single constituent sensor input.

By capturing patient activity in the context of music therapy, interactions applied to music therapy and in developing patient-specific and generalized formal measures of performance and effectiveness of music therapy. and are formalized. By extracting and analyzing video features, it is possible to capture semantic and high-level information about patient behavior.

In processing the video, a learned background subtraction method is used to create a background model that takes into account variations in lighting conditions and occlusions in the physical location where music therapy is performed. The result of background subtraction is a binary map of the foreground with a series of 2D contour blobs in the foreground. Thus, the video is split into individual image frames for future image processing and sensor fusion. Video information is provided with additional metadata by fusing edge-processed sensor data from the IMU, foot pressure pads, and EMG sensors. Sensor data can be time-synchronized with other data using RF triggers. The data can be sent directly to the collector, stored in internal board memory, or analyzed at the edge running the OpenCV library.

The edge processor 104 can be a microprocessor, for example, foot pressure/6DOF enabling fast multi-zone scanning at rates of 100-400 complete foot pressure/6DOF motion profiles per second. It can be, for example, a 32-bit microprocessor built into a motion capture device.

A foot pressure/6DOF motion capture device collects foot pressure/6DOF motion profile data for real-time gait analysis, resulting in feature extraction and classification. In some embodiments, the foot pressure/six degrees of freedom motion capture device includes a microcontroller unit (MCU), continuous operator processing (COP), general purpose input/output (GPIO), serial peripheral interface (SPI), and interrupt Initialize Request (IRQ), Microcontroller Unit Initialization (MCUInit), General Purpose Input/Output Initialization (GPIOInit), Serial Peripheral Interface Initialization (SPIInit), Interrupt Request Acknowledge Initialization (IRQInit), Interrupt Request Acknowledge ( IRQACK), Serial Peripheral Interface Driver Read (SPIDrvRead), and IRQPinEnable to set the desired RF transceiver clock frequency. MCUInit is the master initialization routine that stops the MCU watchdog and sets the timer module to use the bus clock (BUSCLK) with a prescale of 32 as the reference.

The state variable gu8RTxMode is set to SYSTEM_RESET_MODE and the routines GPIOInit, SPIInit, IRQInit are called. The state variable gu8RTxMode is set to RF_TRANSCEIVER_RESET_MODE and IRQFLAG is checked to see if IRQ is asserted. The RF transceiver interrupt is first cleared using SPIDrvRead and the RF transceiver is checked for ATTN IRQ interrupts. Finally, for MCUInit, PLMEPhyReset to reset the physical MAC layer, IRQACK (to ACK pending IRQ interrupts), and IRQPinEnable to pin Enable and IE and IRQ CLR on the negative edge of the signal. A call is made to

The foot pressure/6DOF motion sensor 200 waits for a response from the foot pressure/6DOF motion collection node, eg, 250 milliseconds, to determine whether a default full foot pressure scan is to be performed or the mapped foot pressure. Determines if scanning is started. For mapped foot pressure scans, the foot pressure/six degrees of freedom motion acquisition node sends the foot pressure scan mapping configuration data to the appropriate electrodes.

One aspect of the analysis pipeline is a feature set that defines those captured sensor values and the resulting sensor-fused values that are used to create a feature vector and define the input data structure for analysis. It's an engineering process. Typical values are Ax(i), Ay(i), Az(i), Ph(i), where i is the ith sample and Ax(i) is the x direction transverse to the foot sensor. Ay(i) is the forward y-direction acceleration with respect to the foot sensor, Az(i) is the upward z-direction acceleration with respect to the foot sensor, and Ph(i) is heel contact pressure. Sensor values are shown in Table 1.

In some embodiments, the sensor fusion approach uses the heel pressure values Ph(i) to "gate" the analysis of the following exemplary features to derive the windows of data described below. use. For example, as shown in Table 2 below, "onset" (start) can be determined based on heel pressure above a threshold signifying heel strike, and "stop" can be determined based on heel-off. The determination is based on heel pressure being below a threshold, which means that the It is understood that heel contact pressure is one example of a parameter that can be used for a "gated" analysis. In some embodiments, "gating" is determined using IMU sensor data, video data, and/or EMG data.

Higher-level features are calculated from fused sensor values, such as the exemplary values shown in Table 3.

The system described herein provides the ability to "gate" or provide a "window" on the patient's biomechanical data. Gating biomechanical data is beneficial for repetitive patient motion, such as repetitive strides while the patient is walking. Sensor data from one or more sources, such as pressure sensors, IMU sensors, video data, EMG data, etc., are used to identify repeating cycles of motion over time. For example, when a patient walks, the foot pressure repeatedly increases and decreases as the patient's foot touches the ground and then lifts off the ground. Similarly, foot velocity increases as the foot moves forward and decreases to zero while the foot is on the ground. As a further example, the Y position, or height, of the patient's foot cycles between a low position (above the ground) and a high position (approximately mid-stride). A "gating" approach identifies repeating cycles, or "windows", in such data. If the patient is walking, the cycle repeats with each step. A particular pattern is repeated in each cycle, although there may be variation from cycle to cycle, eg, step to step. Choosing the onset time (start time) for each cycle involves placing an identifiable point (maximum or minimum) of the biomechanical parameter. The selection of parameters for the onset time is chosen based on available data. Thus, in some embodiments, the instant at which heel contact pressure exceeds a threshold can be used to delimit the onset time of each cycle. (See, for example, FIG. 5. The pressures 316a, 316b include periodic characteristics. An "onset" may be determined at the moment the pressure crosses a threshold.) can be demarcated if

In some embodiments, the raw frame data is preprocessed to take instantaneous data and "gate" it, e.g., identify a window and analyze the data within that window to identify outliers or Analyze the data, eg exponential analysis, averaging the data over multiple windows. Fusion of sensor data includes both IMU and heel pressure data to provide more accurate onset times for a single stride or other repeated motor unit than using data from a single sensor. make it possible to identify Sensor data captured during one stride is considered a "window" and information extracted from this analysis includes, for example, stride length, number of steps, cadence, time the step occurred, distance traveled. , stance phase/swing phase, double support time, velocity, symmetry analysis (e.g. between left and right leg), lateral swing, shuffling, output vector, lateral acceleration , the step size, the variability of each of these magnitudes, further parameters derived from the above information, and so on. Feature extraction can be processed on a microprocessor chip, such as a 32-bit chip. Capture of wireless synchronous gated biomechanical sensor data, and video data capture capabilities enable the creation of time-series templates.

Data can be indexed by the patient or therapist during the music therapy session. The "gating" function described above is useful for tying a particular stride or step to an exception condition. For example, a therapist may observe certain exceptional conditions or behaviors (such as anomalies or incidents) in a patient's movements. The indexing feature allows the therapist to use a portable tablet or laptop user interface, such as the wireless trigger unit 250 shown in FIG. It is possible to start. Annotations can be made that include timestamps and comments such as when a patient "stumbles" while walking. Such indexing facilitates the creation of time series templates. These time-series templates are used to review treatment session events and machine learning algorithms such as non-linear multi-layer perceptrons (NLMLP), convolutional neural networks (CNN), recurrent neural networks (RNN) with long short-term memory (LSTM), etc. It is studied for the development of time series templates for training.

In one embodiment, a communication protocol is provided to transfer sensor data (eg, at sensor 200 ) from edge processing 104 to collector 106 . See Table 4 below. In some embodiments, RF timed out if the connection was idle for more than 100ms.

In one embodiment, the scanning of the foot pressure sensor zones is performed by the FootScan routine, which initializes the FootDataBufferIndex to enable the MCU direction mode for output [PTCDD_PTCDDN=Output] and the associated port line. is made low [PTCD_PTCD6=0]. When a foot pressure sensor zone is activated based on the foot pressure zone scan map, the foot pressure sensor zone coupled to the MCU's analog signal port is sampled and the current voltage measurements are converted to digital form (time zone foot pressure).

Some variables, such as FootDataBufferIndex and IMUBufferIndex, are used in the IEEE 802.15.4 RF packet, gsTxPacket. Used to prepare gau8TxDataBuffer[ ]. RF packets are sent using the RFSendRequest (and gsTxPacket) routines. This routine checks to see if gu8RTxMode is set to IDLE_Mode and uses gsTxPacket as a pointer to call the RAMDrvWriteTx routine to read the contents of the RF transceiver's TX packet length register. Call SPIDrvRead. With these contents, the mask length setting is updated, adding 2 to the CRC and adding 2 to the code bytes.

SPISendChar is called to send the second code byte, the 0x7E byte, and then SPIWaitTransferDone is called again to verify that the transfer has taken place. Once these code bytes have been sent, the rest of the packet is sent using a for loop, where psTxPkt→u8DataLength+1 is the number of iterations for the sequential series for SPISendChar, SPIWaitTransferDone, SPIClearReceiveDataReg. Upon completion, the RF transceiver is loaded with packets to transmit. ANTENNA_SWITCH is set to transmit, LNA_ON mode is enabled, and finally a call to RTXENAssert is made to actually transmit the packet.

Collector The primary function of the collector 106 is to capture data from the edge processing 104, transfer data to the analysis system 108, receive processed data from the analysis system 108, and send data to the analysis system 108, as described below. transfer to the music therapy center 110; In some embodiments, the collector 106 provides control functions, e.g., a user interface for logging in, configuring the system, interacting with users, and a display unit for visualizing/displaying data. Prepare. Collector 106 may include lightweight analytics or machine-learned algorithms for classification (eg, lateral tremor, asymmetry, instability, etc.).

Collector 106 receives body, motion and localization data from edge processor 104 . Data received at the collector 106 can be raw or processed at the edge 104 before being transferred to the collector. For example, collector 106 receives fused sensor data that undergoes "windowing" and feature extraction. The transferred data is two levels of data: (1) RF packets transmitted from the right/left foot sensors listed in Table 1, and (2) higher level attributes listed in Tables 2 and 3. and RF packets from the right/left foot sensors containing the features. Collector 106 stores data locally. In some embodiments, the collector 106 classifies motion from the data received, e.g., a locally stored model (pre-downloaded from the analysis system), or a model transmitted to the analysis system and this data. Compare to classify. The collector may comprise a display unit for visualizing/displaying the data.

In some embodiments, collector 106 runs on a local computer with memory, processor, and display. Exemplary devices on which collectors are installed may include augmented reality (AR) devices, virtual reality (VR) devices, tablets, mobile devices, notebook computers, desktop computers, and the like. FIG. 2 shows a portable device 220 having a display 222 and performing collection functions. In some embodiments, connection parameters for transferring data between the patient sensor and the collector include using Windows Device Manager (e.g., baud rate: 38400, databits: 8, parity: none, stop Bits: 1) Enabled. In some embodiments, the collector 106 comprises a processor that is held or worn by the music therapy patient. In some embodiments, the collector 106 comprises a processor that is remote from the music therapy patient, carried by the therapist, and connected with the music therapy patient via a wireless or wired connection.

In one embodiment, the foot pressure/6DOF motion collection node captures RF transmitted data packets containing real-time foot pressure/6DOF motion profile data from the foot pressure/6DOF motion capture device. . This is initiated by the Foot Pressure/6DOF Motion Acquisition node creating an RF packet receive queue driven by a callback function for RF transceiver packet receive interrupts.

When an RF packet is received from the foot pressure/6DOF motion capture device 200, first to determine if this is from a new foot pressure/6DOF motion capture device or an existing one. A check is made. If this is from an existing foot pressure/6DOF motion capture device, the RF packet sequence number is checked to determine continuous synchronization before further packet analysis. If this is a foot pressure capturing 6DOF motion device, a Foot Pressure/6DOF motion capture device context state block is created and initialized. The context state block contains information, eg foot pressure profile.

Above this RF packet session level process for internode communication is the analysis of the RF packet data payload. This payload contains a foot pressure profile based on current variable pressures that follow six degrees of freedom motion. It is constructed as follows.
|0x10|start|F1|F2|F3|F4|Ax|Ay|Az|Pi|Yi|Ri|XOR checksum|

The IEEE 802.15.4 standard specifies a maximum packet size of 127 bytes, leaving 80 bytes for payload as the Time Synchronized Mesh Protocol (TSMP) reserves 47 bytes for operation. IEEE 802.15.4 is adapted for radio frequency (RF) transceivers in the 2.4 GHz industrial, scientific and medical (ISM) band.

The RF module contains a complete 802.15.4 physical layer (PHY) modem designed for the IEEE 802.15.4 wireless standard, supporting peer-to-peer, star and mesh networks. The RF modules are combined with MCUs to create the required wireless RF data links and networks. The IEEE 802.15.4 transceiver supports encoding/decoding of 250 kbps O-QPSK data and full spread spectrum on a 5.0 MHz channel.

In some embodiments, control, read status, write data, and read data are through the RF transceiver interface port of the detection system node device. The detection system node device's MPU accesses the detection system node device's RF transceiver through an interface "transaction" in which multiple byte-long bursts of data are sent over the interface bus. Each transaction is at least 3 bursts long, depending on the type of transaction. A transaction is always a read or write access to a register address. The associated data for any one register access is always 16 bits long.

In some embodiments, RF transceiver control and data transfer of the foot pressure/6DOF motion acquisition node is implemented using a serial peripheral interface (SPI). The normal SPI protocol is based on 8-bit transfers, but the RF transceiver of the Foot Pressure/6DOF motion acquisition node imposes a higher level transaction protocol based on multiple 8-bit transfers per transaction. One SPI read or write transaction consists of an 8-bit header transfer followed by two 8-bit data transfers.

The header indicates the access type and register address. The following bytes are the read or write data. SPI also supports recursive "data burst" transactions where additional data transfers can occur. The recursive mode is primarily intended for packet RAM access and fast setup of RF for foot pressure/6 DOF motion acquisition nodes.

In some embodiments, all foot pressure sensor zones are scanned continuously and the entire process is repeated until they are in a reset state or an inactive power down mode. Six degrees of freedom motion is captured by a serial UART interface from the MCU to an inertial measurement unit (IMU). The sampling rate for all detection dimensions, ie Ax, Ay, Az, pitch, yaw, roll, is 100Hz-300Hz and the sampled data is stored in IMUBuffer[].

SPIDryWrite is called to update the TX packet length field. Next, SPIClearReceiveStatReg is called to clear the status register, followed by SPIClearReceiveDataReg to clear the receive data register, readying the SPI interface for reading or writing. Once the SPI interface is ready, SPISendChar is called to send a character of 0xFF representing the first code byte, and SPIWaitTransferDone is called to verify that the transfer has occurred.

FIG. 5 is an exemplary output 300 that may be provided on the display 222 of the portable device. For example, if the patient's gait is being treated, the display output 300 may include a portion 302 for the right foot and a portion 304 for the left foot. As a function of time, the display for the right foot includes accelerations A _x 310a, A _y 312a, and A _z 314a, and foot pressure 316a. Similarly, the display for the left foot includes accelerations A _x 310a, A _y 312a, and A _z 314a, and foot pressure 316a.

Classification is understood to be the correlation of data (eg, sensor-fused data, feature data, or attribute data) to real-world events (eg, patient activity or temperament). Typically, the classification is created and performed on analysis system 108 . In some embodiments, collector 106 has a local copy of some "templates." Thus, incoming sensor data and feature extracted data can be sorted in a collector or analysis system.

Context refers to the circumstances or facts that form the background of an event, statement, situation, or thought. Context-aware algorithms probe the 'who', 'what', 'when' and 'where' in relation to the environment and time in which the algorithm is executed on the particular data. Some context-aware actions include identity, location, time, and activity being performed. In using contextual information to create deterministic actions, a contextual interface is created between the patient, the environment and the music therapy session.

The context of a patient's response to a music therapy session can include layers of algorithms that interpret the fused sensor data and infer higher level information. These algorithms refine the context of the patient's response. For example, the patient's biomechanical walking sequence is analyzed as it relates to a particular portion of the music therapy session. In one example, "lateral tremor" is a classifier of interest. Therefore, it is determined that the patient's gait is smoother with less lateral tremor.

Analysis system
Analysis system 108, sometimes referred to as a backend system, stores large models/archives and includes machine learning/analysis processing using the models described herein. Some embodiments also provide a web interface for login to view the stored data, and a dashboard. In some embodiments, analysis system 108 is located on a remote server computer that receives data from collector 106 operating on a handheld unit, such as a handheld device or tablet 220 . It is contemplated that the processing power required to perform the analytical and machine learning functions of analytical system 108 may also be located on portable device 220 .

Data is transferred from the collector 106 to the analysis system 108 for analytical processing. As shown in FIG. 6, analysis process 400 includes user interface 402 that receives data from collector 106 . Database storage 404 receives input data from collector 106 for storage. Training data and outputs of analytical processes (eg, ensemble machine learning system 410) can also be stored in storage 404 to facilitate creation and refinement of predictive models and classifiers. A data bus 406 allows data to flow through the analysis process. A training process 408 is performed on the training data to derive one or more predictive models. Ensemble machine learning system 410 utilizes predictive models. The output of ensemble machine learning system 410 is an aggregation of these predictive models. This aggregated output is also used in a classification request using the template classifier 412, such as quiver, symmetry, smoothness, or learned biomechanical parameters such as entrainment, initiation. API 418 connects to collectors and/or music therapy centers. Treatment algorithms 414 and prediction algorithms 416 include multilayer perceptron neural networks, hidden Markov models, radial basis function networks, Bayesian estimation models, and the like.

An exemplary application of the systems and methods described herein is the analysis of a patient's biomechanical gait. The gait sequence is feature extracted to retrieve a set of characteristic features. The presence of these and other features in the captured sensor-fused data informs the context detection algorithm whether the patient's biomechanical gait sequence is valid. Robust context detection is required to capture biomechanical gait sequences, which are then abstracted against a representative population of music therapy patients.

An example of such activity is the patient's location in time and the patient's response to music therapy at that point in time. Recognizing and correlating a patient's response to music therapy allows recognition of specific patterns of patient response to music therapy. By developing a measure of music therapy patient response and correlating this measure to future music therapy patient response, a particular music therapy regime is evaluated and analyzed with respect to performance and effectiveness.

Along with motion detection, a distance measure associated with biomechanical capture of gait determines the trajectory of the patient's path using temporal and spatial variations/deviations between two or more music therapy sessions used for From this sensor-fused data capture, features are extracted and classified to label various key patient responses to therapy. The use of histograms in further analysis of sensor-fused data allows the detection of patterns of response to early music therapy.

Analysis of the sensor-fused data of music therapy sessions initially uses a patient-specific Bayesian inference model using Markov chains. Chain status represents patient-specific response patterns captured from a basic session of music therapy. Inferences are made based on knowledge of the occurrence of the patient's response pattern at each sample interval and the temporal relationship to previous conditions.

Prediction routines and multi-layer perceptron neural networks (MLPNN) use a directed graph node-based model with a top-level root node that reaches subsequent nodes to obtain feature vectors of the patient's sensor-fused data. Predict the requirements for The feature vector of this sensor-fused data contains time-series processed motion data, music signature data, and video image data of particular interest for further processing. In this case, the directed graph looks like a tree drawn upside down, with the leaves at the bottom of the tree and the root as the root node. From each node, the routine can proceed to the left, where left is the left node in the next layer below the top layer where the root node is located, and the left subnode is selected as the next observed node. Alternatively, the routine can proceed to the right, where right is the right node in the next layer below the top layer where the root node is located, whose index is stored in the observed node. based on the values of certain variables that If the value is less than the threshold, the routine proceeds to the left node, otherwise the routine proceeds to the right node. These regions, here left and right, become the prediction space.

The model uses two types of input variables: ordered and categorical. The ordered variable is the value that is compared to the threshold, also stored in the node. Categorical variables are discrete values that are tested and stored in nodes to see if they belong to a particular, restricted subset of values. This variable is applicable to various classifications. For example, mild, moderate, and severe can be used to describe tremors, which are examples of categorical variables. Conversely, a very finely-divided value range or numerical scale can also be used (but numerically) to describe the tremor.

If the categorical variable belongs to the restricted set of values, the routine proceeds to the left node, otherwise the routine proceeds to the right node. At each node, a pair of entities is used to make this decision: variable_index and decision_rule (threshold/subset). This pair is called a partition, and is partitioned according to the variable variable_index.

Once the node is reached, the value assigned to this node is used as the output of the prediction routine. A multi-layer perceptron neural network is built recursively starting from the root node. All training data, feature vectors, and responses are used to split the root node as described above, and the entity, variable_index and decision_rule (threshold/subset) split the prediction region. At each node, the optimal decision rule for the best linear split is found based on the Gini "purity" criterion for classification and the sum of the squared errors of the regression. The Gini coefficient is based on a measure of total variance across classes of a set. A Gini "purity" criterion refers to a small Gini coefficient value, indicating that a node contains predominantly observations from a single class of the desired state.

Once constructed, multi-layer perceptron neural networks can be pruned using cross-validation routines. Some branches of the tree are pruned to avoid overfitting the model. This routine can be applied for independent determination. One of the distinguishing properties of the decision algorithm (MLPNN) described above is the ability to compute the relative determinism and importance of each variable.

Variable importance ratings are used to determine the most frequent interaction types for the patient interaction feature vector. Pattern recognition begins with the definition of a decision space suitable for differentiating different categories of music therapy responses and music therapy events. The decision space can be represented by an N-dimensional graph, where N is the number of attributes or measurements considered to represent music therapy responses and music therapy events. The N attributes constitute a feature vector or signature that can be plotted on a graph. After enough samples have been input, the decision space reveals collections of music therapy responses and music therapy events belonging to various categories, and the decision space is used to associate new vectors with these collections.

The dynamic closed-loop rehabilitation platform music therapy system utilizes several deep learning neural networks to learn and reproduce patterns. In one embodiment, a nonlinear decision space is constructed using an adaptive radial basis function (RBF) model generator. New vectors can be computed using the RBF model and/or using a k-nearest neighbor classifier. FIG. 6 shows the workflow of the machine learning subsystem of the dynamic closed-loop rehabilitation platform music therapy system.

FIG. 7 shows a supervised training process 408 including a number of training samples 502, for example, where the inputs are the features listed in Table 3 above and example outputs are items such as tremor, asymmetry, force, etc. , the degree of these items, the expected change, and the classification of how well the patient is recovering. It is understood that new outputs are learned as part of this process. This process provides higher-level prediction and classification abstractions when applied to different use cases (e.g., various medical conditions, drug combinations, provider notification, adequacy, fall prevention). provide a foundation for innovation. These training samples 502 are run with learning algorithms A1 504a, A2 504b, A3 504c, ..., AN 504n to derive predictive models at M1 506a, M2 506b, M3 506c, ..., MN 506n. be done. Exemplary algorithms include multilayer perceptron neural networks, hidden Markov models, radial basis function networks, and Bayesian inference models.

FIG. 8 shows a number of prediction result data 606a, 606b, 606b, . . . , 606n. An aggregation layer 608 containing, for example, decision rules and votes is used to derive output 610 given multiple predictive models.

The MR ConvNet system has two layers, the first layer is a convolutional layer that supports average pooling. The second layer of the MR ConvNet system is a fully connected layer that supports multinomial logistic regression. Multinomial logistic regression, also called softmax, is a generalization of logistic regression to handle a large number of classes (multiclass). For logistic regression, the labels are binary.

Softmax is a model used to predict the probabilities of various possible outputs. In the following, we assume a multi-class classifier with m distinct classes using the final output layer of softmax.
Y1=Softmax (W11*X1+W12*X2+W13*X3+B1) [1]
Y2=Softmax (W21*X1+W22*X2+W23*X3+B2) [2]
Y3=Softmax (W31*X1+W32*X2+W33*X3+B3) [3]
Ym=Softmax (Wm1*X1+Wm2*X2+Wm3*X3+Bm) [4]
generally,
Y=softmax(W*X+B) [5]
Softmax(X)i=exp(Xi)/(sum of exp(Xj) of j=1 to N) [6]
where Y=classifier output, X=sample input (all scaled (normalized) features), W=weight matrix. Classification is for example asymmetry of the score, for example "moderate asymmetry score, 6 out of 10 (10 with high level of asymmetry to 0 with no asymmetry)", smooth walking (“gait smoothness score is 8 out of 10 (normal)”), and so on. The analysis pipeline is shown in FIG.

Softmax regression allows us to handle a large number of classes beyond two. In logistic regression,
P(x)=1/(1+exp(-Wx))
where W contains the model parameters trained to minimize the cost function. Also, x is the input feature vector,
((x(1), y(1)), ..., (x(i), y(i))) [7]
represents the training set. Multi-class classification uses softmax regression, where instead of 1 and 0 in the binary case, there can be N different values representing the classes. Therefore, for the training set ((x(1), y(1)), . . . , (x(i), y(i))), y(n) is It can be any value in the range.

Next, p(y=N|x;W) is the probability for each value of i=1, . The following shows the softmax regression process mathematically.
Y(x=(p(y=1|x;W),p(y=2|x;W),...,p(y=N|x;W)) [8]
Here, Y(x) is the answer to the hypothesis, and outputs the normalized probability distributions for all classes so that their sum equals 1 given the input x.

The MR ConvNet system convolves all biomechanical data frames in the window as vectors with all biomechanical template filters as vectors and generates responses using an average pooling function that averages the feature responses. The convolution process adds any biases to compute Wx and passes it to the logistic regression (sigmoid) function.

Then, in the second layer of the MR ConvNet system, the subsampled biomechanical template filter responses are moved into a secondary matrix, with each column representing the biomechanical data frame within the window as a vector. Now the softmax regression activation process is started using the following equation.
Y(x)=(1/(exp(Wx)+exp(Wx)+...+exp(Wx))*(exp(Wx), exp(Wx),...,(exp(Wx)) [9 ]

The MR ConvNet system is trained using an optimization algorithm, gradient descent, where the cost function J(W) is defined and minimized.
J(W)=1/j*((H(t(j=1),p(y=1|x;W)+H(t(j=2),p(y=2|x;W)+ . . . +H(t(j),p(y=N|x;W)) [10]
where t(j) is the target class. This averages all cross-entropies over the j training samples. The cross entropy function is
H (t (j), p (y = N | x; W) = - t (j = 1) * log (p (y = 1 | x; W)) + t (j = 2) * log (p ( y=2|x;W))+...+t(j)*p(y=N|x;W) [11]

In FIG. 10, the ensemble machine learning system 408 may, for example, generate template sequence 1 (quiver) 706a, template sequence 2 (symmetry) 706b, template sequence 3 (smoothness) 706c, . . . , further templates (e.g., other learned biomechanical parameters, such as entrainment and onset) 706n, which include, for example, stride length features (x1 , x2), right and left stride length variance features (x3, x4), right and left cadences (x6, x7), right and left cadence variances (x8, x9), etc. Applied to the conditioned input 702, the samples (x1, x2, . . . , xn) are called the vector X input to 702 in the ensemble of ML algorithms. These are adjusted reference normalized and/or scaled inputs. Aggregate classifier 708 outputs such information as a measure of tremor, a measure of symmetry, a measure of smoothness, and the like.

Music Therapy Center The music therapy center 110 is a decision-making system that runs on a processing device such as the portable device or notebook computer 220 of FIG. The music therapy center 110 receives inputs from the sensor data characterized at the collector 106, compares them to prescribed processes for delivering therapy, and produces auditory stimulation content played via the music distribution system 230. supply.

Embodiments of the present invention use contextual information to determine why certain situations are occurring and encode observed behaviors, thereby providing system state, and thus music therapy sessions, with a closed-loop, dynamic, and adjustable approach. change.

The interaction between the patient and the music therapy session provides real-time data for determining the patient's perception of the music therapy context, including movement, posture, stride, and gait response. After the input data is collected by the detection nodes (at the sensors), the embedded nodes process the context-aware data (at the edge processing) to provide immediate dynamic action and/or analyze the data in the analysis system. 108, for example, sending to an elastic network-based processing cloud environment for storage and further processing and analysis.

Based on the input, the program takes any pre-existing song content and analyzes intonations, major/minor chords, time signatures, and musical cues (e.g. melodic cues, harmonic cues, rhythmic cues, intensity). queue). The system can superimpose the metronome onto an existing song. Song content can be beat-mapped so that tuning margins can be calculated using precise knowledge of when beats occur (e.g., in response to an AV or MP3 file if W ), or in MIDI format. Sensors attached to the patient can be configured to provide tactile/vibratory feedback pulses in the musical content.

An example application of the method is described herein. Gait training analyzes the real-time relationship between the beats of the music being played for the patient and the individual steps taken by the patient in response to those specific beats of music. As noted above, a gating analysis is used to determine a window of repeated data in which there is some variability with each step or repetition. In some embodiments, the beginning of the window is determined as the point at which heel contact pressure exceeds a threshold (or other sensor parameter). FIG. 11 is an exemplary time graph showing the beats of the music (“time beats”) and the steps taken by the patient (“time steps”). Thus, in this case the onset time is associated with the "time step". Specifically, this graph shows the time beat 1101 of the music at the time of time beat 1 . After some time, the patient responds to the time beat 1001 by taking a step at the time of time step 1, namely time step 1102 . Tuning room 1103 represents the delay (if any) between time beat 1 and time step 1 .

12-13 show an example of synchronizing a patient's gait using the system described herein. FIG. 12 shows "perfect" tuning, eg, always zero tuning room. This occurs when there is no delay or a very small delay between the time beat and the associated time step taken in response to the time beat. FIG. 13 illustrates phase shift tuning, eg, where the tuning margin is non-zero but remains constant or has minimal variation over time. This occurs when there is an acceptable and constant delay over time between the time beat and the time step.

With continued reference to FIG. 11, the EP ratio is calculated as the ratio of the time between time beats to the time between time steps.
EP ratio = (time beat 2 - time beat 1) / (time step 2 - time step 1) [6]
Here, time beat 1 (1101) corresponds to the time of the first musical beat and time step 1 (1102) corresponds to the time of the patient's step in response to time beat 1. Time beat 2 (1106) corresponds to the time of the second musical beat and time step 2 (1108) corresponds to the time of the patient's step in response to time beat 2. The goal is EP ratio=1, or EP ratio/factor=1. The coefficients are determined as follows.

This factor allows the beats to be subdivided, or allows someone to step every 3 beats or 3 out of 4 beats. This can provide flexibility for different scenarios.

Figures 14 and 15 show a patient's entrainment response over time using the techniques described herein. FIG. 14 (left Y-axis: EP ratio, right Y-axis: beats per minute, X-axis: time) shows scattered dots 1402 representing the average EP ratio of the first patient's gait. The graph shows an upper bound 1404 of +0.1 and a lower bound 1406 of -0.1. Line 1408 shows tempo over time (starting at 60 beats per minute) and increases in steps to 100 bpm. FIG. 14 shows that the EP ratio remains near 1 (±0.1) when the tempo is increased from 60 bpm to 100 bpm. FIG. 15 shows the EP ratio for the second patient's gait, showing that the EP ratio still remains near 1 (±0.1) when the tempo is increased from 60 bpm to 100 bpm.

Figures 16 and 17 (Y-axis: entrainment room, X-axis: time) show time beat changes (e.g. tempo changes) and/or chord changes, tactile feedback changes, foot cue changes (e.g. , left-to-right cues, or right-to-left-to-cane cues). FIG. 16 shows a time-based graph where the patient's gait is balanced with "perfect entrainment" (always zero entrainment margin, or very little entrainment margin), or constant phase shift entrainment margin. As shown, it takes some time (prime time 1602) to reach equilibrium. FIG. 17 shows a time-based graph in which the patient's gait is not balanced, eg, does not reach perfect entrainment or constant phase-shift entrainment margin after a change to the time beat. The prime time is useful because it represents a data set that is independent of the tuning accuracy measurement. The prime time parameter can also be used to screen future songs for relevance. For example, if a patient exhibits longer prime times when a song is used, such song has less therapeutic power.

FIG. 18 shows a technique useful in gait training, where repetitive movements refer to multiple steps taken by the patient during gait. Gait training is tailored to each patient population, diagnosis, and condition to provide a personalized and individualized musical intervention. Based on your input, the program changes content, intonation, major/minor chords, time signatures, and musical cues (e.g., melodic cues, harmonic cues, intensity cues) where applicable . The program can select music using date of birth, listed musical tastes, and tempo to tune in to provide a playlist of passive music to use on a daily basis. Important inputs in gait training are the user's cadence, symmetry, and stride length during physical activity, such as walking. The program uses connected hardware to provide haptic/vibrational feedback on the BPM of the music. Suitable populations for gait training include patients with traumatic brain injury (TBI), stroke, Parkinson's disease, MS, and aging.

The method begins at step 1802 . At step 1804 , biomechanical data based on data from sensors, eg, sensors 200 , 206 , 208 are received at collector 106 . Biomechanical data may include data relating to onset, stride length, cadence, symmetry, assistive devices, or other such collection of patient characteristics stored and generated by analysis system 108 . Exemplary biomechanical data parameters are listed in Tables 1, 2, and 3 above. A basic state is determined from one or more data sources. First, patient gait is detected without music being played. Sensor data and feature data such as patient onset, stride length, cadence, symmetry, and assistive device data constitute the patient's basic biomechanical data in a treatment session. Second, sensor data from previous sessions of the same patient and higher level categorized data from the analysis system 108 constitute the patient's historical data. Third, sensor data and higher-level classified data for other similarly situated patients constitute population data. Thus, the baseline state includes one or more of (a) the patient's baseline biomechanical data at the treatment session, (b) the patient's previous session data, and (c) the population data. may include data from A base beat tempo is then selected from the base state. For example, the underlying beat tempo can be selected to match the patient's current cadence prior to the music playing. Alternatively, the underlying beat tempo can be selected as a fraction or multiple of the patient's current gait. Alternatively, the underlying tempo can be selected to match the underlying beat tempo used in previous sessions for the same patient. As yet another option, the underlying beat tempo can be selected based on underlying beat tempos used in other patients with similar physical conditions. Finally, the underlying beat tempo can be selected based on any combination of the above data. A target beat tempo can also be determined from this data. For example, the target beat tempo may be selected as a percentage increase in the underlying beat tempo with reference to improvements shown by other similarly situated patients. Tempo is understood to refer to the frequency of beats in music.

At step 1806, the music provided from the portable device 220 to the patient via the music delivery device 230 (eg, earphones, headphones, or speakers) begins at the base tempo, or a subdivision of the base tempo. Music having a constant underlying tempo is selected from a database to provide the patient with music at the underlying tempo, or existing music is modified to provide a constant tempo beat signal. For example selectively accelerated or decelerated.

At step 1808, the patient is instructed to listen to the beat of the music. At step 1810, the patient is instructed to walk at the underlying beat tempo and optionally receives cues for the left and right feet. The patient is instructed to walk "to" the beat tempo so that each step precisely matches the beat of the music, for example. Steps 1806 , 1808 , and 1810 may be initiated by the therapist or by audio or visual instructions on portable device 220 .

At step 1812, sensors 200, 206, 208 attached to the patient are used to record patient data such as heel pressure, six-dimensional motion, EMG activity, video recordings of the patient's motion. All sensor data are time-stamped. Data analysis, including the "gating" analysis discussed herein, is performed on the time-stamped sensor data. For example, analysis of sensor data, eg, heel pressure, is performed to determine the onset time of each step. Further data received includes the time associated with each beat signal of the music presented to the patient.

At step 1814, a connection is made to a tuned model (eg, the ensemble machine learning system 410 of the analysis system 108, or a model downloaded on the collector 106 and running on the portable device 220) for prediction and classification. (It is understood that such connections may preexist or may be initiated at this point.) Such connections typically occur very quickly or immediately.

At step 1816, an optional tuning analysis performed by analysis system 108 is applied to the sensor data. Entrainment analysis involves determining the delay between the beat signal and the onset of each step taken by the patient. As an output from the entrainment analysis, entrainment accuracy is determined, eg, a measure of the instantaneous relationship between the underlying tempo and the patient's step, as described above for entrainment margin and EP ratio. If the entrainment is not accurate, e.g., if the entrainment margin is not tolerably constant, then in step 1818, e.g., speed up or slow down the beat tempo, increase volume, increase sensory input, metronome or other relevant sound. are superimposed on each other. If the tuning is correct, eg, if the tuning margin is tolerable and constant, incremental changes to the tempo are made at step 1820 . For example, the underlying tempo of the music played on the portable device may be increased by, for example, 5% towards the target tempo.

At step 1822, a connection is made to the tuned model for prediction and classification. (It is understood that such connections may pre-exist or may be initiated at this point.) At step 1824, an optional symmetry analysis is applied to the sensor data. The output of the symmetry analysis is the symmetry of the patient's gait, e.g., how closely the patient's left leg movement matches the patient's right leg movement during stride length, velocity, stance phase, swing phase, etc. , is determined. If the steps are not symmetrical, eg, below a threshold, then at step 1826 adjustments are made to the music sent by the portable device to the patient. A first modification can be made to the music played while one of the patient's legs is moving, and a second modification can be made to the music played while the other of the patient's legs is moving. It can be done to the music that is played. For example, minor chords (or heightened volume, sensory input, tempo changes, or tone/metronome superposition) are played on one side, e.g., the affected side, and major chords are played on the other side, e.g., the non-affected side. can be played with The machine learning system 410 predicts in advance when a symmetry problem is about to occur based on the "fingerprint" leading to the symmetry problem, for example by analyzing movements that indicate asymmetry. Asymmetry can be determined by comparing someone's normal walking parameters to their background to determine how much that side is affected compared to the other side.

At step 1828, a connection is made to the tuned model for prediction and classification. (It is understood that such connections may pre-exist or may be initiated at this point.) At step 1830, an optional analysis of the equilibrium center, e.g., whether the patient is leaning over is performed on the sensor data. Analysis can be performed by combining the outputs of multiple foot sensors and video outputs. As an output from the equilibrium center analysis, a determination is made as to whether the patient is leaning over. If the patient is leaning over, at step 1832 a cue is issued to the patient to stand up straight, provided by the therapist or by audio or visual instructions on the handheld device.

At step 1834, a connection is made to the tuned model for prediction and classification. (It is understood that such connections may pre-exist or may be initiated at this point.) At step 1836 initiation analysis is applied to the sensor data, e.g. Show or show that it is difficult. As an output from the initiation analysis, a determination is made as to whether the patient exhibits problems with initiation. If the patient is exhibiting problems starting, e.g., below a threshold, tactile feedback can be given to the patient, which may include a countdown in beat tempo, or the beginning of the song at step 1838. may also include a countdown in front.

At step 1840, it is optionally determined whether the patient is using an assistive device such as a cane, crutches, walker, or the like. In some embodiments, portable device 220 provides a user interface for the patient or therapist to enter information regarding the use of assistive devices. If a cane is present, the analysis changes to a three-time signature, e.g., cane, right foot, left foot, and is cued by "left foot", "right foot", and "cane" in step 1842 to be read by the therapist. , or provided by audio or visual instructions on a handheld device.

At step 1844, a connection is made to the tuned model for prediction and classification. (It is understood that such connections may pre-exist or may be initiated at this point.) Although optional tuning analysis 1846 is applied to the sensor data generally as described above in step 1816, , with the differences described below. For example, the entrainment can be compared to previous entrainment data from an earlier point in the session, previous entrainment data from a previous session with the patient, or data related to entrainment of other patients. As an output from the entrainment analysis, the entrainment accuracy is determined, eg, how closely the patient's gait matches the underlying tempo. If the tuning is not correct, adjustments are made at step 1848 generally as described above at step 1818 .

If the synchronization is correct, at step 1850 it is determined whether the patient is walking at the target tempo. If the target tempo is not reached, the method proceeds to step 1820 (discussed above) where incremental changes are made to the tempo. For example, the underlying tempo of the music played on the portable device may be increased or decreased by, for example, 5% towards the target tempo. If the target tempo has been reached, the patient can continue therapy for the remainder of the session (step 1852). At step 1854, music of the desired tempo to be used during non-therapy sessions can be curated and left on the device 220 of FIG. This musical content is used as a challenge/practice by the patient between dedicated therapy sessions. At step 827, the program ends.

It is understood that the steps described above and illustrated in FIG. 18 may be performed in a different order than that disclosed. For example, the evaluations in steps 1816, 1824, 1830, 1836, 1840, 1846, and 1850 may occur simultaneously. Further, multiple connections to analysis system 108 (eg, steps 1814, 1822, 1828, 1834, and 1844) may be made only once throughout the described treatment session.

FIG. 19 shows a technique useful in ignoring training. For neglect training, the systems and methods described herein use connected hardware to provide tactile/vibrational feedback when the patient hits the target correctly. Connected hardware includes devices, video motion capture systems, or connected bells. All of these devices are connected into the system described and have speakers that vibrate when tapped and reproduce auditory feedback. For example, a connected bell provides data to the system in the same manner as sensor 200, eg, data regarding patient tapping of the bell. A video motion capture system provides video data to the system in the same manner as video camera 206 . An important input in neglect training is information about tracking movements to specific locations. The program uses connected hardware to give tactile/vibration feedback when the patient hits the target correctly. Suitable populations for neglect training include patients with spatial neglect or unilateral visual neglect.

The ignoring training flow diagram shown in FIG. 19 is substantially identical to the walking training flow shown in FIG. 18, with the differences noted below. For example, baseline tests define patient status and/or improvement from previous tests. In some embodiments, the basic test includes showing four evenly spaced objects on a screen, eg, display 222 of portable device 220 from left to right. The patient is instructed, either by cues displayed on the display 222 or verbally by the therapist, to hit an object to the beat of the background music. Similar to gait training, patients are instructed to ring a bell to the beat of background music. Feedback is given for each accurate hit. Once the basic information is collected, a number of equally spaced objects appear on the screen from left to right. As described above, the patient is instructed to strike objects in left-to-right order to the beat of background music. Feedback is given for each accurate hit. Similar to gait training, the analysis system 108 evaluates the patient's response and classifies the response to increase or decrease objects or increase or decrease the tempo of the music to reach a target tempo. give instructions.

FIG. 20 shows a technique useful in intonation training. For intonation training, the systems and methods described herein rely on speech processing algorithms. Phrases typically chosen are common words in the bilabial, guttural and vowel categories. Hardware is connected to the patient to provide tactile feedback in beats per minute to one of the patient's hands. Key inputs in intonation training are the timbre of the voice, the words spoken, and the rhythm of speech. Populations suitable for intonation training include patients with Broca's aphasia, expressive aphasia, non-fluent aphasia, apraxia, autism spectrum disorders, and Down's syndrome.

The intonation training flow diagram shown in FIG. 20 is substantially identical to the gait training flow shown in FIG. 18, with the differences noted below. For example, tactile feedback is provided to one hand of the patient to prompt tapping. The patient is then instructed, either by cues displayed on the display 222 or verbally by the therapist, to listen to the music being played. The spoken phrase that needs to be learned is played back in two separate parts, the first of the two parts being high pitched and the second of the two parts being low pitched. The patient is then instructed, either by cues displayed on the display 222 or verbally by the therapist, to sing the phrase using the device using the two pitches being played. Similar to gait training, the analysis system 108 evaluates the patient's response and classifies the response in terms of pitch accuracy, words spoken, and ranking by the patient or assistant/therapist to provide alternatives. Instructions are given to provide phrases and responses are compared to target speech parameters.

FIG. 21 shows an approach useful in musical stimulus training. For musical stimulus training, the systems and methods described herein rely on speech processing algorithms. Familiar songs are used with an algorithm that isolates expected parts (called expectation violation). The hardware includes speakers that receive and process singing by the patient, and in some embodiments, the therapist can manually provide input regarding the accuracy of the singing. Important inputs are information related to timbre of voice, words spoken, rhythm of speech, and musical preferences. Suitable populations include patients with Broca's aphasia, non-fluent aphasia, TBI, stroke, and primary progressive aphasia.

The musical stimulus training flow diagram shown in FIG. 21 is substantially identical to the walking training flow shown in FIG. 18, with the differences noted below. For example, a song is played for the patient and the patient is instructed to listen to the song by cues displayed on the display 222 or verbally by the therapist. Musical cues are added to the song. The word or sound is then excluded at the expected location and a gestural musical cue is played to prompt the patient to sing the missing word or sound. Similar to gait training, the analysis system 108 evaluates the patient's response and classifies the response in terms of pitch accuracy, words spoken, ranking by the patient or assistant/therapist, and categorizes the utterances. Instructions are given to play further portions of the song in order to improve to the target speech parameters.

FIG. 22 shows an approach useful in gross motor training. In gross motor training, the systems and methods described herein are aimed at helping ataxia, range of motion, or initiation. The more difficult portions of the exercise are musically "ascented" using, for example, melodic cues, harmonic cues, rhythmic cues, and/or intensity cues. An important input is information related to motion in the X, Y, Z direction capture using a connected hardware or video camera system. Suitable populations include patients with neurological and orthopedic strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

The flow diagram for gross motor training shown in FIG. 22 is substantially identical to the flow for gait training shown in FIG. 18, with the differences noted below. Similar to gait training, the patient is cued to move to the beat on which the musical selection is based. The analysis system 108 evaluates the patient's response, classifies the response in terms of movement accuracy and coordination as described above, and gives instructions to increase or decrease the tempo of the music being played.

FIG. 23 shows a technique useful in grip strength training. For grip strength training, the systems and methods described herein rely on sensors associated with the gripping device. The hardware comprises a gripping device with a pressure sensor associated with the portable device 220 and a speaker connected. A key input is the pressure exerted by the patient on the grasper in a manner similar to the heel contact pressure measured by sensor 200 . Suitable populations include patients with neurological and orthopedic strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

The flow diagram for grip strength training shown in FIG. 23 is substantially identical to the flow for walking training shown in FIG. 18, with the differences noted below. Similar to gait training, the patient is cued to apply force to the grasper in time with the beat on which the musical selection is based. The analysis system 108 evaluates the patient's response, classifies the response in terms of movement accuracy and coordination as described above, and gives instructions to increase or decrease the tempo of the music being played.

FIG. 24 shows a technique useful in speech cue training. For speech cue training, the systems and methods described herein rely on speech processing algorithms. The hardware may include speakers that receive and process singing by the patient, and in some embodiments, the therapist may manually provide input regarding accuracy of speech. Important inputs are the timbre of voice, the words spoken, the rhythm of speech, and musical preferences. Appropriate populations include patients with problems with robots, word finding, and speech stuttering.

The flow diagram for speech cue training shown in FIG. 24 is substantially identical to the flow for gait training shown in FIG. 18, with the differences noted below. Similar to gait training, the patient is cued, either by cues displayed on the display 222 or verbally by the therapist, to say sentences in single syllables to the beat on which musical choices are based. is given. The analysis system 108 evaluates the patient's speech, classifies the response in terms of speech accuracy and entrainment as described above, and gives instructions to increase or decrease the tempo of the music being played.

FIG. 25 shows an approach useful in training minimally conscious patients. The systems and methods described herein use an imaging device such as a 3D camera to determine the direction the patient is looking if the patient's eyes are open and the resulting pulse or heart rate of the patient. rely on the system. The program explores and optimizes heart rate, stimulation, respiration rate, eye closure, posture, and restlessness. Suitable populations include comatose patients and patients with impaired consciousness.

The minimally conscious patient training flow diagram shown in FIG. 25 is nearly identical to the gait training flow shown in FIG. 18, with the differences noted below. Similar to gait training, the patient is given an increasing stimulus at the patient's respiratory rate (PBR). For example, the patient is first stimulated with a musical chord on the PBR and observed whether the patient's eyes are open. If the patient's eyes are not open, the PBR may range from humming a simple melody to singing an "ahh" on the PBR and singing the patient's name on the PBR (or playing a recording of such a sound). The stimulus is continuously increased until the patient's eyes are open at each input. Analysis system 108 evaluates the patient's eye tracking, classifies responses in terms of level of consciousness, and provides instructions to change stimuli.

Figures 26-28 illustrate techniques useful in attention training. In attention training, the systems and methods described herein operate in a closed loop to help patients sustain, divide, shift, and select attention. No visual cues are allowed to suggest which move to make. Suitable populations include patients with brain tumors, multiple sclerosis, Parkinson's disease, and neurological diseases and injuries.

The attention span training flow diagram shown in FIG. 26 is substantially identical to the gait training flow shown in FIG. 18, with the differences noted below. Similar to gait training, the patient is given an instrument (eg, any instrument that works, such as drumsticks, drums, keyboards, or wirelessly connected ones of each) to be displayed on the display 222. The patient is instructed by the cues present or verbally by the therapist to progress along or work on the audio cues defined in levels 1 through 9 shown in FIG. The analysis system 108 assesses the patient's ability to complete the task correctly and classifies the responses to change the tempo or difficulty of the task. Similarly, FIG. 27 shows a flow diagram for attentional reversal training in which cues displayed on the display 222 or verbally by the therapist accompany audio cues alternating between the left and right ears. or to work on audio cues. FIG. 28 shows a flow diagram for split attention, in this case instructed to follow or work with audio cues using audio signals in both the left and right ears. Given.

The dexterity training flow diagram shown in FIG. 29 is substantially identical to the gait training flow shown in FIG. 18, with the differences noted below. In dexterity training, patients are instructed to tap their fingers on a piano keyboard in order to collect basic movement and range of motor information. A song starts at a certain number of beats per minute and the patient begins tapping with a basic number of fingers. The analysis system 108 assesses the patient's ability to complete the task correctly and classifies the responses to change the tempo or difficulty of the task.

The flow diagram for oral movement training shown in FIG. 30 is substantially the same as the flow for gait training shown in FIG. 18, with the differences described below. In oral motor training, the patient is instructed to perform a task that alternates between two sounds, eg, "wow" and "ahh". The analysis system 108 assesses the patient's ability to correctly complete the task and classifies the responses to vary the tempo or difficulty of the task, for example, by giving different target sounds.

The breathing exercise flow diagram shown in FIG. 31 is substantially identical to the walking exercise flow shown in FIG. 18, with the differences noted below. Breathing exercises determine the basal respiratory rate and shallowness of breathing. Music is presented at a tempo based on the patient's breathing rate, and the patient is instructed to perform the level of breathing work described in FIG. The analysis system 108 assesses the patient's ability to correctly complete the task and classifies the responses to vary the tempo or difficulty of the task, for example, by providing different breathing patterns.

Further described herein are methods, systems, and apparatus for assisting next-generation medical therapeutic systems for improving or maintaining motor function using Augmented Reality (AR) and Augmented Audio (AA). is described. Exemplary embodiments of the systems and methods for augmented neurological rehabilitation, recovery, or maintenance (ANR) disclosed herein utilize additional sensor streams to determine therapeutic efficacy, closed-loop It builds on the tuning approach by informing the treatment algorithm of Exemplary systems and methods for neurological rehabilitation that can be utilized to implement embodiments of ANR systems and methods are shown and described above, No. 62/322,504, filed Apr. 14, 2016, entitled "Systems and Methods for Neurological Rehabilitation and Co-Pending and Commonly Assigned U.S. Patent Application Serial No. 16/16 to McCarthy et al. 569,388, "Systems and Methods for Neurological Rehabilitation," each of these patent applications and patent specifications is incorporated herein by reference in its entirety. incorporated herein as if written.

According to one or more embodiments, ANR systems and methods can include methods for providing AR 3D dynamic models of specific people or objects, including cloud-based and local database It involves getting images and videos of people, or images by querying the . In addition, the ANR system and method can be configured to fuse and/or synchronize context regarding dynamic models, patients or humans, audio content, and the environment, and/or synchronize into a synchronized state, allowing for a greater understanding of the capabilities of music. Neuroscience and how visual images affect recovery (eg, using mirror neurons) can be leveraged to improve motor function.

According to one or more embodiments, ANR systems and methods include methods of combining AA techniques for repetitive motor actions. Augmented Audio (AA) combines real-world sounds with additional computer-generated audio “layers” that enhance sensory input. Neuroscience, with rhythm at its core, uses stimuli to engage the motor system in repetitive motor actions such as walking. Incorporating AA into a treatment protocol provides increased therapeutic efficacy, increased adherence to the treatment protocol, and increased safety in the form of increased situational awareness for the patient. Disclosed embodiments configured to add AA mix many audio signals into a synchronized state, including external ambient sound input, pre-recorded content, rhythmic content, voice guidance, and the neuroscience of music. can be utilized. In addition, it may have the ability to combine algorithmically generated music with underlying rhythmic cues detailed in [0184] for patients with motor or physical disabilities. This can be done by fusing this generated rhythm with input from the patient's real-time biometric data into an interactive feedback state. Exemplary systems and methods for algorithmically generating auditory stimuli for use in connection with neurological rehabilitation are described in U.S. Provisional Patent Application, filed July 24, 2017 under 35 U.S.C. U.S. patent application Ser. No. 16/044, filed Jul. 24, 2018, entitled "Improving Music for Repetitive Motion Exercises," which claims the benefit of priority of patent application Ser. No. 62/536,264; , 240 (now U.S. Pat. No. 10,556,087, issued Feb. 11, 2020), to McCarthy et al. No. 16/743,946, filed Jan. 15, 2020, entitled "Improving Music for Motion Exercise," and the specification of these patent applications and patents. each of which is incorporated herein by reference as if set forth herein in its entirety.

The neural plasticity, entrainment, and science of mirror neurons are the fundamental scientific building blocks behind the disclosed embodiments. Entrainment is a term for the activation of motor centers in the brain in response to external rhythmic stimuli. Studies have shown that the voice-motor nerve tract lies within the reticular spinal tract, the part of the brain involved in movement. The priming and timing of movement through these pathways has demonstrated the ability of the motor system to couple with the auditory system to induce movement patterns (Rossignol and Melville, 1976). Entrainment processes have been shown to effectively increase walking speed (Cha, 2014), reduce gait variability (Wright, 2016), and reduce fall risk (Trombetti, 2011). there is Neuroplasticity refers to the brain's ability to strengthen pre-existing nervous system connections, allowing individuals to acquire new skills over time. Studies have demonstrated that music facilitates changes in specific motor areas of the brain, indicating that music can promote neuroplasticity (Moore et al., 2017). Mirror neurons fire both when you perform an action and when you observe another person performing an action. So when you see another person performing an action, your brain reacts as if you were the one performing the activity. This allows us to learn behavior by imitation. This is important in the context of the disclosed embodiments. This is because the patient can use mirror neurons to imitate those movements while observing a human simulation in augmented reality.

According to one or more embodiments, the ANR system and method responds to patient or therapist exception conditions received as input to the ANR system to detect people or objects that are smaller or larger than a specified size. It is configured to process the images/videos to remove from and/or add to the images/videos. Such exceptions may be patient reactions such as instructions to reduce scene complexity or therapist instructions to incorporate occlusions which may be people/objects to increase scene complexity. Embodiments can accommodate recording of all data for all exceptional conditions of a patient or therapist in addition to the session data itself.

According to one or more embodiments, the ANR system and method includes a telepresence scheme that allows the therapist to connect with the patient at a remote location using the system. In addition to fully accommodating all local functions experienced by patients and therapists when they are co-located, the telepresence approach provides a close-up of the patient's biomechanical movements to AR 3D dynamic models of people/objects. Including tracking.

AR 3D dynamic models generate patient presentations of animated AR/VR visual scenes based on mirror neuron neuroplasticity, entrainment, and the fundamental principles of science to meet clinical or training goals. Software-based algorithmic processes of ANR systems and methods to facilitate targeted outcomes. A telepresence system is configured using the present invention to provide the ability to activate an interactive video link between a therapist and a patient at a remote location. Telepresence corresponds to using the present invention to project images/videos from a patient at a distance to show the patient's relative position in the AR 3D dynamic model. The telepresence method can also provide the ability for the therapist to adjust the AR model in real time (spatial location or which items are available) to allow modifications to the session. Telepresence also allows the therapist to see the patient in comparison to a model.

FIG. 32 is a conceptual illustration of the main components of an exemplary ANR system 3200 that uses closed-loop feedback to measure, analyze, and act on people to facilitate outcomes toward clinical or training goals. It depicts a general overview. It should be appreciated that ANR system 3200 can be implemented using various hardware and/or software components of system 100 described above. As shown, the ANR system relates to real-time feedback on gait parameters, environment, context/user intent (including past performance), physiological parameters, and outcomes (e.g., closed-loop, real-time decision making). Measure or receive input. One or a combination of these inputs can be sent to a clinical thinking algorithm module (CTA) 3208 that controls the analysis of and actions on this information. In some embodiments, an example of real-time feedback is that a measured quality of a user's gait measure (e.g., symmetry, stability, gait cycle time variance) exceeded a modeled safety threshold. determines that this results in a new queued response. Such cued responses can be both visual and audio, e.g., by adding a metronome audio layer to amplify the prominence of musical beats, modify the AR scene it models, or Toward safer gait speeds and motor behaviors.

In the CTA module 3208, actions taken by the ANR system are defined based on analysis of the input. Movements are output to the patient in the form of various stimuli, including music (or other components), rhythm, sonification of sounds, spoken words, augmented reality, virtual reality, augmented speech, or tactile feedback. can do. As shown in FIG. 32, decisions made by the critical thinking algorithm in relation to various inputs, outcomes, etc., are programmed AR/AA output modeling to dynamically generate/modify outputs to the patient. Provided to module 3210 . Output is provided to the patient via one or more output devices 3220 such as visual and/or audio output devices and/or tactile feedback devices. For example, AR visual content can be output to AR glasses 3222 worn by the patient. Enhanced audio content can be provided to the patient via audio speakers or headphones 3225 . As will be appreciated, other suitable visual or audio displays may be used without departing from the scope of the disclosed embodiments. Additionally, although various elements of the system are shown separately in FIG. 32, it should be understood that the features and functions of various aspects of the system may be combined.

The clinical thinking algorithm module 3208 in this medical/treatment system includes: arm, leg, sway, postural stability, foot drop, dynamic stability, respiration, mouth movement, respiration, endurance, heart rate training, respiration frequency, Emphasis on recovery, maintenance, or enhancement of motor function including, but not limited to, optical flow, boundary support training, strategic training (ankles, knees, hips), attractor coupling, muscle firing, training optimization, and gait It can be configured to run a critical thinking algorithm that Also, one or more of these CTAs may be implemented synchronously or in combination with other interventions that share similar goals. Examples include implementing CTA in conjunction with functional electrical stimulation, deep brain stimulation, transcutaneous electrical nerve stimulation (TENS), gamma frequency voice entrainment (20 Hz-50 Hz), or other electrical stimulation systems. be Additionally, medication and manipulation of CTA can be combined with antispasticity drugs, or medication can be combined with infusion of neurotoxins. For example, CTA can be applied or initiated during times when such interventions have been shown to be most effective, or as a method of priming the motor system, where such interventions are most effective. It can also be used before the time is reached.

AANR System Inputs Inputs to the ANR system 3200 are important to allow the system to measure, analyze and operate in a continuous loop to facilitate outcomes towards clinical or training goals.

Receiving input with the system can include using sensors to measure movement of a person to determine biomechanical parameters of movement (eg, temporal, spatial, and side-to-side comparisons). These motion sensors can be placed anywhere on the body and can be one sensor or many. Other types of sensors can also be used to measure other input parameters, including respiratory rate, heart rate, oxygen levels, body temperature, and an electroencephalogram (EEG), which records spontaneous electrical activity in the brain. EEG), electrocardiogram (ECG or EKG), which measures the electrical activity of the heart, electromyography (EMG), which assesses and records the electrical activity produced by skeletal muscle, often measures changes in light absorption in the skin Photoplethysmography (PPG), optical inertial measurement devices, video cameras, microphones, accelerometers, gyroscopes, infrared, It may include ultrasound, radar, RF motion detection, GPS, barometer, RFID, radar, humidity, or other sensors that detect physiological or biomechanical parameters. For example, in the exemplary ANR system 3200 shown in FIG. 32, gait parameters may be measured using one or more sensors 3252, such as IMUs, footpad sensors, smartphone sensors (e.g., accelerometers), environmental sensors, etc. can be done. Additionally, physiological parameters can be measured using one or more sensors 3254, such as PPG, EMG/EKG, respiration rate sensors, and the like.

Additionally, contextual information regarding desired results, usage environment, past session data, other techniques, and other environmental conditions can be received as inputs to the CTA module to tailor the CTA's response. For example, as shown in FIG. 32, contextual information 3258 and environmental input 3256 information may be received as inputs that further inform the operation of CTA 3208. An example of using contextual information is that historical information about a user's walking patterns can be used in conjunction with artificial intelligence (AI) or machine learning (ML) systems to give patients more personalized clinical goals and behaviors. is. These goals may be steps per minute, walking speed, heart rate variability, oxygen consumption (VO2 max), respiratory rate, session length, asymmetry, variability, distance walked, or desired heart rate. Target parameters, such as limits on numbers, can also be modified.

In addition, Bluetooth Low Energy (BLE) beacons or other wireless proximity technologies such as wireless triangulation and angle of arrival are used to detect environmental information, and in the patient's field of view to the AR 3D dynamic model, detected Wireless location triggers can facilitate the appearance and/or disappearance of people/objects depending on location. In some embodiments, the AR 3D dynamic model output by the ANR system can be controlled by the therapist and/or beacon triggers to change or maintain guidance requirements for the patient. These triggers can also be used in conjunction with the ambulation or physiological data described above to provide additional triggers in addition to the wireless beacon triggers. For example, gait data feedback from an IMU product enables a gait feedback loop that provides the ANR system 3200 with the ability to make changes in the software process of the AR 3D dynamic model.

A Clinical Thinking Algorithm According to one or more embodiments, the CTA module 3208 is a clinical thinking algorithm configured to control the applied therapy to facilitate outcomes towards clinical or training goals. Execute thought algorithms. Clinical goals can include items such as those described in connection with FIGS. 18-31, but further examples include interventions for agitation in Alzheimer's disease, dementia, bipolar disorder, and schizophrenia. , as well as training goals/physical activity goals. In this section, various non-limiting example techniques that can be used to convey appropriate rehabilitation responses determined by CTA, such as adjusting rhythmic tempo and synchronized AR visual scenes, are discussed. Each of these approaches can also be implemented using CTA alone or in combination with each other. In one or more embodiments, the system 3200 can be configured to combine CTA with entrainment principles for repetitive motor activity, and otherwise combine these techniques with each other toward other goals. be able to.

By way of example, and not limitation, the CTA module 3208 of the ANR system 3200 utilizes a combination of biomechanical data, physiological data, and context to create a virtual treadmill that is output via the AR/AA output interface. can be configured as follows: While the treadmill keeps the same pace as the movement of a physical belt for someone, the virtual treadmill can be dynamically adjusted using CTA according to the entrainment principle to allow a person to walk or walk as well as other exercise interventions. Independently regulate the pace of exercise. However, in addition to or instead of using rhythmic stimuli to direct the individual towards the aforementioned biomechanical goals, a virtual treadmill can be generated based on the patient's entrainment, listed above as a goal parameter. can be dynamically controlled toward target parameters such as those described in .

Target parameters can also be set based on input from the clinician, user, historical data, or baseline conditions, or to a degree that is clinically meaningful. Additional target parameters are recommendations for length, duration, and intensity of exercise specific to particular conditions such as cardiovascular disease, asthma, COPD, fall prevention, musculoskeletal disease, osteoarthritis, and general aging. can be prescribed or can be recommended.

Further described herein is an exemplary embodiment of the ANR system 3200, in which the CTA module 3208 utilizes biomechanical data, physiological data, and context to provide virtual treadmill and AR/ It is configured to provide gait training therapy in the form of rhythmic auditory stimuli output via AA output interface 3220 .

FIG. 37 is a process flow diagram illustrating an exemplary routine 3750 for using the ANR system 3200 to provide gait training therapy to a patient. FIG. 38 is a mixed system and process diagram conceptually illustrating aspects of an ANR system 3200 performing a gait training routine 3750, in accordance with an exemplary embodiment of the disclosed subject matter. As shown, sensors 3252 , specifically foot-mounted IMUs, capture sensor data related to walking parameters that are provided to CTA module 3208 . In addition, an AA/AR modeling element 3210 comprising a voice engine receives input from the CTA and one of rhythmically cued musical content, interactive voice guidance, spatial voice effects processing, or configured to generate a cueed audio ensemble comprising a plurality; Similarly, an AA/AR modeling element 3210 comprising an AR/VR modeling engine (also referred to as an AR 3D dynamic model) is shown receiving input from the CTA to create virtual AR characters and objects. (e.g., virtual walking person), background motion animations (e.g., virtual treadmill, step/footprint, and animation), and one or more of scene lighting and shading. configured to generate a visual ensemble with

FIG. 39 is a mixed system and process diagram conceptually illustrating in more detail an exemplary audio output device 3225 and enhanced audio generation elements of the ANR system 3200 . In one embodiment, as shown in FIG. 39, the AA device can capture ambient sounds using, for example, stereo microphones. AA devices can also use stereo transducers to produce audio output. The AA device may also include a head-mounted IMU. It will also be appreciated that the AA device may also include hardware for audio signal processing that receives, processes, and outputs the enhanced audio content received from the AA/AR module 3210 alone or in conjunction with other content such as environmental sounds. Ware components and software components may also be provided. As shown, the CTA module 3208 receives gait parameters, including those received from sensors including foot-mounted IMUs and head-mounted IMUs. Additionally, in one embodiment, the CTA receives data related to physiological parameters from other sensor devices, such as PPG sensors.

Returning to FIG. 37, in step 3700, the AR/AA output device 3220 and IMU are controlled as the ANR system 3200 calibrate to collect preliminary gait data such as stride length, velocity, gait cycle time, symmetry, etc. A patient wearing sensor 3252 begins to walk. At step 3701, the CTA module 3208 determines the underlying rhythmic tempo for both music playback and the displayed virtual AR scene. For example, the underlying rhythmic tempo can be determined by CTA, as described in connection with FIG.

With the user's walk cycle time as input, in step 3702 the speech engine (i.e., the speech modeling element of the AR/AA modeling module 3210) uses an auxiliary rhythmic sound, such as a metronome sound, to amplify the spurs of the beat. Generate corresponding music with adjusted tempo along with appropriate cues.

At step 3703, a visual AR engine (ie, the visual modeling elements of AR/AA modeling module 3210) generates a moving virtual scene, such as that understood in the video game industry. More specifically, in one embodiment, a virtual scene is presented under the control of the CTA to share a common reference timing with the audio engine in order to synchronize the elements of the visual scene with the music and rhythmic tempo. contains elements. Although the AR scenes described herein include virtual treadmills or virtual people and footprints, AR scenes include virtual treadmills, virtual walking people, virtual crowds, dynamic virtual scenes, etc. It can also be any one or more of the various examples described herein.

At steps 3704 and 3705, music/rhythm and visual content is delivered to the patient using an AR/AA device 3220 such as a lightweight head-up display with earphones 3225 (eg AR goggles 3222). Under the control of the CTA, the patient is instructed regarding treatment in steps 3706 and 3707 via off-screen audio cues generated by the audio engine. This may include a pre-ambulatory exercise preview to familiarize the patient with the visual scene and audio experience and practice.

FIG. 40 shows a presented exemplary AR virtual scene 4010 to which the patient tunes. As shown, the scene may include a moving 3D image of another person walking "in front" of the patient, whose steps and walking movements are synchronized to the musical tempo. More specifically, in one embodiment, the AR characters walk to the same tempo as the underlying beat tempo of the audio content generated by the CTA and speech engine. In this example, the patient's goal may be for their steps to match rhythmically with the audio and visually with the characters. Additionally, as shown in FIG. 40, an AR scene may contain multiple footprints with additional cues such as L and R indicating left foot, right foot.

A scene containing footprints may be virtually moving toward the patient at a prescribed speed, while a virtual character is walking in front of the patient away from the patient. In one embodiment, the scene may be moving based on the gait cycle time, GCT(right foot)/2*60=musical tempo determined by CTA. Additionally, additional cues generated in conjunction with the AR scene may include rhythmic audio cues that reinforce the visual cues. For example, one effective method of reinforcement may involve the AA system 3210 generating rhythmically synchronized footstep sounds of virtual characters, simulating a mass march to a common beat. In addition to timing the actions of the visual elements with rhythmic audio stimuli, by giving the patient rhythmic audio elements generated based on the visual stimuli, the system further enhances the fundamentals of entrainment and mirror neurons. consolidate a virtuous circle within the therapeutic concept.

As a further example, FIG. 41 shows a presented exemplary AR virtual scene 4110 to which the patient tunes. As shown in FIG. 41, a virtual treadmill can be created and dynamically controlled based on patient entrainment to target parameters such as those listed above as target parameters. In this example, the generated AR treadmill is animating the movement of the treadmill surface 4115, producing virtual steps at the same tempo as defined by the CTA for auditory stimulation. Additionally, the 3D animation of the virtual treadmill can include visually highlighted steps or tiles that the patient can use as a visual target while entraining to the rhythm generated under the control of the CTA. Thus, in this example, the patient's goal may be for their steps to match visually with the animated target steps, both rhythmically with the audio. As further shown in FIG. 41, the animated footprint 4120 is a virtual track moving in a direction toward the patient indicated by the directional arrow, which may be in the opposite direction if the patient is performing a backward walking training exercise. It can be shown on the surface of the red mill 4115. In addition, the animations highlighting the target step 4125L (left foot step) and/or 4125R (right foot step), respectively, are used to encourage the user to walk on the corresponding foot (e.g., left or right), to the rhythm of the CTA. is highlighted in sync with Moreover, as further described herein, virtual scenes such as those shown in FIGS. 40-41 dynamically adjust based on the patient's entrainment margin (EP) and corresponding changes to the audio stimuli. can do. Other adjustments to the virtual scene may include changing the virtual background environment to simulate different walking scenarios, weather, surfaces, lighting, and slopes.

Returning now to FIG. 37, once the patient begins to walk, at step 3708, a biomechanical sensor (eg, sensor 3252) measures real-time data for use in assessing the patient's level of entrainment. At step 3709, entrainment margins (eg, described in FIG. 18) are determined by the CTA module 3208 and used to determine how the goals of the training session should be achieved. As described in previous embodiments of the present disclosure, entrainment room can be the basis for modifying rhythmic audio stimuli and visual scenes, which is done in step 3710 . For example, the CTA analyzes the incoming data history of the patient's gait cycle time compared to the rhythmic intervals of the beats supplied to the patient by the audio device. Exemplary techniques for modifying audio stimuli based on entrainment margins were also described above. In one example, if the calculated EP values for the patient's steps over time are not within the specified EP value tolerances and/or are not sufficiently consistent, the CTA module performs AA/AR modeling. The module can be instructed to adjust (eg, slow down) the tempo of the RAS and adjust the motion speed of the AR scene accordingly to synchronize with the RAS. In a further example, the patient is considered to be in sync with the CTA if enough step times are in sync with the beat time.

Once entrained, in step 3711 the CTA evaluates whether the patient has reached the goal. If the goal has not been reached, one or more goal parameters may be adjusted at step 3712 . For example, in one example, the CTA compares the RAS tempo and associated AR scene speed to target tempo parameters (e.g., training/treatment goals) and the rhythmic tempo and/or scene motion speed is altered to account for this comparison. be. Exemplary ways in which the CTA module 3208 can adjust rhythmic auditory stimuli based on entrainment margins are shown, for example, in connection with FIG. 18 and described above.

For example, if the patient is not reaching his training speed goal, modifying the goal parameters may include increasing or decreasing the music tempo at step 3712 . In doing so, the mechanism of action of the RAS is used to encourage the patient to walk faster or slower. Alternatively, another training goal can be to lengthen the patient's stride length, which can be achieved by slowing the image motion speed parameter. Modifying the visual scene prompts the patient to model the visual example presented with the mechanism of action of mirroring.

It is important to understand that the audio and visual outputs reinforce each other's stimuli, and the visual scenes are layered together synchronously with the rhythmic stimulus. Depending on the program (eg, CTA) selected, the CTA module 3208 dynamically adjusts visual scenes and rhythmic tempo to achieve therapeutic goals.

The preceding example illustrates how the ANR system 3200 can use the CTA module 3208 to control the synchronization of music tempo and AR scenes based on biomechanical sensor input to facilitate gait training. . It should be appreciated that the principles of this embodiment are applicable to many disease indications and rehabilitation scenarios.

In an exemplary configuration, a virtual treadmill can be generated by the ANR system 3200 to adjust the patient's gait toward a target parameter of oxygen consumption. In this example, a virtual treadmill is generated and controlled using entrainment to adjust walking speed toward an oxygen consumption or efficiency target parameter. FIG. 33 graphically illustrates a real-time session conducted using the ANR system 3200 using V02 max as the target parameter, tempo change as the interim target, and entrainment to drive physiological changes associated with v02 max. It is a visualization. Figure 33 shows an example of how this process works in real time. More specifically, FIG. 33 is a graphical user interface showing various salient data points and parameter values that are measured, calculated and/or adjusted in real time by the ANR system during a session. As shown, the top of the interface shows a graph of entrainment margin values calculated for each step in real time throughout the session. Specifically, the top bar shows the individual EP calculated for each step, which in this example is the phase correlation between the step time interval and the beat time interval. The central zone around EP=1 represents steps that are well tuned to the beat, ie steps with EP values within the specified range. The next window below provides a status bar that indicates whether the parameters are within safe limits. The next window below shows the real-time response driven by the CTA based, among other things, on the measured parameters, tuning, other previously mentioned inputs, and feedback to the CTA. Specifically, circular icons represent algorithmic responses, which include both tempo changes and changes in the level of the rhythmic stimulus (eg, volume). The bar below it shows only the tempo, and the tempo changes by itself. The next window below shows the real-time tempo of the rhythmic stimulation given to the patient over time according to the CTA response. The bottom window shows oxygen consumption and target parameters measured over time. Although not shown in FIG. 33, an augmented reality scene (e.g., , virtual treadmill) may be presented to the patient. Examples of how the AA/VR module 3210 can be configured to synchronize with the speed of visual animation and audio include defining the relationship between the speed of displayed repetitive movements and the tempo of audio cues. can include doing For example, calculating treadmill speed and step spacing based on beat tempo defines the relationship between audio and visual elements. In addition, the treadmill reference position, footstep timing, and beat-synchronized animation are synchronized to the beat output time, including the beat tempo. AA/VR module 3210 generates a wide range of synchronized virtual scenes based on defined relationships between audio and visual elements using time-scaling and video frame interpolation techniques known in the animation industry. can be generated on demand programmatically by

FIG. 34 is a graph depicting metabolic changes (each represented by a set of two dots connected by a dashed line) during the first training session for seven patients. FIG. 34 shows data supporting that intentional tuning can improve an individual's oxygen consumption. In this case the graph shows a person's oxygen consumption (oxygen ml/kg/meter) before and after rhythmic training using the ANR system. This figure shows an average reduction of 10%. This result indicates that it is possible to improve endurance and reduce energy expenditure during walking through the entrainment process.

This above process can be similarly performed for each of the various possible target parameters described above (eg, oxygen consumption can be exchanged for another target such as heart rate) and FIG. to ambulation or other interventions discussed in connection with FIG.

According to one or more embodiments, the ANR system 3200 converts real-time measured information about a person's motion into components of AR images and/or music content output to the patient during a treatment session (e.g., momentary tempo, rhythm, harmony, melody, etc.). This can be exploited by the system to calculate tuning parameters, determine phase tuning, or define basic carryover features. For example, the AR image can move at the same speed or pace as the target parameters. Alternatively, AR-related motion of images can be entrained to rhythmic stimuli to synchronize how a person should move.

Examples of AR 3D dynamic model output include projection of a therapist (virtual character) or person (virtual character) walking through the patient's field of view, initiated by the person giving the treatment (real therapist). can include doing For example, FIG. 35 shows a view of a therapist or coach projected in front of a patient or trainer using AR, eg using AR glasses known in the art. This AR 3D dynamic model is controlled using one or various CTAs. When combining multiple CTAs, the virtual therapist may begin with the technique shown and described in connection with FIG. 22, and then proceed with a gait training regimen as shown and described in connection with FIG. can be done. Alternatively, they can be dual-tasked at the same time. During these instances, the virtual character can be controllably displayed by the system as walking or moving backwards or forwards in a smooth motion similar to the patient's unaffected side. This can activate mirror neurons, prompting neurons on the affected side to mirror the behavior of neurons on the unaffected side. This process may also include providing an audio stimulus to synchronize the virtual and/or physical person with this stimulus.

In another example, the AR 3D dynamic model is configured to simulate a scenario in which the patient walks in or around a group of people and/or people and objects in front of and/or to the side of the patient. can be done. For example, FIG. 36A shows a view of a group of people projected in front of a patient using AR. The system should be configured to project a group of people or individuals moving faster or slower than the person's baseline, prompting the patient to move or stop/start at a similar speed in a real-world environment. can be done. A group of people or individuals can also be tuned to the beat of a rhythmic auditory stimulus, or another desired target. Various difficulty levels in navigating can be initiated by the AR 3D dynamic model. As should be appreciated, the AR scene of therapists, crowds, people, obstacles, and the like can be dynamically adjusted using the AR 3D dynamic model according to the output of the CTA.

In another example, the AR 3D dynamic model can be configured to simulate a scenario in which the patient walks in or around an array of cones that provide a virtual obstacle course for the patient to follow. While cones are common obstacles in treatment environments, other embodiments of this can be configured to simulate normal daily life activities (eg, grocery shopping). These cones, along with virtual obstacles, can encourage direction changes by stepping sideways to each side and walking backwards rather than only by walking forwards. Again, wireless beacon triggers can be used to cause the ANR system to present appearing and/or disappearing cones. A beacon is activated based on the detected location of a person associated with the cone. Additionally, various difficulty levels can be initiated with induction times and lengths. The target parameter in this example can be a measure of walking speed or quality of walking. Successful navigation would virtually navigate around the cone without hitting it. As long as the person successfully avoids obstacles and the quality of gait deteriorates (measured by increased variability or worsened asymmetry), the system can be applied to levels of increasing difficulty (e.g., more obstacles and more fast speed).

In another example, the AR 3D dynamic model can be configured to simulate a scenario where the patient is walking in or around caves and/or cliffs that may contain obstacles for realistic effect. . Reality improves the detail required in the guidance over previously presented use cases. In another example in a person with an asymmetrical gait pattern, a tortuous path can be presented that requires the person to take longer steps on the affected side. Also, this winding road can be a separate cliff that must be crossed over a valley to prevent a person from falling. Wireless beacon triggers can be used to cause cave and/or cliff obstacles to appear and/or disappear in the ANR system, resulting in varying difficulty levels with induction time and path length. . Sensor data can be used by the system to synchronize motion to the winding road. A patient-induced requirement can be a biomechanical response that directs a change in the underlying prescribed course. The system is configured so that wireless space-time beacon triggers affect changes in the AR 3D dynamic model. A temporal aspect of such wireless triggers is the ability to turn them on and off. This allows for maximum flexibility in planning the guidepath for the course the patient needs to take as part of the treatment session. This instantaneous target parameter can be a measure of walking speed or quality of walking. Successful navigation would follow the path without straying from the path or falling off a cliff. The system predicts levels that would become more difficult (e.g., more obstacles and can be configured to present a higher speed).

In another example, the AR 3D dynamic model is asked to move forward when the patient is standing or sitting still and a virtual object is presented and approaches each leg. It can be configured to simulate scenarios. For example, FIG. 36B shows a view of a footprint projected in front of a patient using AR. An ANR system can generate a virtual scene in which an object can approach the patient's left or right to encourage the patient to step sideways. Objects are presented as they approach the patient at a predefined tempo or beat and follow the decision tree described in FIG. Also, a vision of correct movement by the therapist or patient from previous treatments may be projected.

In another exemplary AR 3D dynamic model implementation, the ANR system can be configured to incorporate haptic feedback into therapy. Haptic feedback, for example, can be used by the system as a signal when the user gets too close to an object or people in the projected AR environment. Rhythmic tactile feedback can also be synchronized with auditory cues to amplify sensory input. The AR can also adaptively and individually cue the onset of movement, for example, during a frozen episode of walking in someone with Parkinson's disease.

In another example AR 3D dynamic model implementation, the ANR system can be further configured to incorporate optical head tracking. This tracking can be taken as feedback to an ANR system configured to activate auditory input in response to where their eyes or head are pointing. For example, in a person with left-sided neglect who only interacts with the right side of their environment, eye and head tracking may provide input on how much of the person's left hemisphere environment is involved. , causing the system to generate auditory cues to direct more attention to the left hemisphere. This data can also be used to track progress over time, as clinical improvement is measurable by the degree of cognition in each hemisphere. Another example of this is people with oculomotor disorders, which may be ameliorated by left-to-right visual scanning aligned to an external auditory rhythm.

In another example AR 3D dynamic model implementation, the ANR system can be configured to provide a digital presence of past sessions to display user improvement. These models can be replayed after a session to allow session-to-session comparisons or lifetime treatment comparisons. The digital presence of a past session (or an extended session), when combined with audio input of that session, can also be used as a mental imaging task to practice during walking sessions to reduce fatigue. can. The model displays differences in walking speed, cadence, stride length, and symmetry to help show how the user changes over time and how treatments can improve the user's gait. This presence can also be used by a therapist prior to a session to help prepare a training plan or procedure for subsequent sessions. The modeled entity can also be used by researchers and clinicians to better visualize the evolution of a patient's progress and reanimate it within a 3D image.

In another example AR 3D dynamic model implementation, the AR/VR environment synchronized with the music content includes an oval road, a spiral road, a meandering road, and a road that crosses other roads, as well as a dual road. Various walking or dancing patterns can be created to include the walking of the task. Dance rhythms such as tango have been shown to have benefits from neurological music therapy (NMT) and RAS that can be applied to the entire human body.

According to one or more embodiments, ANR systems can be configured to utilize AA techniques to enhance the entrainment process and provide environmental context to humans to aid in AR experiences. To enhance the recovery process, the system is configured to generate the exemplary AA experiences described further herein based on inputs obtained from the environment, sensor data, AR environment, entrainment, and other methods. can be done.

An AA example of a therapeutic/medical use case is addressing safety concerns and mitigating risks to patients undergoing exercise therapy. ANR systems improve situational awareness while listening to music on headphones by mixing external sounds that momentarily exceed the minimum loudness threshold of speech into therapeutic rhythmic, audio-cued content. can be configured as follows: Examples of external sounds are car horn sounds and emergency vehicle sirens, which are automatically synchronized to block normal auditory stimuli and give a person an awareness of potential danger. To perform this and other functions, hearing devices may have additional microphones and dedicated digital signal processing to perform this task.

In a further embodiment, an ANR system implementing AA provides multiple aspects of AA and spatial It can be configured to combine recognition operations. For example, if the patient's right side requires a greater degree of treatment, the content with audio cues can be spatially aligned to the right to emphasize. Exemplary systems and methods for neurological rehabilitation using techniques for rhythmic auditory stimulation to a specific side are described in co-pending and commonly owned, "Systems and Methods for Neurological Rehabilitation." No. 62/934,457, filed November 12, 2019, entitled Incorporated into

In a further embodiment, an ANR system implementing AA can be configured to provide unique auditory cues to improve spatial awareness of head position during gait training, thereby The user can be encouraged to keep their head up and centered and their eyes forward during the entrainment process or other CTA experience to improve balance and spatial awareness.

In a further embodiment, an ANR system that implements AA provides a binaural beat sound that is adapted to human physiology (e.g., respiratory rate, electrical brain activity (EEG), heart rate). can be configured to improve cognition and enhance memory by associating with The ANR system can be configured to provide a binaural beat audio signal input complementary to the RAS signal input. The real-time synchronization and quality of gait measurements being made by the system are complemented by physiological measurements as well. For example, a system configured for binaural beat audio uses differential frequency signals output to the left and right ears, the difference being 40 Hz, the "gamma" frequency of neural oscillations. These frequencies can reduce amyloid accumulation in patients with Alzheimer's disease and can aid cognitive flexibility. A second type of neural entrainment can be achieved simultaneously with biomechanical RAS entrainment by providing such audio signals to the user through the AA device while the user is performing RAS walking training. The network hypothesis for brain activation suggests that both locomotion and cognition are affected. Therefore, such auditory sensory stimulation entrains neural oscillations in the brain, and rhythmic auditory stimulation entrains the motor system.

In a further embodiment, an ANR system implementing AA is designed to provide a coherent sound field (e.g., spatially correct sound) when the patient turns his or her head or changes posture. Can be configured. A sound field is a virtual 3D image created by stereo speakers or headphones. The sound field allows the listener to hear the exact location of the sound source. An example of manipulating the sound field in a therapy session is to keep the virtual coach's voice "in front" of the patient, even while the patient's head is turned to the side. This feature can help avoid disorientation and, as a result, can create a more stable, predictable and safe audio experience while delivering therapy. This feature can also be combined with an AR virtual coach/therapist in front of a person in FIG. This functionality can also be combined with knowledge of the course or direction a person should take in the real world.

In a further embodiment, the ANR system creates a sound field in which virtual sound effects (e.g., encouragement and crowd footsteps) are in phase with the patient's visual gaze when the patient is synchronized with the virtual crowd. As such, AA can be configured to combine with Augmented Reality (AR). This sound can also create a perception of distance or proximity to an object. Altering spatial location or loudness in such a way can also be used as a targeted target in conjunction with AR and 3D images.

In a further embodiment, the ANR system can be configured to combine AA with Augmented Reality (AR) to create virtual musical instrument therapy. Musical instruments such as bells, drums, pianos, and guitars can be common training tools for patients. Creating a digital model of such an instrument and providing AA feedback upon interaction can give the patient an immersive experience and the perception that they are physically playing the instrument. This difficulty can also be changed to help the patient progress and show improvement over time. Modifications may include adding more keys on a piano and more strings on a guitar. In addition to virtual musical instruments, virtual musical scores and musical notation can also be displayed in real-time as the patient plays the instrument, either virtual or real. Other examples could combine the concepts discussed in connection with FIG. 19 and replace the connected hardware with the AR. Similar logic can be used for other documented interventions.

In a further embodiment, the ANR system can be configured to perform AA in combination with telepresence to provide the therapist with a spatially accurate audio experience. This sound can also be generated to create a perception of distance or proximity to an object. By changing the spatial position or loudness of the AA and leveraging the AR model, the system can more effectively determine if the patient is achieving the goals associated with playing the virtual device. , can be used to provide patients with a more accurate and special experience.

According to one or more embodiments, an ANR system can implement a kind of AA, a Rhythmic Stimulus Engine (RSE). The Rhythmic Stimulation Engine is a custom-made rhythmic auditory stimulus that embodies entrainment principles to facilitate therapeutic effects while generating original, custom, rhythmic auditory content for the patient. In conditions such as Parkinson's disease, it can also be beneficial to have a constant rhythmic "soundtrack" in the patient's environment. The RSE can also be configured to provide this continuous rhythmic background neural stimulation without the need to access pre-recorded music. In one example, an ANR system runs AA in combination with a rhythmic stimulation engine (RSE) and AR to generate rhythmic content between incoming biometric data and external audio input from the treatment environment. It can be configured to create fully synchronized feedback conditions for the AR/AA outputs. In another example, the system can be configured to interactively adjust the tempo of rhythmic audio content generated by the RSE according to the patient's walking cadence. In another example, the tempo and time signature of rhythmic audio content generated by RSE can also be adjusted interactively by the tuning accuracy and beat factor of patient-users, such as patients using canes or assistive devices. . In another example, the RSE can also provide neural stimulation that enhances the effectiveness of locomotion therapy in combination with assistive technologies such as reinforced suits, reinforced exoskeletons, and/or FES devices. In another example, the RSE can generate from a stored library of templates of traditional dance rhythms rhythmic audio content that can be therapeutic to the patient's upper body and limbs. This can also be extended by combining the above-mentioned AR techniques such as dancing crowds and virtual dance floors. In another example, machine learning techniques such as self-learning AI and/or rule-based systems use an inertial measurement unit (IMU) input that reports cadence and quality of gait parameters such as symmetry and gait cycle time variability. It can also generate real-time rhythms managed by Using unsupervised ML clustering or decision tree models, different walking patterns can serve as inputs to generative music systems.

According to one or more embodiments, the ANR system can perform a type of AA, or Sonification, which measures how close or far the patient is to the target target. Depending on how it means applying varying amounts of signal distortion to the music content. The degree and type of sonification help move the patient into correction. The novel combination of sonification and entrainment provides the musical content of some other biomechanical or physiological parameter that the individual can tune while providing a feedforward mechanism for auditory movement synchrony using entrainment. It is also possible to simultaneously provide a feedback mechanism using the distortion of . For example, the combination of increasing the volume of rhythmic cues and adding signal distortion to a musical signal has a greater effect than either method alone.

According to one or more embodiments, the ANR system can perform CTA in combination with neurotoxin injection as follows. CTA can also apply entrainment principles to work towards improving motor functions such as walking. Injection of neurotoxins can also be used to target improvement in gait by targeting muscle spasticity. These infusions take 2-4 days to take effect and last up to 90 days (eg, shelf life). Administration of CTA for entrainment principles (e.g., setting one or more parameters of CTA) can also target the efficacy curve of a neurotoxin infusion, and the duration of the infusion's effect is The training is not very intense and the training is intensified during the effective period.

According to one or more embodiments, the ANR system can be configured to synchronize heart rate or respiration rate to music content to calculate entrainment parameters instead of biomechanical motion parameters. Example use cases are for people with various forms of agitation such as dementia, Alzheimer's disease, bipolar disorder, schizophrenia. In this use case, the basic parameter can be determined by heart rate or respiration rate. Entrainment or stepwise entrainment can be determined by comparing musical content to heart rate or respiration rate. In addition, goals can be set to reduce disturbance levels and improve the quality of life for these people.

As can be appreciated from the foregoing description, a system for extended neurological rehabilitation of a patient, according to one or more embodiments, can include one or more of the following items.

A computer system having one or more physical processors is configured with software modules containing machine-readable instructions. The software modules may include a 3D AR modeling module that, when executed by the processor, configures the processor to generate and present augmented reality visual and audio content to the patient during the treatment session. The content includes visual elements that move in a defined spatial and temporal order and rhythmic audio elements that are output in beat tempo.

The computer system is also in communication with the processor and includes time-stamped biomechanical data of the patient related to movements performed by the patient in relation to the AR visual and audio content and one or more an input interface for receiving input including a physiological parameter measured using a plurality of sensors;

The software module also analyzes the time-stamped biomechanical data to determine the spatial and temporal relationships of the patient's movements to visual and audio components, and to determine patient entrainment to target physiological parameters. It also includes a critical thinking algorithm that configures the processor to determine the level. Additionally, the 3D AR modeling module further configures the processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the determined level of entrainment to the target parameters.

The systems, devices, methods, processes, and the like described above can be implemented using hardware, software, or a combination thereof as appropriate to the application. The hardware may include general purpose computers and/or special purpose computing devices. This includes implementation in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices or processing circuits, with internal and/or external memory. . It also or alternatively includes one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device or devices configurable to process electronic signals. can contain Implementation of the processes or apparatus described above may be stored and compiled or interpreted on one of the apparatus described above and heterogeneous combinations of processors, processor architectures, or various hardware and software combinations. , C or other structured programming language, object-oriented programming language such as C++, or any other high-level or low-level programming language, including assembly language, hardware description languages, languages and techniques for database programming It will further be appreciated that it may include written computer-executable code. In another aspect, the method may be embodied in a system that performs its steps, and in some manner may be distributed across multiple devices. Additionally, processing can be distributed across multiple devices, such as the various systems described above, or all functionality can be integrated into dedicated stand-alone devices or other hardware. In another aspect, means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequences and combinations are intended to be within the scope of this disclosure.

Embodiments disclosed herein comprise computer-executable or computer-usable code that performs any and/or all of the steps when executed on one or more computing devices. may include a computer program product comprising The code may be stored non-transitory in computer memory, which may be the memory in which the program executes (such as random access memory associated with a processor), or a disk drive, flash memory, or any other memory. It can be an optical device, an electromagnetic device, a magnetic device, an infrared device, or other device, or a combination of these devices. In another aspect, any of the systems and methods described above are embodied in any suitable transmission or propagation medium that carries/carries computer-executable code and/or any input or output from computer-executable code. can be

It is understood that the apparatus, systems, and methods described above are described by way of example, and not by way of limitation. Unless explicitly stated otherwise, the disclosed steps may be modified, supplemented, omitted, and/or rearranged without departing from the scope of the disclosure. Numerous variations, additions, omissions and other modifications will be apparent to those skilled in the art. In addition, the order or presentation of method steps in the foregoing description and drawings should not be confused with the listed steps, unless an order is explicitly required or otherwise apparent from the context. It is not intended to require you to do this in that order.

The method steps of the implementations described herein are defined in the terms of the appended claims, except where a different meaning is expressly provided or otherwise clear from the context. It is intended to include any method suitable for performing such method steps consistent with patentability. Thus, for example, performing step X includes any method suitable for causing another party, such as a remote user, remote processing resource (e.g., server or cloud computer), machine, etc., to perform step X. be Similarly, performing steps X, Y and Z involves directing or controlling any combination of such other individuals or resources to obtain the benefit of such steps. , and Z can be included. Accordingly, the method steps of the implementations described herein are subject to the terms of the appended claims, except where a different meaning is expressly provided or otherwise clear from the context. It is intended to include any method suitable for causing one or more other parties or entities to perform those steps consistent with the patentability of the scope. Such parties or entities need not be under the direction or control of any other party or entity and need not be located within any particular jurisdiction.

Furthermore, it should be appreciated that the methods described above are provided as examples. Unless explicitly stated otherwise, the disclosed steps may be modified, supplemented, omitted, and/or rearranged without departing from the scope of the disclosure.

It is understood that the methods and systems described above are described by way of example, and not by way of limitation. Numerous variations, additions, omissions and other modifications will be apparent to those skilled in the art. In addition, the order or presentation of method steps in the foregoing description and drawings is recited unless a specific order is explicitly required or otherwise apparent from the context. It is not intended to require the steps to be performed in this order. Thus, while specific embodiments have been illustrated and described, it is understood that various changes and modifications can be made therein in form and detail without departing from the spirit and scope of the disclosure. It will be clear to those skilled in the art.

Claims (19)

1. A system for augmented neurological rehabilitation of a patient comprising a computer system having a processor configured by software modules containing machine-readable instructions stored on a non-transitory storage medium, the system comprising:
The software module is
an AA/AR modeling module that, when executed by said processor, configures the processor to generate augmented reality (AR) visual content and rhythmic auditory stimuli (RAS) for output to a patient during a treatment session; AA/AR modeling, wherein the RAS includes a beat signal output at a beat tempo, and the AR visual content includes visual elements that move in a defined spatial and temporal sequence based on the beat tempo. an input in communication with the processor to receive real-time patient data including time-stamped biomechanical data of the patient associated with repetitive movements performed by the patient in time with the AR visual content and the RAS; an input interface, wherein the biomechanical data is measured using sensors associated with the patient;
including
The software module further:
analyzing the time-stamped biomechanical data to determine the temporal relationship of the repetitive motion of the patient to the beat signal output at the visual element and the beat tempo, and entrainment level to a target parameter; a critical thinking algorithm (CTA) module, which configures the processor to determine
including
The AA/AR modeling module further configures the processor to synchronize the AR vision and the RAS output to the patient and dynamically adjust based on the determined entrainment level. ,
system. said processor dynamically adjusting said AR visual content and said RAS based on said determined entrainment level by adjusting said beat tempo of said RAS in view of said entrainment level; 2. The system of claim 1, adjusting the defined spatial and temporal order of . each of the beat signals is output at a respective output time;
the tuning level is determined based on the timing of the repetitive motion relative to the respective output times of the beat signal;
3. The system of claim 2. The CTA module includes:
analyzing the time-stamped biomechanical data to determine the time of each of the repetitive motions;
measuring a temporal relationship between one or more respective times of the repetitive motion and one or more respective times of the associated beat signal;
determining entrainment margin based on the measured temporal relationships for one or more of the repetitive movements;
configure the processor to determine the tuning level by
the processor is configured to dynamically adjust one or more of the AR vision and the RAS based on the tuning margin;
4. The system of claim 3. the CTA module further configures the processor to determine whether the biomechanical or physiological data measured for the patient satisfies training goal parameters;
The AA/AR modeling module further configures the processor to dynamically adjust the AR visual content and the RAS output to the patient in response to the training goal parameters being underachieved.
3. The system of claim 2. 6. The system of claim 5, wherein the target parameter is the beat tempo and the training target parameter is rehabilitation outcome. an AR video output device configured to present the AR visual content to the patient;
an audio output device configured to output the RAS to the patient;
the sensor associated with the patient and configured to measure the time-stamped biomechanical data of the patient, the sensor comprising an inertial measurement unit (IMU);
2. The system of claim 1, further comprising: a sensor associated with the patient and configured to measure physiological data of the patient, wherein the training goal parameter is a physiological parameter;
further comprising
said physiological parameter is one or more of heart rate, blood oxygenation, respiratory rate, VO2, electrical brain activity (EEG);
6. The system of claim 5. The AR visual content is
a virtual treadmill animated such that a top surface of the virtual treadmill appears to approach the patient at a speed corresponding to the beat tempo;
a plurality of footprints superimposed on the top surface of the virtual treadmill, wherein the footprints are spatially arranged and appear to approach the patient at a speed corresponding to the beat tempo;
an animated person performing said repetitive motion at a speed corresponding to said beat tempo;
including a visual scene that includes one or more of
The system of claim 1. Modifying the AR visual content includes changing the spacing of the footprints to account for changes in the beat tempo; changing the speed at which the animated person performs the repetitive motion; one or more of changing the speed at which the virtual top surface of the treadmill appears to approach the patient; and changing the speed at which virtual obstacles or virtual scene perturbations appear in front of the patient. 2. The system of claim 1, comprising: 1. A method for augmented neurological rehabilitation of patients with physical disabilities, said method being performed on a computer system having a physical processor configured with machine-readable instructions which, when executed, perform said method. realized,
providing rhythmic auditory stimulation (RAS) output to the patient via an audio output device during a therapy session, the RAS comprising a beat signal output at a beat tempo;
generating AR visual content for output to a patient via an augmented reality (AR) display device, said AR visual content moving in a defined spatial and temporal sequence based on said beat tempo; generating, including visual elements that are output in synchronization with the RAS;
instructing the patient to perform repetitive movements in time with the beat signal of the RAS and the corresponding movement of the visual element of the AR visual content;
Receiving patient data in real-time, including time-stamped biomechanical data of the patient associated with the repetitive movements performed by the patient in concert with the AR visual content and the RAS, wherein the biomechanical data is measured using a sensor associated with the patient;
analyzing the time-stamped biomechanical data to determine the temporal relationship of the repetitive motion of the patient to the beat signal output in response to the visual element and the beat signal to provide entrainment room; and
synchronizing the AR visual content and the RAS output to the patient and dynamically adjusting based on the determined entrainment margin that does not meet a predetermined entrainment margin;
continuing the treatment session with the adjusted AR visual content and the RAS;
A method, including measuring the biomechanical data from the patient, comprising providing a sensor associated with the patient; and measuring one or more of motion, acceleration, and pressure associated with movement of the patient. 12. The method of claim 11, comprising measuring, further comprising measuring. Dynamically adjusting the AR visual content and the RAS based on the determined tuning margin comprises: adjusting the beat tempo of the RAS based on the tuning margin; 12. The method of claim 11, comprising adjusting the adjusted spatial and temporal order to synchronize with the adjusted beat tempo. comparing the beat tempo to training target parameters including a target beat tempo;
dynamically adjusting the RAS and the AR visual content as a function of the tuning margin determined with respect to the comparison and the beat tempo;
14. The method of claim 13, further comprising: If the tuning margin satisfies a specified level and the beat tempo is below the target beat tempo, increasing the beat tempo of the RAS toward the target beat tempo to increase the specified level of the visual element. 15. The method of claim 14, further comprising adjusting the spatial and temporal order to synchronize with the adjusted beat tempo. Each of the beat signals is output at a respective output time based on the beat tempo, and measuring the entrainment margin includes determining the onset time of each of the plurality of repetitions of each of the associated beat signals. 12. The method of claim 11 comprising real-time comparison with output time, wherein the onset time for a given repetitive motion is the time at which a defined identifiable event occurs during the given repetitive motion. the method of. determining whether patient data, including the biomechanical or physiological data measured for the patient, satisfies training goal parameters;
further comprising
The AA/AR modeling module further configures the processor to dynamically adjust the AR visual content and the RAS output to the patient in response to the training goal parameters being underachieved.
12. The method of claim 11. further comprising measuring the physiological data from the patient comprising providing a sensor associated with the patient and configured to measure a physiological parameter, wherein the physiological parameter is heart rate, respiration and VO2 max. The AR visual content is
a virtual treadmill animated such that a top surface of the virtual treadmill appears to approach the patient at a speed corresponding to the beat tempo;
a plurality of footprints superimposed on the top surface of the virtual treadmill, wherein the footprints are spatially arranged and appear to approach the patient at a speed corresponding to the beat tempo;
an animated person performing said repetitive motion at a speed corresponding to said beat tempo;
including a visual scene that includes one or more of
12. The method of claim 11.