WO2022207910A1

WO2022207910A1 - Balance prosthesis and auditory interface device and computer program

Info

Publication number: WO2022207910A1
Application number: PCT/EP2022/058761
Authority: WO
Inventors: Bálint VÁRKUTI; Saman HAGH GOOIE; Ricardo SMITS SERENA
Original assignee: CereGate GmbH
Priority date: 2021-04-01
Filing date: 2022-04-01
Publication date: 2022-10-06

Abstract

The present disclosure relates to a balance prosthesis device comprising: a sensor module configured to obtain at least one sensor signal indicative of a balance or equilibrium state of the individual, a processing module operably connected to the receiver module and configured to determine at least one neurostimulation signal based at least in part on the obtained sensor signal and a transmitter module operably connected to the processing module and configured to transmit the determined neurostimulation signal to a neurostimulation device of the individual, or a neurostimulation module operably connected to the processing module, wherein the neurostimulation signal is configured to elicit an artificial sensation in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not directly associated with vestibulocortical pathways of the individual, wherein the elicited artificial sensation provides a balance indication to the individual that is derived at least in part from the obtained sensor signal in order to support, mimic, substitute or enhance the natural sense of balance of the individual. The present disclosure also relates to an auditory neural interface device for sound perception by an individual that may be used as a hearing aid. The auditory neural interface device comprises a receiver module configured to receive sound signals, a processing module operably connected to the receiver module and configured to encode a received sound signal as a multi-channel neurostimulation signal; the neurostimulation signal being configured, to directly stimulate afferent sensory neurons of the central nervous system, CNS, of the individual and thereby to elicit, for each channel of the neurostimulation signal, one or more non-auditory, preferably somatosensory, perceptions in a cortex area of the individual, wherein each channel of the neurostimulation signal is associated with a different non-auditory perception; and a neurostimulation module operably connected to the processing module and configured to apply the multi-channel neurostimulation signal to a neurostimulation means of the individual; or a transmitter module configured to transmit the multi-channel neurostimulation signal to a remote neurostimulation device. The present disclosure further relates to a computer program comprising instructions for implementing such balance prosthesis and auditory neural interface devices when being executed by a neurostimulation device or system.

Description

BALANCE PROSTHESIS AND AUDITORY INTERFACE DEVICE AND COMPUTER PROGRAM

1. Technical field

The present invention relates to computer brain interface devices, methods, systems and computer programs that may be used to substitute, mimic, support or enhance a person’s natural sense of balance in a homologous or non-homologous manner. The present disclosure also relates to an auditory neural interface device for supporting or enabling sound perception by an individual.

2. Technical Background

The vestibular system plays a major role in maintaining body equilibrium and balance by providing information on the motion and spatial orientation of the head in space. This is possible through existence of vestibular periphery organs in the inner ear structure on each side of the head. The balance information together with information from other senses such as vision and proprioception is relayed to higher integrative neural centers within the central nervous system via afferent sensory nerve fibers.

This information enables the central nervous system to generate motor reflexes to keep body's center of gravity over its base of support e.g., an area bounded to foot soles while in standing position.

A range of neuropathologies can impact body’s normal balance function via affecting peripheral or central components of the vestibular or balance-related pathways. Several acute and chronic neuropathies such as brain tumor, traumatic brain injury, neurodegenerative disease (e.g. Parkinson’s Disease, Alzheimer’s disease), neuromuscular abnormalities, or psychiatric disorders (e.g. anxiety) exhibit balance dysfunction and dizziness which are major factors in heightened risk of falls and gait abnormalities. In US 6,546,291 a wearable balance prosthesis is described that provides information indicative of a wearer's spatial orientation. The balance prosthesis includes a wearable motion sensing system and a signal processor in communication with the motion sensing system. The signal processor provides an orientation signal to an encoder. The encoder generates a feedback signal based on an estimate of the spatial orientation of the body and provides that signal to a stimulator coupled to the wearer's nervous system. For instance, the stimulator may stimulate peripheral sensory organs such as the eye, the ear or the skin. Alternatively, the nervous system is directly stimulated via a stimulation electrode stimulating one of the nerves used by a properly functioning vestibular system.

Similarly, EP2974770A1 relates to an implantable vestibular prosthesis comprising an implantable nerve stimulation device that has a sensor system, a data processor in communication with the sensor system, and a nerve stimulation system in communication with the data processor and constructed to provide electrical stimulation to at least one branch of at least one vestibulocochlear nerve.

US 2008/0140137 relates to a sensory data integration system for integrating sensory data generated by a first sensory substitution device and sensory data generated by a second sensory substitution device. The system includes a processor configured to generate an orientation signal indicative of a difference between a subject's orientation and an acceptable orientation. The orientation signal is based on the sensory data generated by the first sensory substitution device and the sensory data generated by the second sensory substitution device. For instance, a stimulator may provide electrical, tactile, auditoiy, or visual orientation cues. Electrical stimulation maybe provided, for example, by stimulating the subject with a low ampere electrical current.

US 2020/0060602 relates to a motion analysis system that includes an image capture device, at least one accelerometer, and CPU configured to receive a first set of motion data from the image capture device related to at least one joint of a subject while the subject is performing a task and receive a second set of motion data from the accelerometer related to the at least one joint of the subject while the subject is performing the task. The CPU also calculates kinematic and/or kinetic information about the at least one joint of a subject from a combination of the first and second sets of motion data, and outputs the kinematic and/or kinetic information for purposes of assessing a movement disorder. US 8,702,629 B2 discloses a closed-loop deep brain stimulation system. A movement measuring device, worn by a subject, and comprising a sensor module and transceiver unit continually measures the subjects movement data. The transceiver unit correlates with a database and uses a trained algorithm to optimize a custom deep brain stimulation treatment protocol for the subject.

Similarly, US 9,974, 478 relates to systems for helping subjects improve safety and efficiency of their movements via monitoring a subject's movement to detect or predict unsafe, undesirable, or impaired movements, or symptoms of movement disorders, and also a system for providing possible treatment methods for such conditions such as providing cues or stimuli to the subject when such unsafe or undesirable movements, instabilities or symptoms are detected or predicted.

US 2009/0082831 relates to a wearable vestibular stimulation system placed inside the ear canal. Further prior art that may be relevant for understanding the technical background of the present invention is provided by US 7,647,120, US 10,376,739, US 9,149,222, US 9,339,649, WO 2018/059431, WO 2012/162658 and US 7,313,440.

The balance support systems known from the prior art have various deficiencies. For instance, if wearable tactile transducers are employed (e.g., see US 6,546,291) for assisting a patient with deteriorated sense of balance the number of transducers that are required scales with the specificity of the sensory feedback. Thus, such wearable transducer systems are bulky and expensive and have a large power consumption in particular when precise balance feedback is desired. Moreover, such tactile transducers and any other kind of balance feedback that is administered via the natural sensory organs of a patient only work if the sensory organs are healthy. In addition, if multi- sensoiy balance feedback is desired different types of transducers (tactile, visual, auditory etc.) have to be integrated into a single balance support system, which further increases complexity, cost and power consumption. Moreover, direct sensory substitution via electro-stimulation of the vestibular system only works if the vestibular system is only partially impaired, e.g. if only the receptor cells of the vestibular sensory organ within the inner ear are impaired but the remaining part of the vestibular nervous system is intact. Such an approach may thus fail if the impairment is located upstream in the vestibular sensory pathway, e.g. in the brainstem. Thus, for many patients direct sensory substitution of the vestibular system is not a preferred treatment option. In addition, in most cases, patient will not have suitable neurostimulation equipment already implanted that can be used to stimulate the vestibular system. Accordingly, for such a treatment implantation of dedicated stimulation electrodes or similar interface devices may be required.

It is thus one problem underlying the present invention to at least partially overcome some of the deficiencies of conventional balance supports systems outlined above. Sound perception is essential for survival and living a normal life in modern society. In particular, the communication between humans relies on spoken language. Also experiencing the joy of music is typically not possible without being able to perceive sound. Proper communication between humans ensures the ability of individuals to develop and evolve in a social environment. This is particularly important for children at their early stage in life.

Presently, more than over 5% of the human population suffer from a disabling hearing loss. This has a detrimental effect in terms of the communication and participation in groups of individuals. It is estimated that this number will increase to 10% by 2050. There are three different types of hearing loss: conductive hearing loss, sensorineural hearing loss, and mixed hearing loss.

Conductive hearing loss can usually be treated or improved byway of a surgery or infection treatment.

Treatment of sensorineural hearing loss on the other hand typically requires a specific type of hearing aid, for instance a cochlear implant or auditory brainstem implant (ABI). It is known that merely about 1 in 20 patients who could potentially benefit from such an implant, do actually receive it. This is mainly attributed to a limited access to the complex surgical procedures necessary for implantation. Further, implanting these devices in the skull can have adverse effects to the patient, because there is the inherent risk of side effects, such as nerve damage, dizziness and/or balance problems, hearing loss, tinnitus, leaks of the fluid around the brain, meningitis etc. Even further, children who could hear some sounds and/or speech with hearing aids, may not be eligible for cochlear implants although improved hearing capability would drastically improve their personal development. Further risks associated with existing cochlear implant technology, or the related surgical procedures are, for instance, risk of losing residual hearing, inability to understand language, complex explanation procedure in case of device failure, and more.

A further detrimental effect of cochlear implants is that they cannot provide for hearing aid when deafness is caused by an injury or an absence of the auditory nerve fibers themselves, for instance in case of Neurofibromatosis type 2. For this scenario, ABIs are used as an alternative that bypasses the cochlear nerve to electrically stimulate second order neurons in the cochlear nucleus. However, implanting an ABI is an extremely invasive surgery accompanied by a high risk of failure, and even if successful, most patients do not achieve open set speech perception even with extensive training.

The folio-wing prior art documents briefly describe the technical background of the present disclosure relating to auditory interface technology.

US 7,251,530 Bi relates to errors in pitch allocation within a cochlear implant. Those errors are said to be corrected in order to provide a significant and profound improvement in the quality of sound perceived by the cochlear implant user. In one example, the user is stimulated with a reference signal, e.g., the tone “A” (440 Hz) and then the user is stimulated with a probe signal, separated from the reference signal by an octave, e.g., high “A” (880 Hz). The user adjusts the location where the probe signal is applied, using current steering, until the pitch of the probe signal, as perceived by the user, matches the pitch of the reference signal, as perceived by the user. In this manner, the user maps frequencies to stimulation locations in order to tune his or her implant system to his or her unique cochlea.

Research article “Can ECAP Measures Be Used for Totally Objective Programming of Cochlear Implants” , 10.1007/S10162-013-0417-9 (DOI), published online on September 19, 2013, relates to an experiment with eight cochlear implant subjects to investigate the feasibility of using electrically evoked compound action potential (ECAP) measures other than ECAP thresholds to predict the way that behavioral thresholds change with rate of stimulation, and hence, whether they can be used without combination with behavioral measures to determine program stimulus levels for cochlear implants.

US 9,786,201 B2 and US 9,679,546 B2 both relate to vibratory motors that are used to generate a haptic language for music or other sound that is integrated into wearable technology. This technology enables the creation of a family of devices that allow people with hearing impairments to experience sounds such as music or other auditory input to the system. For example, a “sound vest” or one or more straps comprising a set of motors transforms musical input to haptic signals so that users can experience their favorite music in a unique way and can also recognize auditory cues in the user's everyday environment and convey this information to the user using haptic signals.

EP 3574951 Bi relates to an apparatus and method for use in treating tinnitus, which employs a sound processing unit, a tactile unit, and an interface therebetween. The tactile unit comprises an array of stimulators each of which can be independently actuated to apply a tactile stimulus to a subject, and the tactile unit comprises an input for receiving a plurality of actuation signals from the interface and directing individual actuation signals to individual stimulators.

US 9,078,065 B2 relates to a method and a system for presenting audio signals as vibrotactile stimuli to the body in accordance with a Model Human Cochlea. Audio signals are obtained for presentation. The audio signals are separated into multiple bands of discrete frequency ranges that encompass the complete audio signal. Those signals are output to multiple vibrotactile devices. The vibrotactile devices maybe positioned in a respective housing to intensify and constrain a vibrational energy from the vibrotactile devices. Output of the vibrotactile devices stimulate the cutaneous receptors of the skin at the locations where the vibrotactile devices are placed. Applicant’s own DE 102019202666 At relates to a system for providing neural stimulation signals. The system is configured to elicit sensory percepts in the cortex of an individual that may be used for communicating conceptual information to an individual. The system comprises means for selecting at least one neural stimulation signal to be applied to at least one afferent axon directed to at least one sensory neuron in the cortex of the individual. The at least one neural stimulation signal corresponds to the conceptual information to be communicated. The system further comprises means for transmitting the at least one neural stimulation signal to stimulation means of the individual. US 2016/0012688 Ai relates to providing information to a user through somatosensory feedback. A hearing device is provided to enable hearing-to-touch sensory substitution as a therapeutic approach to deafness. Byway of signal processing on received signals, the hearing device may provide better accuracy with the hearing-to-touch sensory substitution. The signal processing includes low bitrate audio compression algorithms, such as linear predictive coding, mathematical transforms, such as Fourier transforms, and/or wavelet algorithms. The processed signals may activate tactile interface devices that provide touch sensation to a user. For example, the tactile interface devices maybe vibrating devices attached to a vest, which is worn by the user. For further reference, the following prior art documents may be in part be relevant for characterizing the technological background of the present invention.

US 8,065,013 B2 relates to a method of transitioning stimulation energy (e.g., electrical stimulation pulses) between a plurality of electrodes that are implanted within a patient. US 10,437,335 B2 relates to a wearable Haptic Human/Machine Interface (HHMI) which receives electrical activity from muscles and nerves of a user. An electrical signal is determined having characteristics based on the received electrical activity. The electrical signal is generated and applied to an object to cause an action dependent on the received electrical activity. The object can be a biological component of the user, such as a muscle, another user, or a remotely located machine such as a drone.

US 10,869,142 B2 relates to a new binaural hearing aid system, which is provided with a hearing aid in which signals that are received from external devices, such as a spouse microphone, a media player, a hearing loop system, a teleconference system, a radio, a TV, a telephone, a device with an alarm, etc., are filtered with binaural filters in such a way that a user perceives the signals to be emitted by respective sound sources positioned in different spatial positions in the sound environment of the user, whereby improved spatial separation of the different sound sources is facilitated.

3. Summary of the invention Part of the above-mentioned problems are at least partially solved by a balance prosthesis device, method and computer program as specified by the independent claims l, 13 and 14. Exemplary embodiments of the present invention are specified in the corresponding dependent claims.

Generally, the present invention allows to implement a novel closed-loop approach to restore or enhance a person’s sense of balance. This approach is based on direct neurostimulation of afferent sensory axons (e.g., thalamocortical axons, afferent sensory axons of the brain stem or spinal cord and / or afferent sensory axons of the peripheral nervous system) targeting directly or indirectly (i.e., via multi-synaptic afferent pathways) sensory neurons in a specific sensory cortex area to with highly specific, fine-grained and multi-dimensional balance feedback information. More specifically, the present invention provides a balance prosthesis device for an individual, comprising: a sensor module configured to obtain at least one sensor signal indicative of a balance or equilibrium state of the individual; a processing module operably connected to the receiver module and configured to determine at least one neurostimulation signal based at least in part on the obtained sensor signal; and a transmitter module operably connected to the processing module and configured to transmit the determined neurostimulation signal to a neurostimulation device of the individual; or a neurostimulation module operably connected to the processing module, wherein the neurostimulation signal is configured to elicit an artificial sensation / sensory perception in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not directly associated with vestibulocortical pathways of the individual; and wherein the elicited artificial sensation provides a balance indication to the individual that is derived at least in part from the obtained sensor signal in order to support, mimic, substitute or enhance the natural sense of balance of the individual.

The various modules of the devices and systems disclosed herein can for instance be implemented in hardware, software or a combination thereof. For instance, the various modules of the devices and systems disclosed herein may be implemented via application specific hardware components such as application specific integrated circuits, ASICs, and / or field programmable gate arrays, FPGAs, and / or similar components and / or application specific software modules being executed on multi purpose data and signal processing equipment such as CPUs, DSPs and / or systems on a chip (SOCs) or similar components or any combination thereof. For instance, the various modules of the balance prosthesis device discussed above may be implemented on a multi-purpose data and signal processing device configured for executing application specific software modules and for communicating with various sensor devices and / or neurostimulation devices or systems via conventional wireless communication interfaces such as a NFC, a WIFI and / or a Bluetooth interface.

Alternatively, the various modules of the balance prosthesis device discussed above may also be part of an integrated neurostimulation apparatus, further comprising specialized electronic circuitry (e.g. neurostimulation signal generators, amplifiers etc.) for generating and applying the determined neurostimulation signals to a neurostimulation interface of the individual (e.g. a multi-contact spinal cord stimulation electrode, a deep brain stimulation (DBS) electrode, a peripheral sensory nerve stimulation electrode etc.).

The neurostimulation signals generated by the balance prosthesis device described above may for instance also be transmitted to a neuronal stimulation device comprising a signal amplifier driving a multi-contact DBS electrode, spinal cord electrode, etc. that may already be implanted into a patient’s nervous system for a purpose different than providing a balance indication. Alternatively, dedicated DBS-like electrodes or spinal cord stimulation electrodes may be implanted for the purpose of applying the neurostimulation signals generated by the balance prosthesis device via established and approved surgical procedures that were developed for implantation of conventional DBS electrodes or spinal cord stimulation electrodes etc. Further, as mentioned above the balance prosthesis device may also be integrated together with a neuronal stimulation device into a single device.

It is important to note that the balance indication that is communicated by the artificial sensory perception elicited by the neurostimulation signal differs from mere sensory substitution. As will be explained in detail below any kind of abstract information that can be used to substitute, mimic, support or enhance a person’s natural sense of balance can be transmitted to the individual with a balance prosthesis device according to the present invention. For instance, different neurostimulation signals maybe configured to elicit different specific artificial sensory perceptions (e.g. a tough sensation in the left hand, the lower back, etc.) having different characteristics (e.g. different intensities, frequencies or secondary sensory qualities such as a texture pitch, timbre, color etc.). As will be explained in more detail below, the balance prosthesis device provided by the present invention may then be calibrated such that the different lO characteristics of the elicited artificial sensory perceptions indicate different balance indications such as a degree and /or a direction of a body tilt, an inclination angle of a walking surface, a predictive falling warning, etc.

The present invention also provides a method for providing a balance indication to an individual, comprising the following steps: obtaining at least one motion sensor signal indicative of a balance or equilibrium state of the individual; determining at least one neurostimulation signal based at least in part on the obtained sensor signal; and transmitting the determined neurostimulation signal to a neurostimulation device or module of the individual, wherein the neurostimulation signal is configured to elicit an artificial sensation in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not associated with the natural sense of balance of the individual and wherein the elicited artificial sensation provides a balance indication to the individual that is derived at least in part from the obtained motion sensor signal in order to support, mimic, substitute or enhance the natural sense of balance of the individual.

The present invention also provides a computer program, comprising instructions for carrying out the method described above, when being executed by the signal processing and transceiver modules of a signal and data processing device, a neuronal stimulation device or system.

Further, the sensor module may be configured to obtain at least one motion sensor signal via a wired or wireless interface, such as an accelerometer or a gyroscope signal; Additionally or alternatively, the sensor module may also comprise a motion sensor such as an accelerometer or a gyroscope (see Fig. 8 below). Further, the sensor module may be configured to obtain, via a wired or wireless interface, an auxiliary sensor signal originating from an auxiliary sensor device such as a camera, a LIDAR sensor, a GPS system, a pressure sensor or an elevation sensor, etc.

Further, in some embodiments of the present invention the processing module may be configure to determine, based at least in part on the obtained sensor signal, an estimate of the current body position of the individual with respect to a reference body position and / or an estimate of a future body position with respect to the reference body position.

For instance, the body position of the individual may be characterized by one or more of the following parameters: a body tilt of the individual in the coronal and / or the sagittal plane; a rate of change of the body tilt of the individual in the coronal and / or the sagittal plane, a deviation of the center of gravity of the body of the individual from a reference position or range for the center of gravity and a rate of change of the deviation of the center of gravity from the reference position or range.

In this way the balance prosthesis device provided by the present invention is enabled to determine and transmit highly specific balance indications directly to the brain of the individual. As mentioned above the neurostimulation signal may be determined based on processed input data from multiple sensors such as acceleration sensors, gyroscopes video cameras, pressures sensors and LIDAR sensors etc. The processed information may then be utilized to trigger neurostimulation by activating appropriate perceptual / sensory communication channels. In this manner, the present invention allows the use accurately timed message blocks that provide effective and automatic sensory feedback cues to the individual to substitute, mimic, support or enhance the natural sense of balance of an individual.

Further, the balance prosthesis device described above may be further configured to access a data storage device storing a plurality of relations, specific for the individual, associating a plurality of neurostimulation signals with a plurality of corresponding homologous or non-homologous balance indications or auxiliary balance support information. In some embodiments, the balance prosthesis device may also include the data storage device storing the plurality of relations.

For instance, the data storage device may contain a personalized communication library for the individual, the library storing the relations between a plurality of different balance indications or auxiliary balance support information and a plurality of corresponding neurostimulation signals. Such a stimulation signal library can be calibrated for each individual through neuroimaging and / or individualized testing of the individual. Neuroimaging may first be used to identify theoretically possible ranges of activation for an individual stimulation electrode while individualized testing determines which points in the parameter space of neurostimulation signal parameters can be perceived and decoded by the cortex of the individual. It should be emphasized that conscious individualized testing of an individual is merely one specific example of how to generate the individualized relations stored in the memory. In other embodiments such relations may also be obtained from unconscious patients, e.g. through the non-invasive observation of corresponding functional MRI responses on the somatosensory cortex or EEG recordings. Further, once the communication library is established or while it is being established for an individual a specific training procedure can be executed that links a specific artificial sensory perception to the corresponding homologous or non- homologous balance indication. As long as the cortex of the individual responds to classical conditioning, pair learning can be executed. In the context of the present invention, such a pair consists of a given artificial sensory perception corresponding to a given neurostimulation signal and the corresponding balance indication or auxiliary balance support information to be associated with said given artificial sensory perception and the corresponding neurostimulation signal. Importantly, the type of information to be conveyed via the balance prosthesis device described above whether it is a balance indication or similar information can be chosen more or less freely. Any information or message which can be broken down into message blocks (i.e. pieces of conceptual information that can be decoded by the cortex of an individual) can be transmitted. This includes (quasi-) continuous balance indications such as a (quasi-) continuous indication of a desired compensatory body movement or other information that may be relevant for maintaining once balance (e.g. a contact pressure difference across a foot of the individual, an inclination angle of a walking surface etc.). In particular, the specific relations may be based at least in part on one or more of the following: conceptual or perceptual learning data for the individual, neuro-imaging data for the individual, electrophysiological measurement data for the individual, neuronal connectivity information for the individual, electric field simulation data for the employed neurostimulation interface and neuronal excitability model data for the individual.

In this way, even complex balance indications (see Fig. 5 discussed in section 5. below) can be associated with corresponding artificial sensory perceptions that are specific for each individual.

In particular, the neurostimulation signal may also be configured to stimulate afferent sensory axons of the spinal cord, the brain or the peripheral nervous system projecting directly (i.e., via a monosynaptic pathway) or indirectly (i.e., via a multi-synaptic pathway) to the thalamus or the cortex. For instance, if the neurostimulation signal is to be applied via a conventional spinal cord stimulation electrode the signal parameters of the neurostimulation signal maybe adjusted such that action potentials are elicited in specific sub-populations of afferent sensory nerve fibers of the spinal cord, e.g., in a set of axons projecting via multiple synapses to somatosensory neurons in a specific sensory cortex area.

In general, the sensational modality, location, type, and intensity of the artificial sensory perception that is elicited in the cortex in response to these afferent action potentials can be controlled via precise electrode location and selection of neurostimulation parameters. The present invention uses such artificial sensations to transmit information directly to the brain in form of discrete or continuous message blocks by forming the desired perceptual channel. The perceptual channels may be established via a single or via multiple electrical contacts of a spinal cord stimulation electrode, a peripheral stimulation electrode and / or a DBS electrode which are electrically activated with calibrated neurostimulation parameters to deliver specific sensory messages to the individual. The sensation modality of the respective perceptual channel may include tactile, proprioceptive, visual, or auditory sensations based on the application or location and orientation of the implanted stimulation electrode.

For instance, perceived laterality or location of the elicited artificial sensation may indicate (e.g. in a homologous manner) a direction of a body tilt of the individual or a direction of a compensatory movement to decrease the body tilt and / or a perceived intensity of the elicited artificial sensation may encodes an angle or a degree of a body tilt of the individual relative to a reference position or range or an angle or degree of a compensatory movement to decrease the body tilt. Using such a homologous encoding scheme (for examples see also Fig. 2, Fig. 3 and Fig, 4 discussed in section 5 below) ensures that the balance indications provided by the balance prosthesis device can intuitively understood by the individual without much adaptation or learning necessary. Thus, quick and easy acceptance of the prosthesis device maybe facilitated which maybe particularly desirable for mentally impaired patients.

Further, e.g., in a non-homologous manner, the perceived repetition rate of the elicited artificial sensation may encode a terrain characteristic such as an inclination angle of a walking surface the patient is walking on or a remaining distance to an obstacle. Such secondary balance indications may for example be derived from video camera or LIDAR sensor signals and /or from pressure sensors measuring the contact pressure of the feet of the individual with the walking surface.

Further, a secondary sensory quality of the elicited artificial sensation such as the texture of a somatosensation, the color of a visual sensation or the tone, pitch or timbre of an auditory sensation may encode body balance support information / secondary balance indications such as an inclination of a walking surface or a remaining distance to an obstacle.

In this manner, the natural sense of balance of an individual cannot only be substituted, mimicked, or supported but even enhanced to integrated sensory information that would not be accessible even to a healthy individual.

To improve safety during operation, the processing module of some embodiments may be configured to detect, based at least in part on the obtained motion sensor signal and / or the auxiliary sensor signal, preferably by using a trained machine learning system, whether the body of the individual is at risk to fall and in response to said detection: to generate a neurostimulation warning signal that is configured to elicit an artificial sensation in a specific sensory cortex area providing a falling warning to the individual. In addition, the neurostimulation signal may be synchronized with a walking pace of the individual to provide a continuous body tilt correction indication improving the gait stability of the individual while walking. In a further aspect, the present invention provides a balance prosthesis system, comprising one of the balance prosthesis devices discussed above; and one or more implanted or wearable motion sensors providing input signals to the sensor module of the balance prosthesis device. Such a system may for example include one or more of the following sensor devices providing further input signals to the sensor module of the balance prosthesis device: a camera device, a LIDAR sensor device, a GPS system / receiver, a pressure sensor measuring the contact pressure between a foot of the individual and a walking surface and / or at least two pressure sensors measuring the difference in contact pressure between two points on a walking surface to determine an inclination of the walking surface.

In some particularly advantageous embodiments, the balance prosthesis system may comprise a spinal cord stimulation device comprising a set of implanted spinal cord leads targeting somatosensory ganglions or afferent sensory nerve fibers within or adjacent to the spinal cord.

The present invention can also be used to enhance or reinforce the natural sense of balance of an individual. For instance, artificial sensory perceptions encoding a reinforcing balance indication may help to resolve perceptual conflicts between the natural sense of balance and visual perception and thereby help to overcome motion sickness (see Fig. 9 below).

In addition the present invention also provides a remote balance sensing device for an individual, comprising: a sensor module configured to obtain at least one sensor signal indicative of a balance or equilibrium state of a remote moving object; a processing module operably connected to the receiver module and configured to determine at least one neurostimulation signal based at least in part on the obtained sensor signal; and a transmitter module operably connected to the processing module and configured to transmit the determined neurostimulation signal to a neurostimulation device of the individual; or a neurostimulation module operably connected to the processing module, wherein the neurostimulation signal is configured to elicit an artificial sensation in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not directly associated with vestibulocortical pathways of the individual; and wherein the elicited artificial sensation provides a balance indication for the remote moving object to the individual that is derived at least in part from the obtained sensor signal.

Such a remote balance sensing device may provide an individual, e.g., pilot remotely controlling, e.g., via a brain-computer interface, a drone, car, ship robot etc. with an artificial balance perception for the remote object thereby substantially improve the individual’s performance in controlling the movement of the remote moving object.

As explained above, several attempts have been made in the prior art to provide for an improved hearing aid. However, the prior art methods, devices and systems have various deficiencies. As discussed above, the methods, devices and systems known from the prior art may, for example, not be suited to establish sound perception in an easy and flexible manner and / or may require difficult to perform and / or expensive neurosurgical procedures (e.g., implantation of Cochlea or ABI electrodes). Further, some prior art solutions partially rely on complex devices such as wearables which are bulky and cumbersome and/ or might interfere in an unnatural way with the normal behavior of an individual. It is thus a problem underlying the present invention to overcome such and similar deficiencies of previous technologies.

This and similar problems are at least partially solved by the auditory neural interface device, system and computer program specified in the appended claims 16 to 30. The provided auditory neural interface device, system and computer program allow to restore or support sound perception even for individuals that cannot receive a cochlear implant or ABI and / or provide high-fidelity sound perception that cannot be achieved with prior art technologies. Specifically, some embodiments provide an auditory neural interface device for supporting or enabling sound perception by an individual, comprising: a receiver module (or receiver) configured to receive sound signals (e.g., analog or digital electrical signals generated by a microphone or obtained from remote sound transducer apparatus), a processing module (or processor) operably connected to the receiver module and configured to encode a received sound signal as a multi-channel neurostimulation signal. The multi-channel neurostimulation signal is configured to directly stimulate afferent sensory neurons of the central nervous system, CNS, (i.e., of the brain and / or the spinal cord) of the individual and thereby to elicit, for each channel of the multi-channel neurostimulation signal, one or more non-auditory, preferably somatosensory, perceptions in a cortex area of the individual, wherein each channel of the neurostimulation signal is associated with a different non-auditory perception. The device further comprises a neurostimulation module (or neurostimulator) operably connected to the processing module and configured to apply the multi-channel neurostimulation signal to a neurostimulation means of the individual (e.g., a multi-channel neurostimulation electrode). Alternatively, the device comprises a transmitter module configured to transmit the multi-channel neurostimulation signal to a remote neurostimulation device which in turn is configured to apply the multi-channel neurostimulation signal to a neurostimulation means of the individual.

Essentially, in this manner, it is possible to implement an improved hearing aid based on using low-invasiveness implants to the central nervous system, CNS, (e.g., Spinal- Cord-Simulation, SCS) as a gateway to communicate sound-analogues to the recipient’s brain, to either augment hearing for patients with partial hearing loss or to give a new opportunity to a broader patient population who cannot receive cochlear implants.

Further, in some embodiments, encoding by the processing module may comprise applying a filter operation to the received sound signal to generate a plurality of subcomponent signals of the sound signal and mapping each subcomponent signal to a different channel of the multi-channel neurostimulation signal. For instance, the sound signal can be decomposed with a method that is chosen on the basis of how much information the neural interface can transmit.

Further, said filter operation may involve performing spectral analysis, wavelet analysis, principal component analysis, independent component analysis, using a filter bank, and/or a combination thereof. In a simple example, as illustrated in Fig. 14 in section 15 below, a received sound signal (e.g., a sample of speech or a sample of a piece of music, etc.) maybe subdivided (e.g., via a bank of N bandpass filters) into N subcomponent signals corresponding to N different frequency bands.

Via encoding sound signals in multiple, non-auditory perceptual channels the neural interface device can enable or support sound perception even for patients that cannot i8 be treated via conventional cochlear implants or ABIs. Moreover, not being limited to the physiologic structure and function of the auditory nerve and upstream auditory processing may substantially improve flexibility, channel count and the fidelity of sound signal representation. In this manner, even complex auditory stimuli such as speech in a cocktail party environment or classical music can be perceived with sufficient fidelity.

In the same manner as an infant’s brain is capable of associating syntactic meaning with perceived auditory stimuli through (repetitive) interaction with the physical / auditory environment (e.g., via reinforcement learning), a patient can learn to associate the information content of physical sound signals (e.g., the conceptual information encoded in speech, traffic noise, music, etc.) with the non-auditory perceptions elicited by the multi-channel neurostimulation signal. In order to do so, it is important that the neural representation of the physical sound signal that is generated by the multi channel neurostimulation signal is complex and variable enough that the relevant information content can be preserved during auditory processing and subsequent neurostimulation.

In some embodiments, the processing module may be configured to determine, preferably via an on-line auto-calibration procedure, a maximal number N of different perceivable perceptual channels that are specific for the individual and select the applied filter operation based on the determination, such that a fidelity of a representation of the received sound signal by the plurality of subcomponent signals is maximized for the determined number of perceptual channels. For example, after the maximal number N of usable perceptual channels is determined, independent component analysis or a similar filter operation can be applied to the received sound signal in order to subdivide it into N subcomponent signals in such a manner that the information content / entropy of the neural representation of the sound signal elicited by applying the subcomponent signals to the afferent neurons is maximized.

Such an on-line autocalibration of the neural interface device / neurostimulation signal may be based on observing the excitation behavior or neural activation function of afferent sensory nerve fibers that can be stimulated by a given neurostimulation means such as a SCS-electrode or DBS electrode connected to corresponding a neurostimulation module or device. This approach is based on the insight that there exist strong correlations between the highly non-linear bioelectric response of an active stimulated afferent sensory nerve fiber (e.g., ECAP) or plurality of such fibers and a corresponding artificial sensory perception / artificial sensation elicited in a sensory cortex area of the individual. This non-linear bioelectric response essentially serves as a fingerprint of the afferent sensory nerve fiber that can be measured and used for on line recalibration of neurostimulation signal parameters for direct neurostimulation of afferent sensory neurons targeting directly or indirectly (i.e., via multi-synaptic afferent pathways) sensory neurons in a specific target sensory cortex area. In this manner, long-term stability of highly specific, fine-grained and multi-dimensional information transfer to the brain can be ensured.

More specifically, the auditory neural interface device maybe configured (e.g., via a suitable firmware routine or software application) to carry out an on-line auto calibration procedure that may comprise the following steps: determining a plurality of independently operable stimulation electrodes or contacts of a neurostimulation interface operably connected to or integrated with the neural interface device; choosing a set of test signal parameters preferably associated with a set of N output qualities of a sound processor; generating, based on the chosen set of test signal parameters, a plurality of neurostimulation test signals configured to elicit a bioelectric response in one or more afferent sensory neurons of the individual; applying the generated plurality of neurostimulation test signals to the afferent sensory neurons via one or more of the determined plurality of stimulation electrodes or contacts of the neurostimulation interface; sensing, via the neurostimulation interface, one or more bioelectric responses of the one or more stimulated afferent sensory nerve fibers; and determining, based on the sensed bioelectric responses, a number N of different sensations that can independently be elicited in one or more cortex areas of the individual via neurostimulation of the one or more afferent sensory nerve fibers.

For instance, determining the N different (artificial) sensations may comprises comparing the sensed bioelectric responses with a set of reference responses stored in a memory module of the neural interface device or obtained via a wired or wireless communication interface of the neural interface device.

Further, determining, for one or more of the N determined sensations and based at least partially on the sensed bioelectric responses, a dynamic range of one or more neurostimulation signals that are configured to elicit the one or more determined sensations; and, optionally, subdividing the determined dynamic range into M, preferably equidistant, intervals.

In this manner, the symbol count (e.g., So=low intensity, Si= medium intensity, S2=high intensity) of each perceptual channel can be determined and optimized to maximize channel capacity.

The auto-calibration procedure may further comprise receiving, via a communication interface or user interface of the neural interface device, sensory feedback information from the individual associated with one or more of the sensations elicited by the plurality of neurostimulation test signals; and using the sensory feedback information for determining and / or characterizing the N different sensations and / or using the sensory feedback information for determining and / or subdividing the determined dynamic range of the one or more neurostimulation signals that are configured to elicit the one or more determined sensations.

In this manner, the fidelity of perceptual channel characterization can be improved, since the recorded bioelectric responses can be correlated with the (subjective) sensory feedback information provided by the patient / individual. For instance, the feedback information may comprise one or more indications of one or more of the following characteristics of the elicited sensations: a sensory modality, a location, an intensity and a frequency.

Determining the number N of usable perceptual channels (and the number M of symbols / differentiable perceptual levels / qualities per channel) in this manner allows the filters / signal transformations to be applied in a dynamic manner to the received sound signal, so that the fidelity of the neural representation is adapted (e.g., maximized) in real-time and in an on-line fashion in sync with the auto-calibration. For instance, if the relative distance between the stimulation electrode and the targeted afferent sensory neurons changes (e.g., due to a slow drift of a SCS-electrode or due to a movement of the patient), stimulation parameters can be adjusted such that the number of distinct perceptual channels and thereby sound signal representation fidelity stays as large as possible.

In some embodiments, the processing module may be further configured to apply the filter operation according to multiple selectable filter modes wherein the generation of the subcomponent signals and / or the mapping of the subcomponent signals to the multiple channels of the neurostimulation signal may be based on the selected filter mode. For instance, the filter mode may be user selectable (e.g., via a user interface) or automatically determined by the processing module. For instance, the processing module may be further configured to determine, preferably based on an analysis of the received sound signal, an auditory environment and / or a likely type of sound signal source associated with the received sound signal; and encode the received sound signal based on the determined auditory environment and / or type of sound signal source.

This allows the auditory neural interface device to maximize, for a given number of perceptual channels and a likely sound signal source or auditory environment the information content the neural representation of the received sound signal contains. For instance, certain frequency bands, phoneme subcomponent signals, musical instrument subcomponent signals or more abstract subcomponents signals may, for a whole class or subclass of received sound signals (e.g., speech, classical music), typically contain the majority of the information content of the received sound signal whereas other frequency bands / subcomponent signals mainly contain noise. Thus, by determining the auditory environment and / or the likely type of sound signal source, the processing module can select a filter operation best suited for an expected class sub class of sound signals. For instance, the processing module may select a set of Gabor filters forming a Gabor filter bank best suited for extracting the spectro-temporal information that is typical for speech signals whereas a band pass filter bank with adjustable gains and bandwidths may be better suited for perceiving an orchestra playing classical music.

Moreover, also the set of perceptual channels may be adjusted based on the determined auditory environment and / or a likely type of sound signal source. For instance, a set of distinct somato-sensory sensations (e.g., a subset of the dermatomes or peripheral nerve fields of the back side of the torso; see Fig. 12 below) might be best suited for perceiving classical music and experiencing the joy in doing so whereas a set of phosphenes, e.g., perceived in the periphery of the retina may be best suited for speech perception, e.g., via mapping a set of Gabor-filtered subcomponent signals to a set of phosphenes that can be distinguished by the individual as different vowels, consonants, phonemes etc.

In general, the multiple filter modes may comprise one or more of the following: a speech perception mode, a music perception mode, a closed space mode, an open space mode, a foreign language mode, a multi-source environment mode and a traffic mode. Additionally or alternatively the processing module may be configured to select the filter mode based on the determined auditory environment and / or likely type of sound signal source.

Further, for example to improve the fidelity of the neural sound signal representation, each filter mode may be associated with a plurality of filters being applied to the received sound signal to generate the plurality of subcomponent signals, wherein the filters may comprise bandpass filters, wavelet filters and / or Gabor filters or the like.

Alternatively or additionally, the filters maybe configured to filter out distinct characteristics of the received sound signal that are typical for an auditory environment and / or a likely type of sound signal source associated with the selected filter mode.

For example, different sets of filters / filter functions maybe designed for filtering out vowels, consonants, phonemes, musical instruments, cars, animals, etc. and stored in a memory device of the auditory neural interface device. When the processing module determines, for example, that the likely sound source is music, it might access the memory device and retrieve a set of filters designed for music perception. As discussed above, this pre-co nfigured set may then be further adapted based on the number N of available perceptual channels.

For instance, in some embodiments the number N of channels of the neurostimulation signal may be at least 2 (for representing simple sound characteristics), preferably at least 5 and more preferably at least 20 (for almost natural speech perception).

Additionally, the number of different perceivable perceptual qualities per perceptual channel (e.g., the number of different intensities that can be perceived per channel) may larger than 2 (e.g., loud vs. quiet), preferably larger than 3 (e.g., loud, medium, quiet) and more preferably larger than 10 (e.g., spanning 3odB of sound pressure level in steps of 3dB).

As mentioned in a slightly different context above, the processing module maybe configured to execute an autocalibration procedure, preferably interleaved with normal operation, to determine, for a given neurostimulation means or device of the individual, the number of differentiable perceptual channel and / or the number differentiable levels per channel. To assist the individual’s brain in perceiving sound using the auditory neural interface device of the present disclosure, e.g., assist with extracting the information content of speech, at least one of the multiple channels of the multi-channel neurostimulation signal may be an auxiliary channel that encodes at least one of the following characteristics of the received sound signal, a sound power or amplitude, a sound pitch, a sound timing, a direction of the sound signal source and a motional state of the sound signal source.

For instance, the processing module may be configured to determine the direction, distance and / or the velocity vector (i.e., direction and magnitude) of a (moving) sound signal source and encode this information in one or more of perceptual channels established by the multi-channel neurostimulation signal. For example, if two or more spatially separated sound sensors provide sound signals to the auditory neural interface device, arrival time difference, a phase difference and / or a sound signal amplitude difference may be used to determine the spatial direction of a sound signal source. If the type of sound signal source is known, also the total distance may be determined from an amplitude comparison with a reference sound signal. Finally, by determining a Doppler shift associated with sound signals received from a moving sound signal source also the magnitude and direction (i.e. approaching or receding) of the velocity vector can be determined and subsequently communicated to the individual.

For instance, in some embodiments, the sound signal may be received from at least two spatially separated sound sensors and the processor may be configured to determine a direction of the sound signal source based on information in the sound signal associated with the at least two spatially separated sound sensors, preferably based on a phase difference, a timing difference and / or an sound signal amplitude difference associated with the spatial separation of the at least two sound sensors. Alternatively or additionally, the channel that encodes the sound signal direction maybe configured to elicit somatosensory perceptions in adjacent areas of a body part, wherein each area corresponds to a different direction.

According to some embodiments, such an auxiliary channel may also encode context information associated with the received sound signal such as information about the sound signal source, a sound signal start or stop indication, one or more sign language symbols associated with the received sound signal, an indication of the emotional state of the sound signal source; and indication of the language used by the sound signal source.

For instance, if the disclosed auditory neural interface is operated in conjunction with DBS-equipment, the auxiliary channel may even use a different type of perception than the channels used for sound perception. For instance, in a dual-interface configuration a (multi-channel) SCS-electrode may be used by the auditory neural interface device to elicit a plurality of sound perceptions representing the received sound signal and a DBS-electrode may be used to elicit artificial sensations / perceptions of a different type / modality, such as vision or smell to implement the auxiliary channel. For example, different taste sensations may be used to encode the emotional state of a speaker (sour = angry, sweet = kind, bitter = joyfull, etc.) thereby providing essential context that supports speech perception and extraction of syntactic meaning from the sound signal representations perceived by the individual.

Further, the neurostimulation signal may be configured such that adjacent channels of the neurostimulation signal elicit somatosensory perceptions in adjacent areas of a body part of the individual or in adjacent body parts, preferably in a tonotopic manner. In this manner, patients that were used to normal cochlear sound processing, that also is based on a tonotopic organization of the sensory cells in the cochlear, will more easily adapt to the auditory interface device.

Further, the neurostimulation signal may be configured such that the areas of the body part are arranged in an essentially 2D array and, wherein one direction of the array encodes sound source direction, and the other direction is used for mapping the adjacent channels. More generally, as illustrated in Fig. 12 below different sound representation channels may be mapped to different dermatomes and / or sub-areas of a dermatome, e.g., via using a look-up table.

Some embodiments relate to an auditory neural interface system for sound perception by an individual, comprising the auditory neural interface device as discussed above and one or more sound sensors providing input signals to the receiver module of the auditory neural interface device and optionally, a neurostimulation device for stimulating afferent sensory neurons in the brain and / or the spinal cord of the individual.

Further embodiments relate to a computer program, comprising instructions for carrying out the following steps, when being executed by a neural interface device of an individual: receiving a sound signal, encoding the received sound signal as a multi channel neurostimulation signal, the neurostimulation signal being configured, to directly stimulate afferent sensory neurons of the central nervous system, CNS, of the individual and thereby to elicit, for each channel of the neurostimulation signal, one or more non-auditory, preferably somatosensory, sensations in a cortex area of the individual, wherein each channel of the neurostimulation signal is associated with a distinct type of non-auditory perception; and applying the neurostimulation signal to a neurostimulation means of the individual or transmitting the neurostimulation signal to a remote neurostimulation device. Such a computer program may comprise further instructions for operating the neural interface device in order to implement the functionalities as described above for the various embodiments of the neural interrace device.

More specifically, the various modules of the devices and systems disclosed herein can for instance be implemented in hardware, software or a combination thereof. For instance, the various modules of the devices and systems disclosed herein may be implemented via application specific hardware components such as application specific integrated circuits, ASICs, and / or field programmable gate arrays, FPGAs, and / or similar components and / or application specific software modules being executed on multi-purpose data and signal processing equipment such as CPUs, DSPs and / or systems on a chip (SOCs) or similar components or any combination thereof.

For instance, the various modules of the auditory neural interface device discussed above may be implemented on a multi-purpose data and signal processing device configured for executing application specific software modules and for communicating with various sensor devices and / or neurostimulation devices or systems via conventional wireless communication interfaces such as a NFC, a WIFI and / or a Bluetooth interface.

Alternatively, the various modules of the auditory neural interface device discussed above may also be part of an integrated neurostimulation apparatus, further comprising specialized electronic circuitry (e.g., neurostimulation signal generators, amplifiers etc.) for generating and applying the multi-channel neurostimulation signal to a neurostimulation interface of the individual (e.g. a multi-contact spinal cord stimulation electrode, a deep brain stimulation (DBS) electrode, etc.).

The neurostimulation signals generated by the auditory neural interface device described above may for instance also be transmitted to a neuronal stimulation device comprising a signal amplifier driving a multi-contact DBS electrode, spinal cord electrode, etc. that may already be implanted into a patient’s nervous system for a purpose different than providing a hearing aid. Alternatively, dedicated DBS-like electrodes or spinal cord stimulation electrodes may be implanted for the purpose of applying the neurostimulation signals generated by the auditory neural interface device via established and approved surgical procedures that were developed for implantation of conventional DBS electrodes or spinal cord stimulation electrodes etc. Further, as mentioned above the auditory neural interface device may also be integrated together with a neuronal stimulation device into a single device. 4. Short Description of the Figures

Various aspects of the present disclosure are described in more detail in the following by reference to the accompanying figures. These figures show: Figures relating to balance prosthesis

Fig. l a diagram illustrating an individual being equipped with a balance prosthesis device and system comprising said device according to an embodiment of the present invention;

Fig. 2 a diagram illustrating how a homologous balance indication can be encoded using a balance prosthesis device according to an embodiment of the present invention;

Fig. 3 a further diagram illustrating how a homologous balance indication can be encoded using a balance prosthesis device according to an embodiment of the present invention; Fig. 4 a further diagram illustrating how a homologous balance indication can be encoded using a balance prosthesis device according to an embodiment of the present invention;

Fig. 5 a diagram illustrating how a non-homologous balance support information can be encoded using a balance prosthesis device according to an embodiment of the present invention;

Fig. 6 a functional block circuit diagram illustrating a balance prosthesis device according to an embodiment of the present invention;

Fig. 7 a functional block circuit diagram illustrating a balance prosthesis device according to another embodiment of the present invention;

Fig. 8 a functional block circuit diagram illustrating a balance prosthesis device with integrated motion sensors according to another embodiment of the present invention; Fig. 9 a diagram illustrating how a balance prosthesis device according to an embodiment of the present invention can be used to mitigate motion sickness; Fig. lo a diagram illustrating the operation of a remote balance sensing device according to a further aspect of the present disclosure;

Fig. li a further diagram illustrating the operation of a remote balance sensing device according to a further aspect of the present disclosure. Figures relating to auditory interface

Fig. 12 a diagram illustrating an individual being equipped with an auditory neural interface device according to an embodiment of the present disclosure;

Fig. 13 a functional block circuit diagram illustrating an auditory neural interface device to an embodiment of the present disclosure;

Fig. 14 a diagram illustrating how an auditory neural interface device according to an embodiment of the present disclosure applies a filter operation to received sound signal generating three subcomponent signals that can be mapped to three different perceptual channels;

Fig. 15 a homunculus diagram illustrating how the three perceptual channels of Fig. 14 are implemented via three different peripheral perceptive nerve fields;

Fig. 16 a diagram illustrating how an auditory neural interface device according to an embodiment of the present disclosure operates in a multi-source outdoor auditory environment; Fig. 17 a diagram illustrating the use of auxiliary sematic channels to improve sound perception facilitated by an auditory neural interface device according to an embodiment of the present disclosure; Fig. 18 a diagram illustrating how an auditory neural interface device according to an embodiment of the present disclosure can be used to support sound perception for individuals with residual hearing capabilities;

Fig. 19 a diagram illustrating how an auditory neural interface device according to an embodiment of the present disclosure can be recalibrated in an automatic and on-line manner during operation as hearing aid.

5. Detailed Description of some exemplary embodiments

In the following, some exemplary embodiments of the present invention are described in more detail. While specific feature combinations are described in the following with respect to the exemplary embodiments of the present invention, it is to be understood that not all features of the discussed embodiments have to be present for realizing the invention. The disclosed embodiments may be modified by combining certain features of one embodiment with one or more features of another embodiment if technically feasible and functionally compatible. Specifically, the skilled person will understand that features, components and / or functional elements of one embodiment can be combined with technically compatible features, components and / or functional elements of any other embodiment of the present invention which is defined by the appended claims.

Exemplary embodiments relating to balance prosthesis In the following, some exemplary embodiments of the present invention are described in more detail, with reference to a balance prosthesis device that can be interfaced with neuronal stimulation electrodes such as spinal cord stimulation electrodes, DBS electrodes and / or peripheral axonal stimulation electrodes, e.g., via an intermediate neuronal stimulation device. However, the present invention can also be used with any other neuronal stimulation interface that is capable of stimulating afferent sensory axons of the central or peripheral nervous system targeting a sensory cortex area of an individual.

Figure l depicts an individual 100, e.g., a patient with impaired natural sense of balance that is equipped with a balance prosthesis system according to an embodiment of the present invention. The individual loo has already been implanted (e.g., for pain management) with a pair of spinal cord stimulation electrodes 101 that may have multiple independently controllable electric contacts. In other configurations, a neuronal stimulation electrode may also be implanted into the brain of the individual loo for the purpose of providing a neuromodulation therapy, e.g., for treating PD symptoms. Such a neurostimulation electrode may also be implanted for other purposes such as for the purpose of neuronal communication and /or treatment of other movement impairments and neurological diseases such as Alzheimer’s disease, epilepsy, depression, etc. Alternatively, the electrode 101 may also be implanted as a dedicated neurostimulation interface for the balance prosthesis device and system provided by the present invention.

The balance prosthesis system shown in Fig. l includes an array of motion sensors 103 such as acceleration sensors or gyroscopes distributed on the body of the individual 100 e.g., through a wearable enclosure. Alternatively or additionally, similar motion sensors can also be integrated within an implanted neurostimulation device 102 such as an implantable pulse generator (IPG) that drives the spinal cord stimulation electrodes

101. The motion sensors 103 may continuously relay positional information to a balance prothesis device 104 according to some embodiments of the present invention. In other embodiments, the balance prothesis device 104 or its functions may also be integrated with the implanted neurostimulation device 102. The processing circuitry of the balance prosthesis device 104 calculates various parameters including the level of body orientation with regard to a reference set of parameters based on the body’s upright position.

The balance prosthesis device 104 is wirelessly linked 105 to an implanted stimulator

102. The balance prosthesis device 104 can therefore functionally trigger the stimulator and adjust the stimulation parameters in terms of amplitude, frequency, pulse-width, burst duration and other parameters determined through a calibration process. The neurostimulator 102 is linked to a pair of implanted spinal cord leads 101 targeting somatosensory ganglions or afferent sensory nerve fibers within or adjacent to the spinal cord. The balance prosthesis device 104 receives as input a set of motion sensor signals and after processing, determines which contacts on the implanted spinal cord leads shall become active. The processor also determines a set of most suitable parameters which are most relevant to determine the location while encoding the desired type, locality and /or intensity of the artificial sensory perception that is to be elicited as discussed in detail in section 3 above.

Figure 2 illustrates exemplary scenarios in which the system is expected to activate / elicit artificial sensory perception on different body locations (e.g., left and right hand) to encode the deviations from the body’s upright equilibrium position 201 within the mediolateral plane. When the body is wayed to the left or right side 202, the ipsilateral perceptual channels 203 become active. In this way, the activation patterns of the perceptual channels provide a remedy for a totally lost or a compromised vestibular function. Patients can learn to associate the sensory cues into useful sensory input substituting, mimicking, supporting or even enhancing natural sense of balance and in response perform functionally relevant actions such as maintaining body equilibrium even when walking, running, cycling etc. In an alternative embodiment, the activation pattern of the perceptual channels could be adjusted in a reverse manner that is contra lateral to the tilt side. This mode would provide necessary cues resembling compensatory muscle reflexes which are naturally situated contralateral to direction of the tilt e.g. muscle contractions on the left side when body is leaning towards the right- hand side etc.

Figure 3 illustrates an embodiment where the intensity of the perceptual channel is used to encode the degree of body tilt. In the upright body position 301, both perceptual channels are silent, and the subject does not feel anything. In a moderate tilt position 302 of, e.g., 10 degrees to the left side, the ipsilateral perceptual channel 305 is activated with calibrated neurostimulation parameters eliciting a weak artificial sensation. In a more extreme case 303 the same perceptual channel is activated but stimulation parameters are adjusted so that they cause an intense somatosensations 306 thereby alerting the subject of an eminent fall prompting the user for an immediate corrective action.

To complement the balance prosthesis device and system provided by the present invention, additional information regarding body state or other environmental variables could be integrated using various sensors and other types of transducers. Typical balance systems must contain a minimum set of two independent orthogonal motion detection axes (i.e., back-front & left-right) to achieve a simple, upright balance feedback. In the present invention, the balance information in these additional planes could be achieved via a positional feedback system rendering information from both the coronal plane (as described in Fig.2 and Fig.3) as well as sagittal plane (as illustrated in Fig. 4.

In Figure 4 reference sign 401 corresponds to an upright body position, 402 to a leaned back body position and 403 to a leaned forward body position. Reference sign 404 indicates non-activated perceptual channels of the balance prosthesis system and 405 the activated perceptual channels providing a balance indication to the individual to substitute, mimic, support or enhance the natural sense of balance of the individual in a homologous manner.

In other preferred embodiments, further sensor information can be converted and integrated into the system including but not limited to a LIDAR sensor signal obtained from a LIDAR sensor such as a neck-worn personal phone, GPS- and map-position information, elevation sensor information from a wrist-worn smartwatch and others. This additional information is either utilized to optimize the position/balance- corrective cues in an intelligent manner (i.e., optimizing the resulting balance-gain outcome) or alternatively, is presented as additional one- or multi-dimensional balance-correcting axes. Consequently, a set of separate perceptual channels corresponding to artificial sensory perceptions felt in separate parts of body (or even within the same body region but with a separate quality of sensation) could be reserved to communicate specific balance support / auxiliary information to the individual.

The examples discussed above constitute essentially homologous embodiment of the present invention. For instance, the artificial sensations depicted in Fig. 2 emerge in left & right sides of the body which encode balance indications associated with left & right body tilts.

However, the present invention is based on a general sensory computer brain interface based on a patient specific communication library and is thus capable of relaying abstract information to the patient. Consequently, the patient can learn to relate sensory messages with virtually any kind of abstract balance-related information. In this context, the relayed messages or the communicated data are independent from body template, side, area, or type of sensation and thus constitute non-homologous embodiments of the present invention. In such non-homologous context, new relationships and correlations can be is achieved in a way that, for instance, artificial sensations relating to road inclination are translated as graded sensations emergent between left index finger and left thumb as depicted in Fig. 5. Figure 5 depicts an exemplary embodiment where crucial information required to maintain balance on a non-even surface could be transferred in a non-homologous manner via activating sensory messages on the index finger and thumb. Abstract information such as an upward inclination 503 of a walking surface could be associated by the subject, after training, to sensations on the thumb 505. In a similar analogy, a down-ward inclination 502 can be communicated via artificial sensations on index finger and a flat surface 501, can be encoded via absence of artificial sensations.

The illustrated embodiment in Fig. 5, can also be realized such that artificial sensations are graded in relation to a quantitative aspect of terrain characteristics. This can be achieved, for instance, by varying the intensity or repetition rate of the artificial sensations with respect to the angle of upward or downward inclination.

In such exemplary cases, the anatomical layout of target points of electrode-to-nervous- system interface locations (e.g. lumbar, thoracal and/or cervical spine) can be optimized by a submodule of the system to achieve an optimal level of communication across multiple independently-varied input channels. Homologous input (left-right tilt - left-right hand sensations) can here be combined with abstract sensations in thumb indicating upcoming incline on walking path 503. Figures 6 - 8 illustrate various possible embodiments of balance prosthesis devices provided by the present invention.

Figure 6 illustrates an exemplary balance prosthesis device according to an embodiment of the present invention. In this embodiment the balance prosthesis devices comprises an integrated neurostimulation module 6io (e.g. comprising a neuronal signal generator and an output amplifier) that is connected to a plurality of output signal leads that may be interfaced with a neurostimulation interface of the individual (e.g. a set of spinal cord stimulation electrodes or a DBS electrode). The balance prosthesis devices further comprises a communication antenna operably connected to a transceiver / sensor module 630, configured for wireless communication (e.g. via NFC, WIFI, Bluetooth or a similar wireless communication technology). The transceiver / sensor module 630 is configured, for example, to receive one or more sensor signals from one or more sensors (as discussed above), indicative of a balance or equilibrium state of the individual. The transceiver / sensor module 630 is operably connected to a data / signal processing module 640 configured to generate one or more neurostimulation signals and /or signal parameters (e.g. waveform, pulse shape, amplitude, frequency, burst count, burst duration etc.) for generating the one or more neurostimulation signals. For instance the processing module 640 may access a data storage module 650 configured to store a plurality of relations, specific for the individual, associating a plurality of neurostimulation signals (or parameters used for generating a plurality of neurostimulation signals) with a plurality of corresponding balance indications, such as a medium intensity tingling sensation in the right hand associated with a medium degree of body tilt in the left direction.

The generated neurostimulation signal and / or the signal parameters are input into the integrated neurostimulation module 610 that may be configured to process (e.g. modulate, switch, amplify, covert, rectify, multiplex, phase shift, etc.) the one or more neurostimulation signals generated by the processing module 640 or to generate the one or more neurostimulation signals based on the signal parameters provided by the processing module 640. The generated and processed neurostimulation signals are then output by the neurostimulation module 610 and can be applied to one or more electric contacts of a neurostimulation electrode (e.g. a DBS electrode or spinal cord stimulation electrode; not shown) via the output leads.

The balance prosthesis device may also comprise a rechargeable power source 660 that, for instance may be wirelessly charged via a wireless charging interface.

Figure 7 illustrates a further exemplary balance prosthesis device according to an embodiment of the present invention. In this embodiment, the balance prosthesis device does not comprise an integrated neurostimulation module (see Fig. 6 above). Instead and similar as in the discussion for Fig. 1 above the data / signal processing module 740 is connected to a wireless transmitter module 710 that is connected to a wireless transmit antenna 770. The processing module 740 maybe configured for generating one or more neurostimulation signals and /or signal parameters (e.g. waveform, pulse shape, amplitude, frequency, burst count, burst duration etc.) for generating the one or more neurostimulation signals. For instance the processing module 740 may access a data storage module 750 configured to store a plurality of relations, specific for the individual, associating a plurality of neurostimulation signals (or parameters used for generating a plurality of neurostimulation signals) with a plurality of corresponding balance indications.

The transmitter module 710 is configured for wireless communication (e.g. via NFC, Bluetooth, WIFI or a similar wireless communication technology) with a neurostimulation device of the individual (not shown; see Fig 1). The transmitter module 710 may be configured to transmit the generated neurostimulation signal and / or the generated signal parameters to the neurostimulation device of the individual such as an IPG (see Fig. 1) that may be configured to process (e.g. modulate, switch, amplify, covert, rectify, multiplex, phase shift, etc.) the one or more neurostimulation signals received from the transmitter module 710 or to generate the one or more neurostimulation signals based on the signal parameters received from the transmitter module 710. The balance prosthesis device may further comprise a wired receiver / sensor module 730 that is configured to receive / obtain one or more sensor signals from one or more sensors (as discussed above), indicative of equilibrium or balance state of the individual (e.g. gyroscope and accelerometer signals allowing the proce3ssing module to estimate the current or future body position of the individual with respect to a reference position). In the embodiment of Fig. 7 the sensor signals are not received wirelessly but are obtained via sensor signal leads 720. Naturally, wireless reception is also possible.

The neurostimulation device of the individual is configured to output and apply the generated and processed neurostimulation signals to one or more electric contacts of a neurostimulation electrode (e.g. a spinal cord stimulation electrode; not shown) to elicit the desired artificial sensory perception in the desired sensory cortex area. The balance prosthesis device 710 may also comprise a power source 760 that, for instance may be a removable battery.

Figure 8 illustrates a further exemplary balance prosthesis device according to an embodiment of the present invention. In this embodiment, the balance prosthesis device comprises an integrated motions sensor, such as a 3-axis acceleration sensor 862 and a 3-axis gyroscope 864. In this case, balance indications can be determined and communicated to the individual (e.g. via two spinal cord stimulation leads) even without obtaining information from external sensor devices such as the wearable sensors discussed for Fig. 1.

Figure 9 illustrates a scenario where a balance prosthesis device according to an embodiment of the present invention can be used to mitigate the effects of motion sickness (e.g. terrestrial motion sickness, space motion sickness and / or virtual reality motion sickness) by reinforcing the natural sense of balance of an individual 900 via an additional balance indication encoded via artificial sensory perceptions provided by the balance prosthesis. For instance, a person 900 on-board a ship 910 may experience a perceptual conflict between its visual and vestibular system, e.g. if the person 900 cannot see the horizon / waterline 920 and the ship 910 performs a combined roll, pitch and yaw movement. In such situations, that may also occur in virtual reality environments the balance prosthesis device may transmit a balance indication 930 that helps to reinforce the correct balance perception 940 of the individual. Fig. 10 illustrates the reciprocal situation where the person 1000 is not on-board the ship 1010 but controls the movement of the ship 1010 via a remote control terminal 1050 in this situation, sensor equipment on-board the ship transmit sensor signals indicative of the movement / balance state 1030 of the ship 1010 to a remote balance sensing device of the individual. The remote balance sensing device then provides a remote balance indication for the moving ship to the individual that is derived at least in part from the obtained sensor signals.

Fig. 11 illustrates another application scenario where a spinal cord stimulation (see Fig. 1 above) based remote balance sensing device supports a pilot 1100 in remotely piloting an unmanned aerial vehicle, e.g., via a conventional remote control or a brain computer interface remote control device.

Exemplary embodiments relating to auditory interface In the following, some exemplary embodiments of the present disclosure are described in more detail, with reference to an auditory neural interface device that can be interfaced with neuronal stimulation electrodes such as spinal cord stimulation electrodes, DBS electrodes, etc., via an intermediate neuronal stimulation device. However, the present disclosure can also be used with any other neuronal stimulation interface that is capable of stimulating afferent sensory axons of the CNS targeting one or more sensory cortex areas of an individual.

While specific feature combinations are described in the following with respect to the exemplary embodiments of the present disclosure, it is to be understood that not all features of the discussed embodiments have to be present for realizing the technical advantages provided by the devices, systems, methods and computer programs provided by the present disclosure. The disclosed embodiments may be modified by combining certain features of one embodiment with one or more features of another embodiment if technically feasible and functionally compatible. Specifically, the skilled person will understand that features, steps, components and / or functional elements of one embodiment can be combined with technically compatible features, steps, components and / or functional elements of any other embodiment of the present invention which is defined by the appended claims.

Moreover, the various modules of the devices and systems disclosed herein can for instance be implemented in hardware, software, or a combination thereof. For instance, the various modules of the devices and systems disclosed herein may be implemented via application specific hardware components such as application specific integrated circuits, ASICs, and / or field programmable gate arrays, FPGAs, and / or similar components and / or application specific software modules being executed on multi-purpose data and signal processing equipment such as CPUs, DSPs and / or systems on a chip (SOCs) or similar components or any combination thereof.

For instance, the various modules of the auditory neural interface device discussed herein above may be implemented on a multi-purpose data and signal processing device configured for executing application specific software modules and for communicating with various sensor devices and / or neurostimulation devices or systems via conventional wireless communication interfaces such as a Near Field Communication (NFC), a WIFI and / or a Bluetooth interface. Alternatively, the various modules of the auditory neural interface device provided by the present disclosure may also be part of an integrated neurostimulation apparatus, further comprising specialized electronic circuitry (e.g., neurostimulation signal generators, amplifiers etc.) for generating and applying the determined neurostimulation signals to a neurostimulation interface of the individual (e.g. a multi- contact electrode, a spinal cord stimulation electrode, a DBS electrode etc.).

Figure 12 illustrates a person / individual 100X that is equipped with an auditory neural interface device as described in section 3 above and illustrated in an exemplary manner in Fig. 13 below. In the illustrated embodiment, the auditory neural interface device is implemented via direct neurostimulation of afferent sensory nerve fibers in the spinal cord via one or more multi-contact electrodes 104X driven by an implantable pulse generator (IPG) 102X that may be operatively / communicatively connected to or integrated with an auditory neural interface device as disclosed herein. For establishing multiple perceptual communication channel to the brain of the individual 100X the auditory neural interface device may be calibrated such that neurostimulation signals generated by the auditory neural interface device and applied via the IPG 102X and the multi-contact electrode 104X elicit one or more action potentials 106X in one or more afferent sensory nerve fibers of the spinal cord 106X targeting (e.g. via multi-synaptic afferent sensory pathways) one or more sensory cortex areas 110 of the individual 100 where the one or more action potentials 106X generate (directly or indirectly) artificial sensory perceptions that can be used to represent a received sound signal (se Fig. 14 below) to be perceived by the brain of the individual 100X. As discussed in detail in US 2020/0269049, fully incorporated herein by reference, artificial sensory perceptions that are elicited in a sensory cortex area (e.g. a sensory cortex area processing touch sensations on the left or right hand) can also be associated with any kind of abstract information that is intelligible (i.e. consciously or subconsciously) by the individual 100X.

In operation, the auditory neural interface device receives sound signals recorded via one or more sound sensors / microphones 108X that may be worn by the individual 100X, be integrated with the auditory neural interface device and / or be provided by a general purpose data and signal processing device such as a smart phone. For instance, some or all functionalities of the auditory neural interface devices discussed in detail in section 3 above, maybe implemented via application specific software modules executed by such a general-purpose data and signal processing device which in turn maybe interfaced (e.g., wirelessly) with the IPG 102X or a similar neurostimulation device operating in conjunction to implement a sensory substitution-based hearing aid.

For the embodiment illustrated in Fig. 12 the perceptual channels correspond to different dermatomes H4Xa - H4Xg innervated by spinal nerve fibers branching of the spinal cord at location H2Xa to H2Xg. In this general example different contacts of the stimulation electrode may be used to stimulate regions of the spinal cord typically relaying sensory information from a given dermatome (e.g., a dermatome H4Xa located on the front torso of the person).

In other embodiments, complex, multi-contact neural stimulation signals may also be used to selectively stimulate single peripheral nerve fields within a given dermatome or combinations of dermatomes and / or peripheral nerve fields.

Figure 13 shows an exemplary auditory neural interface device 200X according to an embodiment of the present disclosure. In this embodiment, the CBI device comprises an integrated neurostimulation and sensing module 230X (e.g. comprising a neuronal signal generator and an output amplifier as well as a sensing amplifier and an analog to digital converted and similar circuitry) that is connected to a plurality of output signal leads 235X and a plurality of separate or identical sensing signal leads 235X that may be interfaced with a neurostimulation interface of the individual (e.g. a multi-contact spinal cord stimulation electrode such as the electrode 104X shown in Fig. 12). The exemplary auditory neural interface device may further comprise a communication antenna 260X operably connected to a communication interface module 210X, configured for wireless communication (e.g., via NFC, Bluetooth, or a similar wireless communication technology).

The communication interface module 210X maybe configured, for example, to receive one or more sound signals from one or more sound sensors (not shown; e.g., a set of microphones worn by the individual) and / or control information from a control device such as a remote control or a smart phone. The communication interface module 210X is operably connected to a data / signal processing module 220X configured to generate one or more neurostimulation signals and /or signal parameters (e.g., waveform, pulse shape, amplitude, frequency, burst count, burst duration etc.) for generating the one or more neurostimulation signals. For instance, the processing module 220X may access a data storage module 240X configured to store a plurality of sound signal filters for the various filter modes as described in section 3. above and / or relations, specific for the individual, associating a plurality of neurostimulation signals (or parameters used for generating a plurality of neurostimulation signals) with a plurality of corresponding pieces of auxiliary information to be communicated to the individual, e.g., for establishing a perceptual channel used to indicate the sound source direction, the motional state of the sound signal source and / or context information such as the emotional state of a speaker.

The generated neurostimulation signals and / or the signal parameters are input into the integrated neurostimulation and sensing module 230X that may be configured to process (e.g., modulate, switch, amplify, covert, rectify, multiplex, phase shift, etc.) the one or more (multi-channel) neurostimulation signals generated by the processing module 220X or to generate the one or more neurostimulation signals based on the signal parameters provided by the processing module 220X.

The generated and processed neurostimulation signals are then output by the neurostimulation and sensing module 230X and can be applied to one or more electric contacts of a neurostimulation electrode (e.g., a DBS electrode or spinal cord stimulation electrode as shown in Fig. 12) via output leads 235X. The auditory neural interface device of Fig. 13 may also comprise a rechargeable power source 250X that, for instance may be wirelessly charged via a wireless charging interface 265X.

As discussed above, the data / signal processing module 220X may be further configured to, e.g. in conjunction with the data storage module 240X and the neurostimulation and sensing module 230X, to execute an on-line autocalibration method as discussed in section 3 above. Further, the auditory neural interface device may also comprise a transmitter module (e.g., the communication interface 210X) as an alternative to the neurostimulation and sensing module 230X to communicate with a remote neurostimulation device. Figures 14 and 15 illustrate a general example how some embodiments of the present invention can be used to establish a three-channel, non-auditory hearing aid for a patient. Specifically, the processing module filters a received sound signal (see waveform in top trace of Fig. 14) via a three-channel filter bank (see spectrogram in lower trace of Fig. 14).

The output signal of each bandpass filter of the filter bank (i.e., a subcomponent signal as discussed in detail in section 3 above) is then separately sampled and used to generate a three-channel neurostimulation signal. As shown in the homunculus diagram of Fig. 15 each of the subcomponent signals is configured to elicit an artificial sensation perceived by the individual in the lips (channel 1; high frequency components of the received sound signal), in the right hand (channel 2, medium frequency components of the received sound signal) and the left hand (channel 3, low frequency components of the received sound signal). As discussed in detail in section 3 above, instead of a filter bank, other filter operations such as wavelet or Gabor filters may also be used to subdivide a received sound signal into subcomponent signals that are then mapped to different perceptual channels.

In some embodiments, the disclosed auditory neural interface device may be calibrated and N perceptual channels are identified as discussed in section 3 above. Each different channel could then be mapped to a different frequency band. The number N (and the differentiated levels within each channel) will define the maximum resolution or bandwidth of the perceptual / transmission matrix, which relate to a specific characteristic of the implant type and implant location with respect to the neural tissue defined per individual patient. The decomposition algorithm / filter operation of sound signals can be customized, so that e.g., an ICA is conducted which solves for a target number of components equals N. This decomposition matrix may be fixed for the patient and subsequently a completely customized translation of the sound signal occurs that is optimized for the respective patient. In some embodiments, here, pre calculated ICA decomposition matrices may be applied which are based on e.g. language-specific audio file training sets. Figure 16 illustrates how some embodiments of the disclosed auditory neural interface device 200X can be equipped with source detection / discrimination modules (soft- and/or hardware based) that can enable the auditory neural interface device 200X to determine which part of a complex auditory environment should be perceived by the individual (not shown) with high fidelity and / or priority (e.g., the sound of an approaching car), which sounds with low fidelity / priority (e.g., a person 520X directly talking to the individual) and which sounds are to be filtered out completely (e.g., background noise generated by a remote group of people 530X talking).

As discussed in section 14 above, the filter modes and / or filter function stored in the memory module 240X of the auditory neural interface devices 200X, can, for example, automatically be selected by the processing module, after a determination that the individual is located in an outdoor environment with likelihood of motorized traffic. A traffic filter mode may for example use a specialized spatio-temporal filter operation to filter out sounds typically generated by dangerous objects (e.g., cars) with high fidelity and select one of the perceptual channels to transmit this subcomponent signal with high priority and / or signal strength.

Figure 17 illustrates an embodiment of the disclosed auditory neural interface devices that is configured to transmit auxiliary information such as a sound signal duration or context information such as the emotional state of a speaker via a separate DBS electrode 610X, while at the same time an SCS-electrode 104X (as illustrated in detail in Fig. 12 above) is operated to transmit the multi-channel neurostimulation signal used for sound signal representation.

As discussed above, the processing module of the auditory neural interface device is configured to map, based on a selected filter mode and / or operation different types of sound signal sources (music, speech, alarms) to different perceptual channel addressable via the SCS-electrode. In addition to the source discrimination and priorization module discussed for Fig. 16 above, the processor may also comprise or execute a semantics and /or context detection module that allows the auditory neural interface device to determine relevant context information, such as the language used by a sound source.

For instance, an auxiliary taste channel may be used to signal to the individual whether a sound signal source uses a foreign language (sweet) or the native language of the individual. In another example the emotional state may be encoded as artificial taste sensations, e.g. (aggressive = bitter; empathic = sweet). For instance, modern speech processing software (e.g., trained multi-layered neural networks) may be used automatically extract meaning and / or context of received speech signals.

Figure 18 illustrates that some embodiments of the present disclosure can also be used to supplement or support persons having residual hearing providing even further benefits over conventional Cochlear implants. Fig. 18 also illustrates, that in some embodiments, the auditory neural interface device may also comprise a hard- and / or software implemented sign language encoder module that can support sound perception by the individual by operating in a sign-language assistance mode. For example, all or part of the typical sign-language hand poses can be translated into a combination of individually detectable perceptual channels and be used to support sound perception by the individual. Figure 19 illustrates the auto-recalibration procedure that is discussed in detail in section 3 above. For instance, in some embodiments, while the disclosed auditory neural interface device receives sound signals and processes (e.g., filters, maps, etc.) them as discussed above the neuronal sensing module 230X (see Fig. 13 above) constantly records the bioelectric responses (e.g., ECAP or somatosensory EESP, or extracellularly measured action potentials or similar bioelectric response) of the stimulated nerves / nerve fibers / neurons and derives an activation function that can be compared to a reference activation function 810X (as disclosed in US patent application 17/224,953, incorporated herein in it’s entirety). Alternatively or preferably additionally, sensory feedback 820X from the patient can be used to determine whether the fidelity of the sound signal representation is still optimal or may be improved by readjusting the signal parameters and / or the filter operation used to generate the multi-channel neurostimulation signal. In this manner, the performance of the non- auditory hearing aid implemented by the auditory neural interface device can be maintained as good as possible even in normally behaving (e.g., moving) patients.

Claims

1. Balance prosthesis device for an individual, comprising: a sensor module configured to obtain at least one sensor signal indicative of a balance or equilibrium state of the individual; a processing module operably connected to the receiver module and configured to determine at least one neurostimulation signal based at least in part on the obtained sensor signal; and a transmitter module operably connected to the processing module and configured to transmit the determined neurostimulation signal to a neurostimulation device of the individual; or a neurostimulation module operably connected to the processing module; wherein the neurostimulation signal is configured to elicit an artificial sensation in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not directly associated with vestibulocortical pathways of the individual; and wherein the elicited artificial sensation provides a balance indication to the individual that is derived at least in part from the obtained sensor signal in order to support, mimic, substitute or enhance the natural sense of balance of the individual.

2. Balance prosthesis device of claim l, wherein the sensor module obtains at least one motion sensor signal via a wired or wireless interface, such as an accelerometer or a gyroscope signal; and / or wherein the sensor module comprises a motion sensor such as an accelerometer or a gyroscope; and / or wherein the sensor module obtains, via the wired or wireless interface, an auxiliary sensor signal originating from an auxiliary sensor device such as a camera, a LIDAR sensor, a GPS system, a pressure sensor or an elevation sensor.

3. Balance prosthesis device of claim l or 2, wherein the processing module determines, based at least in part on the obtained sensor signal, an estimate of the current body position of the individual with respect to a reference body position and / or an estimate of a future body position with respect to the reference body position.

4. Balance prosthesis device of claim 3, wherein the body position of the individual is characterized by one or more of the following parameters:

- a body tilt of the individual in the coronal and / or the sagittal plane;

- a rate of change of the body tilt of the individual in the coronal and / or the sagittal plane;

- a deviation of the center of gravity of the body of the individual from a reference position or range for the center of gravity; and

- a rate of change of the deviation of the center of gravity from the reference position or range.

5. Balance prosthesis device of any of the preceding claims, wherein a perceived laterality or location of the elicited artificial sensation indicates a direction of a body tilt of the individual or a direction of a compensatory movement to decrease the body tilt; and / or wherein a perceived intensity of the elicited artificial sensation encodes an angle or a degree of a body tilt of the individual relative to a reference position or range or an angle or degree of a compensatory movement to decrease the body tilt.

6. Balance prosthesis device of any of the preceding claims, wherein a perceived repetition rate of the elicited artificial sensation encodes a terrain characteristic such as an inclination angle of a walking surface or a remaining distance to an obstacle.

7. Balance prosthesis device of any of the preceding claims, wherein a secondary sensory quality of the elicited artificial sensation such as the texture of a somatosensation, the color of a visual sensation or the tone and timbre of an auditory sensation encodes body balance support information such as an inclination of a walking surface or a remaining distance to an obstacle.

8. Balance prosthesis device of any of the preceding claims, wherein the processing module is configured: to detect, based at least in part on the obtained motion sensor signal and / or the auxiliary sensor signal, preferably by using a trained machine learning system, whether the body of the individual is at risk to fall; and in response to said detection: to generate a neurostimulation warning signal that is configured to elicit an artificial sensation in a specific sensory cortex area providing a falling warning to the individual.

9. Balance prosthesis device of any of the preceding claims, wherein the neurostimulation signal is synchronized with a walking pace of the individual to provide a continuous body tilt correction indication improving the gait stability of the individual while walking.

10. Balance prosthesis system, comprising the balance prosthesis device of any of the preceding claims 1 - 9; and one or more implanted or wearable motion sensors providing input signals to the sensor module of the balance prosthesis device.

11. Balance prosthesis system of claim 10, further comprising one or more of the following sensor devices providing further input signals to the sensor module of the balance prosthesis device: a camera device a LIDAR sensor device a GPS system; a pressure sensor measuring the contact pressure between a foot of the individual and a walking surface; at least two pressure sensors measuring the difference in contact pressure between two points on a walking surface to determine an inclination of the walking surface.

12. Balance prosthesis system of claim 10 or 11, further comprising a spinal cord stimulation device comprising a set of implanted spinal cord leads targeting somatosensory ganglions or afferent sensory nerve fibers within or adjacent to the spinal cord.

13. Method for providing a balance indication to an individual, comprising the following steps: obtaining at least one motion sensor signal indicative of a balance or equilibrium state of the individual; determining at least one neurostimulation signal based at least in part on the obtained sensor signal; and transmitting the determined neurostimulation signal to a neurostimulation device or module of the individual; wherein the neurostimulation signal is configured to elicit an artificial sensation in a specific sensory cortex area of the individual via directly stimulating afferent sensory axons of the central or peripheral nervous system of the individual targeting sensory neurons of the sensory cortex area not associated with vestibulocortical pathways of the individual; and wherein the elicited artificial sensation provides a balance indication to the individual that is derived at least in part from the obtained motion sensor signal in order to support, mimic, substitute or enhance the natural sense of balance of the individual.

14. Computer program, comprising instructions for carrying out the method of claim 13, when being executed by signal processing and transceiver modules of a signal and data processing device, a neuronal stimulation device or system.

15. Computer program of claim 14, comprising further instructions for implementing the operation of the balance prosthesis device of the preceding claims 2 - 9, when being executed by the signal processing and transceiver modules of a signal and data processing device, a neuronal stimulation device or system.

16. Auditory neural interface device for sound perception by an individual, comprising: a receiver module configured to receive sound signals; a processing module operably connected to the receiver module and configured to encode a received sound signal as a multi-channel neurostimulation signal; the neurostimulation signal being configured, to directly stimulate afferent sensory neurons of the central nervous system, CNS, of the individual and thereby to elicit, for each channel of the neurostimulation signal, one or more non- auditory, preferably somatosensory, perceptions in a cortex area of the individual, wherein each channel of the neurostimulation signal is associated with a different non-auditory perception; and a neurostimulation module operably connected to the processing module and configured to apply the multi-channel neurostimulation signal to a neurostimulation means of the individual; or a transmitter module configured to transmit the multi-channel neurostimulation signal to a remote neurostimulation device.

17. Auditory neural interface device according to claim 16, wherein at least one of the multiple channels is an auxiliary channel that encodes at least one of the following characteristics of the received sound signal, a sound power or amplitude, a sound pitch, a sound timing, a direction of the sound signal source and a motional state of the sound signal source; and / or that encodes context information associated with the received sound signal such as information about the sound signal source, a sound signal start or stop indication, one or more sign language symbols associated with the received sound signal, an indication of the emotional state of the sound signal source; and indication of the language used by the sound signal source; and / or wherein the auxiliary channel uses a different type of perception than the channels used for sound perception.

18. Auditory neural interface device according to any of the preceding claims 16 or 17, wherein encoding by the processing module comprises: applying a filter operation to the received sound signal to generate a plurality of subcomponent signals of the sound signal; and mapping each subcomponent signal to a different channel of the multi-channel neurostimulation signal.

19. Auditory neural interface device according to claim 17 or 18, wherein the processing module is configured to determine, preferably via an on-line auto-calibration procedure, a maximal number of different perceivable perceptual channels that are specific for the individual; and select the applied filter operation based on the determination, such that a fidelity of a representation of the received sound signal by the plurality of subcomponent signals is maximized for the determined number of channels.

20. Auditory neural interface device according to claim 18 or 19, wherein the filter operation involves performing spectral analysis, wavelet analysis; principal component analysis, independent component analysis, using a filter bank, and/ or a combination thereof.

21. Auditory neural interface device according to any of the claim 18 to 20, wherein the processing module is further configured to apply the filter operation according to multiple selectable filter modes, and wherein the generation of the subcomponent signals and / or the mapping of the subcomponent signals to the multiple channels of the neurostimulation signal is based on the selected filter mode.

22. Auditory neural interface device according to any of the proceeding claims 16 to 21, wherein the processing module is further configured to: determine, preferably based on an analysis of the received sound signal, an auditory environment and / or a likely type of sound signal source associated with the received sound signal; and encode the received sound signal based on the determined auditory environment and / or type of sound signal source.

23. Auditory neural interface device according to claim 21 or 22, wherein the multiple filter modes comprise one or more of the following: a speech perception mode, a music perception mode, a closed space mode, an open space mode, a foreign language mode, a multi-source environment mode and a traffic mode; and / or wherein the processing module is configured to select the filter mode based on the determined auditory environment and / or likely type of sound signal source.

24. Auditory neural interface device according to any of the claims 21 to 23, wherein each filter mode is associated with a plurality of filters being applied to the received sound signal to generate the plurality of subcomponent signals; and wherein the filters comprise bandpass filters, wavelet filters and / or Gabor filters; and / or wherein the filters are configured to filter out distinct characteristics of the received sound signal that are typical for an auditory environment and / or a likely type of sound signal source associated with the selected filter mode.

25. Auditory neural interface device according to any of the preceding claims, wherein the number of channels of the neurostimulation signal is at least 2, preferably at least 5 and more preferably at least 20; and / or wherein the number of different perceivable perceptual qualities per perceptual channel is larger than 2, preferably larger than 3 and more preferably larger than 10; and / or wherein the processing module is configured to execute a calibration procedure, preferably interleaved with normal operation, to determine, for a given neurostimulation means or device of the individual, the number of differentiable perceptual channel and / or the number differentiable perception per channel.

26. Auditoiy neural interface device according to any of the preceding claims 16 to

25, wherein the neurostimulation signal is configured such that adjacent channels of the neurostimulation signal elicit somatosensory perceptions in adjacent areas of a body part of the individual or in adjacent body parts, preferably in a tonotopic manner.

27. Auditory neural interface device according to any of the preceding claims 17 to

26, wherein the sound signal is received from at least two spatially separated sound sensors and the processor is configured to determine a direction of a sound signal source based on information in the sound signal associated with the at least two spatially separated sound sensors, preferably based on a phase difference, a timing difference and / or a sound signal amplitude difference associated with the spatial separation of the at least two sound sensors; and / or wherein the channel that encodes the sound signal direction is configured to elicit somatosensory perceptions in adjacent areas of a body part, wherein each area corresponds to a different direction.

28. Auditory neural interface system for sound perception by an individual, comprising: the auditory neural interface device of any one of the preceding claims 16 to 27; and one or more sound sensors providing input signals to the receiver module of the auditory neural interface device; and optionally, a neurostimulation device for stimulating afferent sensoiy neurons in the brain and / or the spinal cord of the individual.

29. Computer program, comprising instructions for carrying out the following steps, when being executed by a neural interface device of an individual: receiving a sound signal; encoding the received sound signal as a multi-channel neurostimulation signal, the neurostimulation signal being configured, to directly stimulate afferent sensory neurons of the central nervous system, CNS, of the individual and thereby to elicit, for each channel of the neurostimulation signal, one or more non-auditory, preferably somatosensoiy, perceptions in a cortex area of the individual, wherein each channel of the neurostimulation signal is associated with a distinct type of non-auditory perception; and applying the neurostimulation signal to a neurostimulation means of the individual; or transmitting the neurostimulation signal to a remote neurostimulation device.

30. Computer program of claim 29, further comprising instructions for operating the neural interface device to implement the functionalities defined in the preceding claims 17 - 28.