GB2585364A

GB2585364A - Method for a device to be worn within or near the ear, to monitor health, physiology and status of the ear and the wearer, and to provide authentication

Info

Publication number: GB2585364A
Application number: GB1908601.6A
Authority: GB
Inventors: Roy Gompertz Nicholas
Original assignee: Individual
Current assignee: Individual
Priority date: 2019-06-14
Filing date: 2019-06-14
Publication date: 2021-01-13
Also published as: GB201908601D0

Abstract

A method and device 6 for monitoring any combination of: health, disease, biological and physiological qualities, processes and changes; and of user identification and user authentication, comprising a sensor 1, locatable in, or near to, the ear canal 2, capable of: detecting a plurality of properties of the ear drum 3 and surrounding structures; and communicating this as an input to a processor, for generating an output to a connected device; and capable of being located to sense these properties during the user’s daily activities. Sensor and processor of the device detect and interpret a plurality of data, including data on pulse, oxygen saturation, temperature and structural changes due to disease processes. The method is capable of image recognition to authenticate the identity of the wearer of the earphone device containing the sensor.

Description

Title: Method for a device to be worn within or near the ear, to monitor health, physiology and status of the ear and the wearer, and to provide authentication of the identity of the wearer.

Background:

There is an increasing demand and need for continuous health and welfare monitoring of individuals, including for exercise performance monitoring and general wellbeing in both commercial and leisure uses, and for health monitoring for both formal medical management and personal health interest. An ideal monitoring platform is one that is unobtrusive, comfortable and can be incorporated within items that the user may already be using during day to day life, and not restrict activity, and be able to monitor multiple physiological parameters continuously. The current invention addresses these issues in personal monitoring.

There is also need for easily accessible, but secure access to computer based devices such as smart phones. Biometric user authentication is being used in the place of passwords to unlock such devices. The advent of smart earphones allows a further method of user authentication if the earphone or device can recognise the individual features of the ear, and use this to authenticate the individual and unlock the security features either on the smart earphone itself or a connected device. This can therefore unlock a wirelessly connected device without needing to handle the device. Biometric authentication can also be used in other areas such as passport control at borders. The current invention provides a method of biometric authentication.

The wide use of devices such as earphones and hearing aids, that are dependent on transmission of sound efficiently to the ear drum, raises the problem of transmission of the sound effectively in the presence of obstruction of the ear canal, for example by wax or other debris. This would currently normally need the assessment by a health care worker. The current invention provides a method for monitoring of the ear canal to ensure that any obstructions to sound transmission can be identified early to allow prevention of reduction of acoustic performance of the device. It also allows assessment of the response of the ear drum and associated structures to loud noise exposure, to ensure that adequate ear protection is provided during occupational exposure to loud noises, and to assess sound transmission from earphones to monitor response to adjust earphones to provide peak acoustic performance.

The prevalence of ear-associated technological devices including ear-phones, smart earphones and hearing aids is rapidly increasing They represent a user acceptable method of mounting a device on the user, and allow easy stable access to the ear canal, ear drum and associated structures.

The ear drum represents a portal into the health and status of the ear itself and also the health and physiological functions of each person. It is also different in shape, colour, contour and movement in each individual.

The appearance and qualities of the ear canal, ear drum and associated structures can therefore give useful information on health and physiological processes. It can also be used to identify each individual.

The combination of the ability to mount a detector of the ear drum and associated structures, in a comfortable and user acceptable method, combined with the portal that these structures provide on health and physiology, allows the use of imagers within ear associated structures to be an ideal way to monitor health and wellbeing. Using imaging sensors that record images (including infrared and visible light) provides images and measurements that are useful to monitor health. These imaging sensors are widely available and small enough to be incorporated within existing ear related devices such as earphones, smart earphones, and hearing aids. The method of this invention is of incorporating imagers within these devices, so that these devices become physiological and health monitors in addition to their function as devices used in normal day to day life.

As these devices are becoming increasingly connected to smart phones by wireless technology the image obtained of the eardrum and associated structures will be used to recognise and authenticate the user of the device. This will allow comparison of the physiological and health data with previous data from the same user, and will also allow the ear device to confirm the identity of the user to unlock security features on the attach technology, such as a smart phone. The authentication benefit of this innovation may also be used for any other biometric identity processes.

Imaging sensors may detect movement, such as ear drum movements with change in pressure, including breathing, to detect respiratory rate, and larger scale movements of the ear drum and malleus associated with eustachian tube function, swallowing, yawning and eye closing, and changes in colour, associated with blood flow, pulse rate and oxygen saturation, and changes of the diameter of the blood vessels, associated with pulse rate and blood flow/pressure, and the change in infrared images associated with inflammation and body temperature. It will also detect structural change and pathology, including changes due to infection, perforation, Eustachian tube dysfunction and debris, discharge and wax.

Smart ear phones already exist with the capability of measuring some biological variables including oxygen saturations and heart rate. The current innovation is the first to use an imager associated within an ear associated device to record, which images and measures these variables with such a range of potential measurements.

The current invention is the first to use an imager in an ear worn device to monitor physiological changes and health continuously within a device designed to be used in normal daily activities.

Other inventions have proposed methods to authenticate the identity of users of earphones by detecting response to ultrasound waves projected into the ear. However there are concerns regarding the use of ultrasound near the ear. This is the first authentication method based on visible imaging.

The present invention is for a method of monitoring health and physiological processes, and authenticating the identity of an individual by using an imager incorporated in a device worn similarly to a hearing aid or earphone.

Summary:

The current invention is a method of detecting health and physiological data using an imager of the eardrum and associated structures, and using this imager to authenticate the identity of the individual. Imagers of ear drum and associated structures have not previously been used, within earphone type structures used in daily life, to monitor health and authenticate identity.

The method for detecting changes of the ear drum complex (comprising the ear drum, malleus, and adjacent ear canal) in this invention is the use of an imaging sensor wholly or partially sited in or adjacent to the ear canal, directed towards the ear drum complex. This sensor may be located in a structure worn like a hearing air, ear phone or earplug (subsequently jointly referred to as earphone). This sensor detects changes including changes in the position, movement, shape, structure or colour of the ear drum complex. These changes occur during normal physiological processes and vary according to health, fitness, activity and illness. The blood vessel running along the handle of the malleus bone (which is adherent to the ear drum), is visible and recordable by the imager, and measurement of the pulsations will record heart rate. Relative expansion of the vessel will vary according to blood flow and pressure. Oxygen saturation changes absorption of red light within biological tissues, and can be detected with imaging sensors. Movement of ear drum and malleus occur during breathing, yawning, eye closure and swallowing, through contraction of the tensor tympani muscle, and pressure transmission through the Eustachian tube, and can be seen with visual imaging. Infrared radiation (heat) can be imaged and will vary according to body temperature and inflammation. Disease processes and changes can be recorded visually with the imager, including showing wax, debris, discharge, inflammation and structural changes such as perforations, infections and Eustachian tube dysfunction. All these changes can be recorded and measured with the imaging sensor of this current invention.

Each individuals ear drum complex is of unique visual appearance. The imager of the current invention provides the ability to record the image from a known individual, and use that information to compare this with the new image obtained each time the device containing the imager is worn. This will be used to authenticate the identity of the user. This will enable the physiological and health data to be compared with the data for the same individual. It will also allow connected technology such as wirelessly attached smart phones to be unlocked in place of using a passcode, by image recognition of the eardrum complex. This biometric user authentication will be useable for any biometric recognition process.

Embodiments of the current invention include sensors which are imagers, including cameras, video cameras, or laser detectors, using any combination of visible or non visible spectrum light or infrared radiation, which are directed towards the ear drum complex. These sensors detect images of the ear drum complex and the associated processors detect physical parameters including movement, position, colour, and shape of the ear drum complex and canal contents, and detect changes in these. Simple imagers will detect a two dimensional image of the ear drum complex. This image is communicated as an input to a processor that may be within the structure containing the sensor (earphone) or connected by wire or wirelessly in another device. Algorithm of the processor will analyse the two dimensional image and detects movement or change in colour or other qualities of the image of the ear drum complex by changes in the image over time (image analysis). This image analysis will be individually configured according to the measurements of interest. The imager may have an associated light source, such as LED emitting visible or infra red light, located within the sensor structure or earphone. The use of infra-red imagers may avoid the need for a light source as the ear structures emit infra-red thermal energy that can be detected as images due to body temperature.

The sensor may be within the ear canal, or located adjacent to the outer ear, with the image relayed to the sensor by conduits of light from the ear canal, such as fibre-optics.

In a further embodiment, sensors consisting of combined laser emitters and receivers, detect movement of the ear drum complex towards and away from the sensor located in the ear canal. These embodiments include measurement methods using laser triangulation, optical coherence tomography, and laser Doppler vibrometry to detect movement of the ear drum complex. The output from the sensors is communicated as an input to a processor that may be within the structure containing the sensor (earphone) or connected by wire or wirelessly in another device. Embodiments include the algorithm of the processor detecting a change in distance between the sensor and the eardrum complex, or by three dimensional analysis of the output detecting a change in the three dimensional shape of the ear drum complex.

The processor generates an output dependent on the algorithm detecting a change of the image of the ear drum complex, which may include change in colour, movement, position, shape or change in size or colour of part or whole of the ear drum complex or canal contents. The algorithm is configurable and programmed on an individual basis according to the specific sensor, application and user, to determine the combination of biological and physiological data that is selected to be obtained by the specific device. This output from the processor representing health and physiological data, and visual images, is communicated to a user interface device such as a desktop computer or smartphone, to analyse the data obtained.

The processor or attached interface device can store an image of the ear drum complex and register this to a specific user. On re-attaching the earphone device to the user the processor in either the device or associated user interface device, can be configured to authenticate the identity of the user, by comparing the new image of the ear drum complex with the registered image. If the image recognition process matches the new image with the registered image it allows the future measurement data/ images to be stored with and compared wth the existing data or stored against a different data set. This same process can be used to authenticate a users identity and prevent use or access to a smart ear phone or wirelessly connected interface or communication device. This biometric recognition process would be applicable in other processes should a recognising individuals to access personal data such as medical data, financial services or individual tracking and security, such as passport control.

The benefit of the current invention over existing physiological and health monitoring devices is that it can be mounted and incorporated in devices that are already incorporated and widely accepted into normal daily living such as earphones and hearing aids. It represents a single sensor that can measure a wide range of physical changes, and also the more complex process of monitoring and detecting disease by recording visual images that can be viewed by the user or healthcare workers on associated devices. These data sets and images can be accessed remotely, allowing remote diagnosis and monitoring, without the need for complex mounting structures or attendances by healthcare workers to change, adjust or maintain the sensors. The imaging sensors are cheap, small and easily available and so easily incorporated into existing earphone type technologies. They will also facilitate easier health care by identifying wax obscuring the canal at an early stage without relying on examination by a health care worker, and earlier enough to allow removal before it has affected hearing function. The invention can be incorporated into a multitude of structures with a multitude of different functions which may complement the health monitoring benefit, such as a falls detector and emergency call aid for frail elderly individuals.

Other embodiments of the invention include the incorporation in earphones and hearing aids, which allows the imager to identify wax or debris collection before it is large enough to obstruct transmission of sound which would affect acoustic performance of earphones or hearing aids. Within these structures it can monitor the response of the ear drum in response to sound transmission form the device to allow optimisation of the sound production in response to this.

Another embodiment is within a safety monitoring earphone device, to monitor the response of the ear drum to occupational noise exposure. Contraction of the tensor tympani muscle which is a middle ear muscle attached to the malleus, occurs in response to loud noises. This movement is visible by detecting movement of malleus, and recording this movement would indicate excessive noise exposure, and help to inform the need for better hearing protection procedures.

Biometric recognition data can be transmitted wirelessly to local sensors, for example to identify individuals rapidly at passport control at country borders, by wireless transmission of this data to dedicated devices to match individual biometrics against other data such as passports to confirm the individuals identity.

The device can be used and worn like a hearing aid or earphone, or ear plug structure (subsequently jointly referred to as earphone), in or adjacent to the ear canal, and can be incorporated into other existing technologies such as hearing aids, earphones, and multi function ear devices which may include many functions including, telephone and entertainment audio play back, microphone for telephone, voice communication and voice commands, accelerometers, pulse oximetry, temperature sensors, technology switches, and any other sensors that are included (often termed smart "hearable" devices).

Brief description of drawings

Figure 1 shows a pictorial representation of a cross section of the right ear canal and partial view of middle ear showing the sensor in the ear canal in relation to ear drum, malleus and tensor tympani muscle.

Figure 2 is a schematic view of the external aspect of a right ear drum showing the impression of the embedded malleus bone.

Figure 3 is a pictorial representation of the sensor worn as an earphone in the right ear canal.

Figure 4 is a pictorial representation of the ear sensor located partially within the ear canal close to the ear drum.

Figure 5 is a flow chart of an embodiment showing biometric recognition of a user to unlock a smartphone's security lock using images from the sensor of the ear drum complex.

Figure 6 is a flow chart of an embodiment showing the use of imaging of the earphone imaging sensor to display and record heart rate on a smartphone.

Figure 7 is a diagram showing various embodiments of different forms of data that can be interpreted from and transmitted from the images obtained from the earphone imaging sensor.

Detailed description:

The present invention describes a sensor worn by a user in or adjacent to the ear canal, with the sensor detecting images of the ear drum complex, which generates an output of data and images that reflect health and physiological data and user identification data.

Figure 1 illustrates one embodiment of the sensor 1, as a cross sectional view of the right ear canal 2. The sensor being located in the ear canal 2, with the sensor being part of an ear phone, hearing aid or ear plug assembly 6. The sensor is directed towards the ear drum complex 5a, partially consisting of the ear drum (tympanic membrane) 3 and the malleus 4 (a middle ear bone) that is connected to the inner aspect of the ear drum 3. The sensor 1 may have a configuration including an imager, which may be a video camera or an infrared video camera, with or without light source, or have a laser emitting and receiving combination which may have a static or scanning laser element (including laser Doppler vibrometry, Optical coherence tomography, laser triangulation), or a combination of these. Movement of the ear drum complex can be in any plane 12,13 (in Figure 2) and is detected by the sensor 1. The tensor tympani muscle 5b attaches to the malleus.

Figure 2 illustrates the external view of the ear drum complex 5a, showing the ear drum 3 and the malleus 4 which is visible through the ear drum 3, where it is located attached to the inner aspect of the ear drum 3. A possible plane of movement of the malleus is shown 13 Figure 3 demonstrates the sensor 1 as one embodiment configured in an earphone or ear plug or hearing aid structure 6 (subsequently jointly referred as earphone) with the sensor 1 being positioned (worn) partially within the ear canal 2, with the external aspect being located within the external ear (pinna) 14. The sensor 1 is directed towards the ear drum 3. Other embodiments include the sensor 1 being located in a structure totally within the ear canal 2 or situated (worn) within or adjacent to the external ear 14, either as an individual structure or physically connected to a similar structure adjacent to the other external ear 14.

Figure 4 demonstrates the sensor 1, within the earphone 6, partially sited within the ear canal 2, directed towards the ear drum complex 5a, showing the ear drum 3, and the malleus 4. Methods of detecting movement of the ear drum complex, or change of shape or colour, include by detecting a visible movement of the ear drum complex 5a by a camera sensor 1, within the earphone, hearing aid or ear plug assembly 6.

Figure 5 is a flow chart of an embodiment showing the imager being within an earphone device 6, and the imager detects an image 16 of the ear drum complex 5a. The image from the imager is transmitted to a processor within the earphone device 17. In this embodiment the image is transmitted wirelessly to a connected smartphone 18. The processor of the connected smartphone receives the image data and analyses the image data with an image recognition algorithm. This algorithm compares the image of the ear drum complex with an existing recorded image of an ear drum complex of a registered user of the smart phone 19. If the image recognition algorithm confirms that the current transmitted image is identified as the same ear drum complex as the recorded image of a registered user of the smart phone, then the security lock on the smart phone is unlocked so that functions of the smart phone can be accessed by the user of the earphone 20. If the ear drum complex is not recognised then the security lock on the smart phone stays active and is not unlocked, so the functions of the smartphone are not available to the user of the earphone.

Figure 6 is a flow chart of another embodiment showing the imager being within an earphone device 6, and the imager detects an image 16 of the ear drum complex 5a. The image from the imager is transmitted to a processor within the earphone device 17. The processor has an algorithm 21 which identifies the blood vessel that runs longitudinally along the malleus 4. This algorithm tracks changes in the image of the blood vessel by comparing changes of diameter of the blood vessel, represented by change of width of the blood vessel in the image. The algorithm records this with respect to time, and calculates heart rate from the information that this provides regarding the pulsatile change in diameter of the vessel over time 22. The processor transmits the information on heart rate wirelessly to a connected smartphone 23. The smartphone processor receives and records the heart rate data and displays this on a graphical user interface, on the smartphone display. In this embodiment the invention can record and display continuous heart rate data.

Figure 7 is a diagram demonstrating embodiments of the invention with a plurality of monitoring modes, measurements, outputs and functions. The embodiments of these sensor modes and outputs are not exclusive, and may be present in any combination in an earphone embodiment of the invention. Embodiments include, but are not limited to the following sensor modes, measurements, outputs and functions. The measurement of the width of the vessel running longitudinally along the malleus 4 provides data on heart rate and blood flow and pressure 25, and this data is provided as an output from a processor in the earphone or associated device, such as a smart phone, and is output to other devices or user interfaces, such as graphical screen displays 34. The measurement of movement of the ear drum provides data on breathing movements, respiratory rate and changes in pressure of the middle ear due to pressure changes via the eustachian tube. These pressure changes may include changes in eustachian tube patency and function, yawning and equalisation such as during the Valsalva manoeuvre 26. The measurement of movement of malleus provides data on the protective contraction of the tensor tympani muscle 5b which is connected to the malleus 4, and contracts in response to loud noises to prevent damage to the ear 27. The measurement of infrared radiation from the ear drum complex 5a, using an infrared imager, provides data on body temperature and local ear infection 28. The measurement of absorption of red components of light, by the imager and processor provides data on the oxygen saturation levels of the blood 29. The recording of image of the eardrum complex provides images that can detect pathology of the ear including, but not limited to, ear infections, ear drum perforations, ear inflammation, cholesteatoma and eustachian tube dysfunction 30. The recording of the image of the eardrum complex provides data to recognise the ear drum complex of an individual wearing the earphone device, so that the data from other embodiments of the invention including health and physiological data, are recognised as belonging to a specific individual and recorded within a dataset for that individual 31. This allows serial recording and monitoring of data overtime to monitor changes in health and physiological variables. The recording of the image of the eardrum complex provides data to recognise the ear drum complex of an individual wearing the earphone device, to provide biometric user recognition data to validate access to, and unlock a wirelessly connected smartphone device 32. This biometric recognition data can also be transmitted to other processors to recognise the user for other uses, such as financial transactions and passport control at borders. The recording of the image of the ear canal provides images of any ear canal contents between the imager 1 and the eardrum complex 5a, and can detect contents including but not limited to, ear wax, debris and pus 33. This can enable decisions to be made as to whether there is pathology or whether the ear canal needs cleaning of debris and ear wax, in order to aid hearing function. All the above embodiments include output from processors which may be in the earphone, in an associated device, which may include a smartphone, or from both sorts of processors, to other devices or processes which may include, but are not limited to graphical user interfaces, databases and other devices 34.

Claims

Claims 1. A method of monitoring any combination of: health, disease, biological and physiological qualities, processes and changes; and of user identification and user authentication, comprising a sensor, locatable in, or near to, the ear canal, capable of: detecting a plurality of properties of the ear drum and surrounding structures; and communicating this as an input to a processor, for generating an output to a connected device; and capable of being located to sense these properties during the user's daily activities.
2. The method of claim 1, wherein the sensor is located wholly or partially within the ear canal, or adjacent to the ear canal in, or close to the external ear.
3. The method of claim 1, wherein the sensor is an imager including a camera, located in the ear canal.
4. The method of claim 3, wherein the ear structures are illuminated by a light source to enable the imager to record images, which may emit visible or infra-red or other non-visible light.
5. The method of claim 4, wherein the imager is an infra-red camera which detects images of the ear structures.
6. The method of claim 1, wherein the the sensor comprises laser Doppler vibrometry, digital laser C;) holography, optical coherence tomography, laser triangulation or other laser measurement C\I analysis.
Nt 7. The method of claim 1, wherein the the sensor includes detection of ear structures using O ultrasound, visible or non-visible spectrum light analysis, or other proximity sensor.
N 8. The method of claim 1, wherein the sensor detects properties of the detected structures, including but not limited to: visual image, two dimensional image, appearance, measurement, size, colour, shape, structure, contour, pattern, distance, position, movement, temperature, optical absorbance, optical reflectivity, infrared image, three dimensional shape and surface features, to include, but not limited to, blood vessel diameter.
9. The method of claim 1, wherein the sensor is incorporated into a hearing aid, which may be located in the ear canal, or adjacent to the ear canal, in or close to, the external ear.
10. The method of claim 1, wherein the method is incorporated in an earphone incorporating any combination of hearing improvement function, microphone, audio playback including telephone receiver, health monitoring such as temperature and oxygen saturations, head movement by accelerometer, and voice communication using a microphone, camera or any other device locatable in, or near to, the ear canal.
11. The method of claim 1, wherein the method is incorporated into any device worn in or adjacent to the ear canal.
12. The method of claim 1, wherein the processor is housed in a structure partially or wholly located in the ear canal, and also containing the sensor.
13. The method of claim 1, wherein the processor is connected to the sensor by wire or by wireless communication, to include but not limited to, a processor located within another device such as a user interface device or mobile phone, database or other device.
14. The method of claim 1, wherein the processor stores data from the sensor within an electronic storage medium, to include but not limited to: storage within the structure containing the sensor, an attached user interface device or portable electronic device.
15. The method of claim 1, wherein the processor algorithm generates an output dependent on detecting a change in property of the data from the sensor, included but not limited to change in image, two dimensional image, appearance, measurement, size, colour, shape, structure, contour, pattern, distance, position, movement, temperature, optical absorbance, optical reflectivity, infrared image, three dimensional shape and surface features, to include, but not limited to, blood vessel diameter, or any other change in data from the sensor.
16. The method of claim 1, wherein the processor algorithm interprets the detected properties, and changes in properties, as biometric data, to include but not limited to: pulse rate, pulse rhythm, oxygen saturation, breathing movement, breathing rate, ear drum movement, Eustachian tube function, blood flow, blood pressure, body temperature and inflammation.
17. The method of claim 14, wherein the processor algorithm compares the stored data with real time data, and generates an output dependent on whether the data is sufficiently similar to reach a predefined level of confidence that the real-time data represents data from the same individual's ear structures.O
C\I 18. The method of claim 17, wherein the the output of the processor determines whether the functions of: the processor, or of a connected processor, or remote wirelessly connected processor, are unlocked or recognised as a biometric user authentication, or user identification.
CD 19. The method of claim 1, wherein the processor output transmits data, to include images of the Ne. ear structures, including but not limited to images of the normal ear structures, images of obstruction to the ear canal, and disease processes, to any connected user interface, that may be locally connected, or remotely connected, for example through an internet connection.
20. The method of claim 1, wherein the algorithm detects movement of the eardrum or associated structures, related to sound, including from external sound, or from sound transmitted from an earphone or hearing aid or other device in which the sensor is located.
21. The method of claim 20, wherein the algorithm output provides feedback to change properties of the sound production from the earphone or hearing aid, or other function of the device in which the sensor is located.
22. An apparatus for monitoring any combination of: health, disease, biological and physiological qualities, processes and changes; and of user identification and user authentication, comprising a sensor, locatable in, or near to, the ear canal, capable of: detecting a plurality of properties of the ear drum and surrounding structures; and communicating this as an input to a processor, for generating an output to a connected device; and capable of being located to sense these properties during the user's daily activities.
23. An apparatus as claimed in claim 22, wherein the apparatus is a hearing aid, arranged to be locatable in the ear canal, or adjacent to the ear canal, in or close to, the external ear.
24. An apparatus as claimed in claim 22, wherein the apparatus is an earphone incorporating any combination of the group comprising: hearing improvement function; microphone; audio playback, including telephone receiver; health monitoring, such as temperature and oxygen saturations; head movement, by accelerometer; voice communication, using a microphone; camera; and/or any other in-ear locatable device.