RU2802853C1 - Spatial sound system for navigation of visually impaired people - Google Patents

Abstract

FIELD: medicine.

SUBSTANCE: invention relates to devices that enable visually impaired patients to replace direct visual perception with another kind of perception. A spatial sound system for navigation of people with visual impairments comprises an ultrasonic distance sensor (5), a lidar sensor (7), an infrared proximity sensor (6), a processing unit (2), a memory unit (10) and a power source (4) placed on the frame, as well as a sound reproduction module (3). The three-axis gyroscope-accelerometer (8) of the system is combined with an ultrasonic sensor (5), a lidar sensor (7) and an infrared sensor (6) into a sensor unit (1). The processing unit (2) includes an analogue-to-digital converter (11), a computing device (9), a digital-to-analogue converter (12) and a memory unit (10). The power supply (4) feeds the sensor unit through the processing unit (2). The system is configured to transmit data from the sensor unit (1) to the processing unit (2) through an analogue-to-digital converter (11), which generates spatial sounds after combining the data from the sensor unit (1) with the data of the gyroscope-accelerometer (8). The memory unit (10) stores and uses the readings of the sensors to generate the amplitude and frequency of the spatial sound.

EFFECT: accurate determination of the position of obstacles in space by the user is provided using an autonomous device for navigating in space for people with visual impairments.

1 cl, 9 dwg

Description

The invention relates to the field of devices that give patients with visual impairments the opportunity to replace direct visual perception with another type of perception.

Visual impairment, also known as vision impairment or vision loss, is a decrease in the ability to see to such an extent that problems occur that cannot be corrected by conventional means such as glasses. Visual impairment is more common than it seems. Here's a summary from WHO:

• Worldwide, the number of people with visual impairment is estimated at 2.2 billion, of whom 39 million are blind.

• People aged 50 years and older account for 82% of all blind people.

• Significant causes of visual impairment are:

uncorrected refractive errors (43%) and cataracts (33%); The first cause of blindness is cataracts (51%).

• Visual impairment is a major global health problem in 2010, with preventable causes accounting for up to 80% of the total global burden.

The number of people with visual impairments (VIs) is large enough to interest medical and biomedical scientists in finding new ways to improve the quality of life of people with visual impairments.

In general, all concepts have the same principle. A sensor is placed on a part of the body that is constantly free from interference. It receives a signal from the environment, which is analyzed by the processor. This block will produce an output signal that the viewer will perceive using other senses. Like the cane or guide dog, traditional methods are outdated, risky, and ineffective.

In particular, US 20200228914 A1 describes a mobile device used to assist blind people, people with dementia and people in general who do not know (or have forgotten) how to navigate a particular place. The device connects to external headphones used to produce directional sound waves of a continuous tone, beep, or voice phrase. Sounds have variable frequency and intensity. The device must have a predetermined (selected) purpose to assist these people. The device is also used as a location data receiver. It uses GPS data to determine the user's direction of movement along a non-linear trajectory.

Several flaws were found with this device. Namely, the device can only provide guidance regarding a given target location, without providing the user with any information about recurring obstacles that he may encounter, such as other people, kiosks, bus stops, and others. Moreover, the use of GPS may be limited because GPS reception is affected by external factors such as wooded areas, water, and man-made structures. Additionally, the GPS receiver consumes a lot of power, requiring more charging cycles, which drains the battery faster.

WO 2015112651 A1 describes an optical device for people with low vision and a method for operating such a device by people who retain at least a residual ability to distinguish between light and colors. This in itself limits the target group of potential users. The design is described as a head-mounted device in the style of a glasses frame, consisting of optical and acoustic sensors to help people navigate and navigate their surroundings. The glass lenses are equipped with LED matrices that light up when an obstacle is detected. Two wide-angle video cameras identify obstacles. In addition to optical feedback, the user is also provided with additional acoustic feedback. The device takes advantage of the person's residual visual function as the main channel for transmitting information. The components are mounted on a frame and connected to a portable computing device. The computing device is powered by an internal battery, which may be rechargeable. The battery also powers the components on the frame. The device is capable of recognizing objects, text and faces. The device is equipped with voice control.

Although this device is multifunctional, it does have a drawback. First of all, the functionality of the device is based on the user's residual vision. This is a limitation, but it is also questionable since there is no way to ensure that the user will be able to correctly identify the feedback provided by the LEDs. Moreover, if an object is not identified, the user will be instructed to re-present it from a different angle (e.g., by changing its relative position). This can be potentially dangerous in an unfamiliar space. It is also unknown how the device will correctly recognize many objects, text and people, since there is no data on the accuracy and processing time of the algorithm, since image recognition is based on comparing the outlines and shades of detected objects with those stored in created libraries. Additionally, the internal battery is stated to be possibly rechargeable and underpowered, meaning the device may not be functional for all-day use.

The closest analogue is an echolocation device (US 2014269189), which can be used by a person without vision, and a method for generating sounds that can facilitate navigation of a person without vision in the environment. This portable device was created to help visually impaired people improve their echolocation skills and teach them to identify their surroundings (the location and size of objects). The device contains a frame of glasses for mounting on the head, an ultrasonic sensor, which is an acoustic signal source and an echo signal converter, a lidar sensor, an infrared (IR) sensor, providing feedback to the protection (piezoelectric speaker) and a processing unit (processor), as well as the source power supply, user input interface, power amplifier, voltage regulator, memory unit.

The functionality of the device is based on two memory blocks. Each of them generates sounds (which are already stored in the memory block) in accordance with the size of the room. The device makes sounds, and the user must determine what the environment is made of by the echo. The Braille cell interface allows the device to operate through a user interface.

Although this device has a different approach than those previously discussed in the sense that it is used to train users to use echolocation on their own, there are some points that need to be emphasized regarding its use. Most importantly, the device's accuracy relies too heavily on the user's ability to identify objects by their echolocation. This means that human error, which can be limiting and potentially dangerous, not to mention fatal, is an inevitable parameter in the use of the device. Moreover, the accuracy of the device depends too much on the user's ability to identify objects by their echo signal. Finally, the battery's performance is not guaranteed, as it is noted that it can last at least 7 hours with no maximum time specified. This may be sufficient for some people, but may not meet the requirements of others.

An analysis of existing patents and an assessment of their strengths and weaknesses showed that none of them were completely independent of external sources of information, such as GPS technologies. Most existing patents describe devices that require a specific route to be pre-defined.

The objective of the invention is to create a reliable autonomous device for navigation in space for people with visual impairments.

The technical result is to ensure the accuracy of the user's determination of the position of obstacles in space.

The problem is solved due to the fact that the spatial sound system for navigation of people with visual impairments contains an ultrasonic distance sensor, a lidar sensor, an infrared proximity sensor, a processing unit, a memory unit and a power source located on the frame. The device additionally contains a three-axis gyroscope-accelerometer combined with an ultrasonic sensor, a lidar sensor and an infrared sensor in a sensor unit located on the frame, as well as a sound reproduction module. The processing unit includes an analog-to-digital converter, a computing device, a digital-to-analog converter and a memory unit. Moreover, the output of the sensor unit is connected to the input of the processing unit, which is the input of the analog-to-digital converter, while the output of the analog-to-digital converter is connected to the first input of the computing device, the first output of which is connected to the digital-to-analog converter, and the second to the memory block, the output of which is connected to the second input of the computing device. The output of the digital-to-analog converter is connected to the input of the audio reproduction module. The power supply powers the sensor unit through the processing unit. The system is configured to transmit data received from the sensor unit to the processing unit through an analog-to-digital converter, which generates spatial sounds after combining data from the sensor unit with data received by the gyroscope-accelerometer. The memory unit is configured to store and use sensor readings to generate the amplitude and frequency of spatial sound.

The technical result is achieved due to the fact that the device contains a set of sensors that make it possible to objectively determine the position of obstacles relative to the user, and generates an information sound signal for the user, generated based on the measurements of these sensors, the frequency and amplitude of which will be different for different positions of the obstacle.

The invention is illustrated by figures, where:

Fig. 1 - Systematic diagram of the blocks of the spatial sound navigation system,

Fig. 2 - Schematic representation of the main components and processes of the device,

Fig. 3 - Distance perception step structure showing the communication between the environment and the device, and also between the device and users,

Fig. 4 - Schematic explanation of the perception of obstacles when moving in space. As the user moves (circle), the adjacent vertical remains the same, while the opposite vertical and hypotenuse change. The hypotenuse (distance to the object) is calculated based on the Pythagorean trigonometric identity theorem,

Fig. 5 - Illustration of the change in horizontal resolution calculated using azimuthal resolution , which arises when subtracting the maximum and minimum azimuthal angles And , formed by the edges of the obstacle,

Fig. 6 - Expanding the human field of view (FOV) using spatial audio,

Fig. 7 - Schematic representation of the user turning at an angle with simultaneous operation of all sensors,

Fig. 8 - Schematic illustration of determining the location of an obstacle in 2D,

Fig. 9 - Demonstration of spatial sound generation using HRTF for 8 obstacles in the user's environment. HRTF - Head Relatively Transmitted Function is a mathematical expression of a person's ability to determine the position of a sound source by filtering sound waves as they travel from the source through the air to reach the listener's ears.

The device diagram is shown in Fig. 1 showing all sections and components. The device has four main blocks, indicated by a dotted line. Namely: sensor unit (BD) 1, processing unit (PU) 2, sound reproduction module (SPM) 3 and power supply (PS) 4. Sensor unit 1 includes four types of sensors: ultrasonic (USD) 5, infrared ( ICD) 6, lidar sensor (LD) 7 and three-axis gyroscope (G) 8. Processing unit 2 is an all-in-one (AIO) development board. It includes a computing device 9, a memory unit (MU) 10, an analog-to-digital (ADC) 11 and a digital-to-analog (DAC) 12 converters. The output of the sensor unit 1 is connected to the input of the ADC 11, which is the input of the processing unit 2. The output of the ADC 11 is connected to the first input of the computing device 9. The first output of the computing device 9 is connected to the input of the PDA 12. The second input and the second output of the computing device 9 are connected to the output and the input of the memory block 10, respectively. The output of PAP 12 is connected to the input of the audio reproduction module 3. The audio reproduction module 3 includes the main components for generating spatial sounds and providing users with an audio signal. The power supply 4 supplies power to the processing unit 2, which in turn powers the sensor unit 1 (the power supply is shown in Fig. 1 with dotted lines). The components of all sections are indicated by a solid line, and the data and power channels have their own solid and dotted arrows, respectively.

The power supply 4 supplies the processing unit 2 with a constant voltage of 5 V DC. Processing unit 2, in turn, powers sensors 5, 6, 7 and 8 via a parallel connection. The sensors use a serial protocol to communicate with the processing unit.

Signals from sensor unit 1 are transmitted to ADC 11, where they are digitized and then transmitted to computing device 9, where sensor readings are processed, which transmits them to memory unit 10 for storage and use in generating the amplitude and frequency of spatial sound. They are sorted, selected and re-transmitted to the computing device 9, which sends them to the DAC 12. To generate spatial sounds, these readings are converted back to analog in the DAC 12. Based on the calculations of the computing device 9, the DAC 12 generates signals that are transmitted to the audio reproduction module 3 .

In fig. Figure 2 shows a schematic representation of the main components and processes of the device. Sensors 5, 6 and 7 interact with the environment 13 and estimate the distance to objects. The data enters processing unit 2 and, using gyroscope data 8, is converted into spatial sound. Four sensors (5, 6, 7 and 8) are shown scanning and measuring the distance to detected objects in the user's environment 14. The received data is sent and temporarily stored in the processing unit 2. The gyroscope 8 tracks the user's orientation with respect to a reference point. This data is combined with data obtained by sensors. The sound reproduction module 3 generates spatial sounds perceptible to users 14, notifying them of the exact location of obstacles in space.

Time of flight (TOF) is a measurement of the time it takes an object, particle, or wave (acoustic, electromagnetic, or other nature) to travel a distance through a medium. This information can be used to measure speed or path length.

Since the signals transmitted by the sensors are quantized, the board easily calculates the distance between the sensor and the object. The calculation is based on the time of flight principle. For each quantized signal transmitted by a sensor, there is a specific time required for it to collide with an obstacle at a given distance and return to the sensor.

Based on a simple equation

or

where v is the speed of the signal, Δx is the distance traveled by the signal, Δt is the time it took the signal to travel this distance.

The distance to each object is calculated instantly.

The presented diagram in Fig. 3 is a more compact, but at the same time explanatory version of the operation of the device. This diagram shows the processes of the signal from the sensors of block 1 from environmental obstacles 15. The received data is transmitted to the signal processing unit 2 through the ADC 11, which generates spatial sounds after combining this data with the data received by the gyroscope 8. Then, through the DAC 12, the audio reproduction module 3 provides the user 14 with spatial audio that helps him in navigation, helping choose a clear path.

Providing navigation of the device when an obstacle 15 is detected is best seen in FIG. 4, illustrating the perception of obstacles when moving in space. As the user 14 moves, the adjacent vertical A remains unchanged, while the opposite vertical B and hypotenuse C change. The hypotenuse C (distance to the object) is calculated based on the Pythagorean trigonometric identity. This diagram shows three right triangles. As the user moves along the X-axis, always keeping the detected obstacle diagonally to their right, the adjacent vertical A remains unchanged, while the opposite vertical B and hypotenuse C change. The spatial sound perceived by the user follows the trace of the hypotenuse in each triangle, constantly alerting the user to the location of the object.

The user's field of view (FOV) is divided into two categories: horizontal and vertical (taking into account neck movement).

The horizontal PV is represented by the following equation:

where FOV _H is the horizontal FOV, - maximum azimuthal angle, and - minimum azimuthal angle. The minimum azimuth angle corresponds to the beginning of the user's head rotation, and the maximum azimuth angle corresponds to the end.

A full area scan can be better understood by considering the following equations:

where H _RES is the horizontal resolution, and azimuthal resolution in degrees achieved by horizontal movement of the human neck, and:

where V _RES is the vertical resolution, ad _ϕ is the elevation angle resolution in degrees achieved by the vertical movement of the human neck, and:

where N is the total number of scanning points in the user's PP.

Fig. 5 illustrates the concept of resolution described in equations (1)-(4). Illustration of the change in horizontal resolution calculated using azimuthal resolution arising when subtracting the maximum and minimum azimuthal angles formed by the edges of an obstacle, resolution is shown on the horizontal plane, but the same principles apply to the vertical plane.

An important aspect of spatial sound generation is timing. Spatial audio will accompany the user every step during navigation; Therefore, it is necessary to calculate the time required to completely scan a frame equal to 360° as:

where T _LINE is the time required for one full scan in the PP and:

where T _RES is the required time to collect data from one point. The T _FRAME variable is the time required to collect data on one full frame, calculated according to the equation:

where A _v is the angular vertical velocity of each sensor, set by the manufacturer.

Based on the above equations, the angular velocity in the user's FZ can be calculated as:

where A _HR is the angular velocity on the horizontal plane. These equations apply to LIDAR, IR, and ultrasonic sensors.

Scanned points are stored in the device's memory block, spatial sounds are generated and transmitted to the playback module for the user. By storing scan points in this way, the audio representation of obstacles and sound source will change as users move through the environment, thereby creating a spatial audio effect. These audio scanning points will accompany the user while navigating within a limited safe distance.

This auditory representation of objects has a number of advantages, the most significant of which is the perception of objects that are not usually perceived. A person's visual field of vision along the X axis, including peripheral vision, is 170 degrees. However, although peripheral vision is vital to human vigilance, the useful PV is approximately 135 degrees (angle β). After a complete scan of the room, a person’s PV can be artificially expanded to 180 degrees (angle γ). Since human ears are positioned anti-diametrically, panning the sound is key to this. This concept is better illustrated in FIG. 6. User 14 is depicted as a circle and the X-axis field of view is shown in red and is indicated as 135 degrees (angle β), while the extended field of view is 180 degrees (angle γ). When user 14 turns at an angle to the right, the detected obstacle located in the center of the field of view is heard by the left ear more than by the right. When equal to 90 degrees, the sound reaches only the left ear of the user 14, and vice versa.

Understanding the user's rotation at an angle to the right along the X axis is shown in Fig. 7. The user changes his PZ at an angle At the end of the turn, the user is able to perceive obstacles from both steps 16 and 17. Obstacles detected in intermediate steps are ignored. Step 16 shows user 14, depicted as a circle, standing straight on the X-axis; step 18 shows the transition the user is about to make relative to his orientation; step 19 shows the transition itself, and step 17 shows the end of the transition. The solid line illustrates the start and end points of the transition, the transition step is illustrated by the dark dotted line, and the start and end points during the transition are illustrated by the pale dotted line.

Determining the amplitude of the acoustic signal reproduced by the sound reproduction unit from the system is similar to the calculated distance.

For the distance D between the user's eyes and the obstacle,

where f _IR , f _LiDAR and f _US are weighting factors that adjust the signal output of each sensor according to its importance, represented in terms of the accuracy specified by the manufacturer (calculated using the accuracy of each sensor as specified by the manufacturer (see after Equation 16) ), and O _IR , O _LiDAR and O _US are the digital amplitudes of the sensor output signal. The IR, LiDAR and US indices correspond to infrared, lidar and ultrasonic sensors, respectively.

The weighting coefficients are calculated based on the characteristics of each of the three sensors. The characteristics that determine the formation of the coefficients are the PV of the sensors and the maximum range of each of them. The PV of sensors is inversely proportional to the ranges. Humans in general are more sensitive to sound frequencies in the range of 2000 to 5000 Hz. The frequency range of feedback from lidar and ultrasonic sensors is set between 300 and 500 Hz. Frequencies in this range are still perceptible to users and do not interfere with other audio signals, such as traffic signals. Constant 1000 (Hz) is our choice of alert frequency for the PC sensor, given its binary use.

Therefore, the determination of system frequency can be expressed by the following equation:

where F is the linearly generated frequency depending on the distance to the scanned object, and C _fD is the slope. C _fD is calculated as the fraction of the difference between the maximum and minimum distance that the sensors can detect in the range of frequencies perceived by users as feedback, and is equal to

Head-relative transmission function (HRTF) is a mathematical expression of a person's ability to determine the position of a sound source by filtering sound waves as they travel from the source through the air to reach the listener's ears. HRTF is the frequency response of the human ear. This frequency response describes how the acoustic signal is filtered through the entire upper torso of a person, namely the head, shoulders and especially the pinna of the external auditory canal, and is fundamental to the generation of spatial sounds in binaural models.

The HRTF is capable of generating sound for each ear separately according to the location of the obstacle. Location is defined in two dimensions, defined by the horizontal and vertical plane of the object.

This concept is better illustrated in FIG. 8, where the user 14, depicted as a circle, stands in front of an obstacle 15, which is at an angle ϕ from his eye level, since its height h is less than the eye level of the user H. The two planes of the obstacle are formed by vertical and horizontal lines passing through its center. As the vertical line extends upward, it intersects with the line passing through the eye level of user H. The eye level of each user is different, and it must be predetermined in the algorithm. The red dot formed by the two extended lines is called the intersection point, and it is necessary to estimate the 2D coordinates of the obstacle, which are expressed by the equations:

And

where X is the distance between the vertical planes 20 of the user's eyes 14 and the obstacle 15, Y is the distance between the horizontal planes of the user's eyes 14 and the obstacle 15, and ϕ is the angle formed by the horizontal plane 21 of the user and his distance D from the obstacle.

The distance D varies depending on the sensor ranges and is summarized as:

From equations (11) and (15) it follows that:

Where And - weighting coefficients, acc _US - ultrasonic sensor accuracy, and acc _LiDAR - lidar sensor accuracy, are provided by the manufacturer.

In conclusion, the left and right ear sounds generated by HTRF are expressed by the equation:

where A _R is the amplitude for the right ear, A _L is the output signal for the left ear, and X and Y are the calculated distances according to equations (13) and (14).

The device can be implemented based on the following components.

The main sensor is the LiDAR TFmini Plus sensor. It is the most powerful sensor, providing a maximum range of 12 meters to 0.1 meters with an accuracy of 0.1m~12m at 90% reflectivity and an accuracy of ±5cm at 0.1-6m and ±1% at 6m - 12 m. The principle of the method he used is - the IP that is required for the laser beam to reflect off the surface and return to the sensor. It is no coincidence that it is chosen by many ground vehicles and aircraft to estimate location from the surrounding environment.

The sensor used to accompany and complement the lidar sensor is an ultrasonic distance sensor - the HC-SR04 can be used as it. It serves the same purpose as the lidar sensor and performs the function of measuring IP, although instead of a laser beam it emits an ultrasonic wave. The disadvantage of this sensor is that it is affected by external sound interference; therefore, it is not a primary sensor and is used primarily in indoor, quiet environments. It also has a shorter range than lidar; however, the PP is broader. It is the preferred choice as a secondary sensor, rather than using another lidar, due to lower power consumption, lower cost, and the ability to compare and verify cross-signals.

The third sensor used can be an infrared proximity sensor, Waveshare IR obstacle avoidance sensor, estimated distance - 2 ~ 30 cm. VP is, again, the principle of the method used. It has the shortest range of the three and the widest viewing angle. However, it was necessary to have a sensor that would act as a switch and alert the user to nearby obstacles in a wider FOV, which could potentially be a problem for people with low vision.

A three-axis gyroscope-accelerometer, namely MPU-6050 GY-521 3-Axis Accelerometer & Gyroscope Sensor Module, can be used as a gyroscope. It's an inexpensive yet powerful component that combines a 3-axis gyroscope and accelerometer in a small form factor. The gyroscope is equipped with a digital motion processor (DMP). The spatial amplitude of the sound for the left A _l and right A _p ear is generated using the head-relative transfer function (HRTF) after determining the azimuth X and height of obstacles Y from the data obtained from the sensors 5-7 and the gyroscope 8, shown in FIG. 9.

The components above can be connected to the Arduino development board, and audio playback can be realized with any pair of wired stereo headphones.

Claims (1)

Spatial sound system for navigation of people with visual impairments, containing an ultrasonic distance sensor, a lidar sensor, an infrared proximity sensor, a processing unit, a memory unit and a power source located on the frame, characterized in that the device additionally contains a three-axis gyroscope-accelerometer, combined with an ultrasonic sensor, a lidar sensor and an infrared sensor into a sensor unit located on the frame, as well as a sound reproduction module, wherein the processing unit includes an analog-to-digital converter, a computing device, a digital-to-analog converter and a memory unit, wherein the output of the sensor unit is connected with the input of the processing unit, which is the input of the analog-to-digital converter, and the output of the analog-to-digital converter is connected to the first input of the computing device, the first output of which is connected to the digital-to-analog converter, and the second to the memory block, the output of which is connected to the second input of the computing device. device, and the output of the digital-to-analog converter is connected to the input of the audio reproduction module, wherein the power source powers the sensor unit through the processing unit, and the system is configured to transmit data received from the sensor unit to the processing unit through the analog-to-digital converter, which generates spatial sounds after combining data from the sensor unit with data received by the gyroscope-accelerometer, and the memory unit is configured to store and use sensor readings to generate the amplitude and frequency of spatial sound.