CN110559127A - intelligent blind assisting system and method based on auditory sense and tactile sense guide - Google Patents
intelligent blind assisting system and method based on auditory sense and tactile sense guide Download PDFInfo
- Publication number
- CN110559127A CN110559127A CN201910793409.6A CN201910793409A CN110559127A CN 110559127 A CN110559127 A CN 110559127A CN 201910793409 A CN201910793409 A CN 201910793409A CN 110559127 A CN110559127 A CN 110559127A
- Authority
- CN
- China
- Prior art keywords
- target object
- module
- information
- stimulation
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
- A61F9/00—Methods or devices for treatment of the eyes; Devices for putting-in contact lenses; Devices to correct squinting; Apparatus to guide the blind; Protective devices for the eyes, carried on the body or in the hand
- A61F9/08—Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
Landscapes
- Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Vascular Medicine (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
an intelligent hearing and haptic guidance based blind assistance system comprising: the system comprises a sound acquisition module, a camera module, an information processing module, an auditory stimulation module and a tactile stimulation module, wherein the sound acquisition module acquires an external sound signal and outputs the external sound signal to the information processing module; the invention transmits the prompting voice which has a virtual three-dimensional stereo sound effect and contains the position and space estimation distance information of the target relative to the camera module to the user through the combined guidance of the auditory sense stimulation module and the tactile sense stimulation module, and gives the user the prompting information of the direction and the distance of the target relative to the object-taking hand so as to efficiently finish the daily complex activities.
Description
Technical Field
The invention relates to a technology in the field of biomedical engineering, in particular to an intelligent blind assisting system and method based on auditory sense and touch sense guidance.
Background
the existing blind assisting device only has the function of guiding the blind to approach a required article, and does not relate to the guidance of the blind in the process of extending hands to fetch articles; the blind assisting device singly adopting voice prompt does not utilize the sharp touch perception capability of the blind, and is difficult to efficiently complete more complex daily tasks; the blind assisting device for acquiring the spatial depth information by adopting the depth camera has high cost, large power consumption and high complexity.
disclosure of Invention
Aiming at the defects in the prior art, the invention provides an intelligent blind assisting system and method based on auditory sense and tactile sense guide, wherein a user is guided by the auditory sense and the tactile sense in a combined manner through an auditory sense stimulation module and a tactile sense stimulation module, a prompt voice which has a virtual three-dimensional stereo sound effect and contains position and space estimated distance information of a target relative to a camera module is transmitted to the user, and the user is given the prompt information of the target relative to the direction and the space estimated distance of a fetching hand through the tactile sense stimulation module, so that the user can more efficiently finish complex activities in daily life, and inconvenience caused by blindness is reduced.
The invention is realized by the following technical scheme:
The invention relates to an intelligent blind assisting system based on auditory sense and tactile sense guide, which comprises: sound collection module, camera module, information processing module, auditory stimulation module and tactile stimulation module, wherein: the sound collection module collects external sound signals and outputs the external sound signals to the information processing module, the camera module collects external scene image information and outputs the external scene image information to the information processing module, the information processing module analyzes and processes the external sound signals and the scene image information and outputs processing results to the auditory stimulation module and the tactile stimulation module respectively through a listening tactile information distribution algorithm, the auditory stimulation module outputs voice prompts according to the processing results, and the tactile stimulation module implements tactile stimulation according to the processing results.
the invention relates to a control method of the system, which comprises the following steps:
step 1: when the voice recognition device is used for the first time, a voice sample of a user is recorded in a quiet environment through the voice collection module, and after voice characteristics of the user are extracted and analyzed through the information processing module, a user voice characteristic model is established and stored in a voice characteristic model library;
step 2: in the object searching process, the sound acquisition module and the camera module continuously acquire external sound and images of scenes around the user and send the external sound and the images to the information processing module;
and step 3: the information processing module compares the external sound signal with the user voice feature model, and when the external sound signal is judged to be absent, the information processing module does not generate any output information or instruction and continues to analyze and process the external sound signal; when the voice signal of the user exists, the information processing module extracts the voice signal, converts the voice signal into characters and generates the name of the target object and the feature description of the name;
and 4, step 4: the information processing module identifies, detects and estimates depth information of a target object in a scene image according to the name of the target object and the characteristic description of the target object, judges whether the target object exists in the image through a target object existence judgment algorithm, generates a processing result without the target object in the image and sends a prompt instruction to the auditory stimulation module when the information processing module judges that the target object does not exist in the image, the auditory stimulation module informs a user that the target object is not found and asks the user to adjust the direction in a voice prompt mode, and then the step 3 is repeated; otherwise, entering step 5;
And 5: the information processing module converts the direction and space estimated distance information of the target relative to the camera module into corresponding prompt voice signals, performs virtual three-dimensional stereo modification processing and sends the prompt voice signals to the auditory stimulation module;
step 6: the user receives the corresponding voice prompt, when the number of the prompted target objects is only one, the user sends voice to confirm whether the target objects are the required target objects, the information processing module obtains the voice information of the user according to the method in the step 3, if the user confirms that the object identified by the information processing module is the required target objects, the step 7 is carried out, and if not, the step 3 is returned; when the number of the prompted target objects exceeds one, the user sends out voice to describe or select the target objects more specifically, and the step 3 is returned;
and 7: the information processing module presets a target object distance threshold, after the position calibration, tracking and distance estimation are carried out on the target object, whether the distance between the target object and the camera module is greater than the target object distance threshold or not is judged, if so, the step 8 is carried out, and if not, the step 9 is carried out;
And 8: the information processing module identifies and detects the obstacles in the image, plans a user fetching path, transmits the obtained target object and obstacle information and path guiding information to an auditory sense information distribution program, transmits the processed corresponding information to the auditory sense stimulation module and the tactile sense stimulation module respectively, and enters step 11;
The path guiding information comprises: planning the path direction and the remaining path distance.
The target and obstacle information includes: object or obstacle name, direction of presence, relative distance, shape, color, and numerical information.
and step 9: the information processing module judges whether an object taking hand exists in the image, and when the judgment result is that the object taking hand does not exist in the image, the auditory stimulation module prompts the user to 'please stretch the hand to take the object', and the step 4 is returned; otherwise, entering step 10;
Step 10: the information processing module identifies and detects obstacles in the image after carrying out position calibration, tracking and distance estimation on a target object and a user fetching hand, plans a fetching path for the fetching hand of the user by combining the obtained direction and space distance information of the target object relative to the fetching hand, transmits the obtained target object, obstacle information and path guide information to an auditory sense information distribution program, and transmits the processed corresponding information to the auditory sense stimulation module and the tactile sense stimulation module respectively;
step 11: the auditory stimulation module gives a voice prompt to the user, and the tactile stimulation module gives tactile stimulation to the object-fetching hand of the user to cooperatively guide the user to seek and fetch objects; returning to the step 4;
Step 12: the information processing module analyzes and judges whether the user grabs the target object according to the relative position and the spatial distance relationship between the target object and the object fetching hand in the image, and when the user grabs the target object is judged, the blind assisting task is finished, the step 2 is returned, and the user is waited to send out the object searching voice again; otherwise, returning to the step 4.
The form of the tactile stimulation comprises: vibration stimulation, pressure stimulation, electrical stimulation, cold and hot stimulation.
Technical effects
compared with the prior art, the invention can carry out combined guidance of auditory sense and tactile sense to the user, fully and effectively utilize the sharp auditory sense and tactile sense of the user, enable the user to more efficiently finish the complex activities in daily life, and reduce the inconvenience caused by blindness;
The invention not only identifies and detects the target object and guides the user to approach the target object, but also identifies and detects the object-taking hand of the user, and can perform tactile guidance on the movement of the object-taking hand of the user according to the position of the target object relative to the object-taking hand, so that the user can timely receive tactile prompt feedback of the position of the target object relative to the object-taking hand during the object-taking process, and the user can be helped to quickly and accurately find the object;
The invention adopts an image algorithm based on deep learning to estimate the image depth information, so that a single common camera can have the functions of a structured light camera, a binocular camera or a time of flight (TOF) camera and other complex depth cameras, and the manufacturing cost, complexity, failure rate and power consumption of the system are reduced; the program in the information processing module can be directly upgraded and perfected, and the system is convenient to maintain and update.
drawings
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic diagram of the system of the present invention;
FIG. 3 is a schematic cross-sectional view of a tactile stimulation module of the present invention;
FIG. 4 is a schematic flow chart of the method of the present invention;
in the figure: the device comprises a sound collection module 1, a camera module 2, an information processing module 3, an auditory stimulation module 4, a tactile stimulation module 5, a microphone 6, an earphone 7, a fixed connecting ring 8, a power supply module 9, special glasses 10, an elastic material 11, a Bluetooth microcontroller 12, a power supply battery 13, a front vibrator 14, a rear vibrator 15, a horizontal vibrator 16, an upper right vibrator 17 and a mobile device 18.
Detailed Description
As shown in fig. 1, the intelligent blind assisting system based on auditory and tactile guidance according to this embodiment includes: sound collection module 1, camera module 2, information processing module 3, auditory stimulation module 4 and tactile stimulation module 5, wherein: the sound collection module 1 is connected with the information processing module 3 to collect and transmit external sound signals, the camera module 2 is connected with the information processing module 3 to collect and transmit scene image information, the information processing module 3 is respectively connected with the auditory stimulation module 4 and the tactile stimulation module 5 to receive input information, analyze and process the input information and transmit a processing result, the auditory stimulation module 4 receives the processing result and outputs a voice prompt, and the tactile stimulation module 5 receives the analysis result and implements tactile stimulation.
As shown in fig. 2, the camera module 1 and the microphone 6 as a sound collection module send collected scene images and external sounds to the information processing module 3, and after the information processing module 3 analyzes and processes the information from the camera module 1 and the microphone 6, on one hand, the information processing module sends prompt information to the earphone 7 as the auditory stimulation module 4; on the other hand, the control command bluetooth is transmitted to the tactile stimulation module 5.
The information processing module 3 includes: the device comprises a memory for storing the names, the directions, the relative distances, the shapes, the colors and the quantity information of the objects and the obstacles identified in the scene image, a processor for executing a voice processing algorithm, an image processing algorithm, a path planning algorithm and a listening tactile information distribution algorithm, and a communication module for outputting the planned path direction, the remaining path distance and the guidance information for completing the grabbing action of the fetching hand to the tactile stimulation module.
In this embodiment, the auditory stimulation module 4 is realized by an air conduction earphone or a bone conduction earphone, and the earphone 7 is provided with a fixed connection ring 8 for preventing falling off, wherein: the air conduction earphone has the advantages of simple structure, good sound effect, low cost and low energy consumption, but can influence the normal receiving of external sound by a user and is easily interfered by external noise; the bone conduction earphone does not influence the normal receiving of external sound signals by a user, is comfortable to wear and is beneficial to the health of the auditory canal of the user, but has the problems of low tone quality, sound leakage, larger energy consumption and the like.
the earphone 7 carries out corresponding voice prompt for the user according to the audio information from the information processing module 3, wherein the voice prompt comprises the names, the directions, the relative distances, the shapes, the colors and the quantity information of the recognized target objects and obstacles and has a virtual three-dimensional stereo sound effect. For example, when the information processing module 3 determines that the object is located at a distance of 5 meters in the upper left front direction of the user, the user will hear "XXX (object name) at a distance of 5 meters in the upper left front direction" through the headphones 7, and the user will feel that the prompt voice is uttered at a distance of 5 meters in the upper left front direction thereof. Voice prompts are played to the user at intervals, such as every 20 seconds.
camera module 2, information processing module 3, microphone 6, earphone 7, fixed go-between 8 and power module 9 all set up on special glasses 10, the wholeness is better, easily grasps the specification parameter of each part, the optimization degree of system architecture is high, is equipped with the power module 9 that is used for the power supply on this special glasses 10.
the tactile stimulation form of the tactile stimulation module 5 adopts, but is not limited to: vibration stimulation, pressure stimulation, and temperature stimulation. The tactile stimulation module 5 specifically comprises the following components:
The tactile stimulation module 5 is provided with an elastic material 11, and the elastic material 11 is provided with a Bluetooth microcontroller 12, 10 vibrators representing different guiding directions and a power supply battery 13.
When vibration stimulation is adopted, the vibrator embedded in the elastic material 11 is used as a touch stimulation unit; when the pressure stimulation is adopted as the tactile stimulation, the pressure block embedded in the elastic material 11 is used as a tactile stimulation unit; when the electric stimulation is adopted as the tactile stimulation, the stimulation electrode embedded in the elastic material 11 is used as a tactile stimulation unit; when a hot and cold stimulus is used as the tactile stimulus, a hot and cold temperature block embedded in the elastic material 11 is used as the tactile stimulus unit.
The oscillator comprises: a front directional vibrator 14, a rear directional vibrator 15, and 8 horizontal directional vibrators 16; the corresponding touch stimulation module 5 controls the front-direction vibrator 14 or the rear-direction vibrator 15 to vibrate when the corresponding touch stimulation module needs to move to the front or rear area according to the planned path, and the vibrator corresponding to the direction of the target object in the 8 horizontal-direction vibrators 16 also vibrates; according to the planned path, when the target object is in the left/right area (without moving forwards or backwards), the front directional vibrator 15 and the rear directional vibrator 16 are controlled not to vibrate, and only the vibrator corresponding to the direction of the target object in the horizontal directional vibrator 16 vibrates; the intensity of the vibrator vibration depends on the estimated distance of the surplus path, and the smaller the distance, the greater the vibration intensity, and conversely, the smaller the vibration intensity.
for example, when the user should move to the upper right direction of the front area according to the planned path, the front vibrator 14 and the upper right vibrator 17 vibrate, and the strength of the vibration depends on the size of the estimated distance of the remaining path.
the embodiment relates to a working process of the system, which specifically comprises the following steps:
Step 1: when the voice recognition device is used for the first time, a voice sample of a user is input in a quiet environment through the microphone 6 through the voice collection module 1, the information processing module 3 carries out spectrum analysis on the voice sample, voice characteristics of the user are extracted, a user voice characteristic model is established, and the user voice characteristic model is stored in a voice characteristic model library;
Step 2: in the process of finding objects, the microphone 6 and the camera module 2 continuously collect external sounds and images of scenes around the user and send the external sounds and images to the information processing module 3; preferably, the camera module 1 has a wide-angle lens so as to acquire a wider range of scene image information;
And step 3: the information processing module 3 compares the external sound signal with the user voice feature model through a signal processing technology, and when the external sound signal is judged to be absent from the user voice feature model, the information processing module 3 does not generate any output information or instruction and continues to analyze and process the external sound signal; when the voice signal of the user exists in the information processing module 3, the information processing module extracts the voice signal, converts the voice signal into characters and generates the name of the target object and the characteristic description of the name of the target object through a semantic analysis program;
and 4, step 4: the information processing module 3 identifies a target object, marks a position and estimates depth information in a scene image according to the name of the target object and the characteristic description of the target object, the information processing module 3 judges whether the target object exists in the image through a target object existence judgment algorithm, when the target object does not exist in the image, a processing result without the target object in the image is generated and a prompt instruction is sent to the earphone 7, the earphone 7 prompts a user by voice that the target object is not found and the position is required to be adjusted, and then the step 3 is repeated; otherwise, entering step 5;
the target object existence judgment algorithm is used for judging the reason that the target object cannot be detected, and comprises the following steps: firstly, a target object does not appear in an image; secondly, when a target object appears in the image, the camera module 1 cannot acquire the image information of the target object due to the change of the camera view field or the movement of the target object; and thirdly, in the process of finding and fetching objects by the user, the object fetching hand shields the target object, so that the information processing module 3 cannot identify the target object.
the target object existence judgment algorithm specifically comprises the following steps:
1) judging whether a target object appears or not, if so, judging that the target object does not exist in the image and ending the judging program, wherein the judgment is of a first reason;
2) Acquiring the positions of the target objects in the first frames of images;
3) analyzing the change path of the position of the target object, if the target object is continuously moved from the image to the edge of the image and finally the target object is not detected, judging that the second reason is that the target object does not exist in the image, and ending the judgment program; and if not, the third reason is that the target object is still in the image, the position of the target object in the image detected last time is taken as the current position of the target object, subsequent analysis processing is carried out, after the target object is identified and detected again, the position information of the target object is updated, and the judgment program is ended.
and 5: the information processing module 3 marks the position of the target object in the image by using the rectangular frame, estimates the spatial distance of the target object, obtains the direction information of the target object relative to the camera module 1, converts the direction information into a corresponding prompt voice signal, performs virtual three-dimensional stereo modification according to the number of sound channels of the earphone 7, and sends the signal to the earphone 7;
the target object identification and position calibration are realized by adopting a target detection algorithm, and specifically comprise the following steps: the method is realized by adopting an algorithm based on a convolutional neural network, such as algorithms like YOLO, SSD, RCNN and the like; the convolutional neural network is trained by using a huge training image set, and with the increase of training times, the prediction accuracy of the convolutional neural network model on the existence and the position of a target in an image can reach an ideal state.
since the object finding by the user is a dynamic process, the information processing module 3 needs to track the position of the object after identifying the object and calibrating the position, and an object tracking program can be adopted.
the spatial estimation distance information is realized by an image depth information estimation algorithm, and specifically comprises the following steps: estimating the depth information of each pixel point in the image by adopting a monocular image depth information estimation algorithm based on a convolutional neural network and through a single-angle two-dimensional image; the training set can adopt image data sets with huge data size, pixel-level Depth information marks, semantic labels and other rich information, such as NYU Depth, KITTI and the like.
the virtual three-dimensional stereo processing is to design a virtual three-dimensional stereo model through physical or mathematical modeling, so that sound with spatial sense is played in each channel, and a listener can position a sound source in a simulated sound environment. When designing a virtual three-dimensional stereo model, the spatial auditory characteristic and auditory psychology characteristic of human ears are fully considered, and the binaural effect, the head-related transfer function, the reverberation filter and the like are reasonably utilized. Thus, the user will hear the prompting voice coming from the direction of the target object approximately, and the farther the target object is, the smaller the prompting voice is.
The virtual three-dimensional stereo model is based on a Head Related Transfer Function (HRTF), and the HRTF can be subjected to personalized processing according to human body parameters so as to obtain more real stereo feeling.
step 6: when the number of the prompted target objects is only one and the user sends out voice to confirm whether the object is the required target object, the information processing module 3 obtains the voice information of the user in the same way as the step 3, the step 7 is carried out only when the user confirms that the object identified by the information processing module 3 is the required target object, and the step 3 is returned when the number of the prompted target objects exceeds one, the user sends out voice to describe or select the target object more specifically or otherwise;
and 7: the information processing module 3 presets a target object distance threshold, after the position calibration, tracking and distance estimation are carried out on the target object, whether the distance between the target object and the camera module is greater than the target object distance threshold is judged, if so, the step 8 is carried out, and if not, the step 9 is carried out;
and 8: the information processing module 3 identifies, detects and estimates the distance of the obstacles in the image, plans a user fetching path, transmits the obtained target object and obstacle information and path guiding information to a listening touch information distribution program, transmits the processed corresponding information to the earphone 7 and the touch stimulation module 5 respectively, and then enters step 11;
and step 9: the information processing module judges whether the image has a fetching hand, and prompts the user to 'please stretch the hand to fetch' when the judgment result shows that the image does not have the fetching hand, and the step 4 is returned; otherwise, entering step 10;
Step 10: after the information processing module carries out position calibration, tracking and distance estimation on a target object and a user fetching hand, the information processing module carries out identification detection on obstacles in an image, plans a fetching path for the fetching hand of the user by combining the obtained direction and space distance information of the target object relative to the fetching hand, transmits the obtained target object, obstacle information and path guide information to an auditory tactile information distribution program, and transmits the processed corresponding information to an earphone 7 and a tactile stimulation module 5 respectively;
step 11: the auditory stimulation module gives a voice prompt to the user, and the tactile stimulation module gives tactile stimulation to the object-fetching hand of the user to cooperatively guide the user to seek and fetch objects; returning to the step 4; step 12: the information processing module analyzes and judges whether the user grabs the target object according to the relative position and the spatial distance relationship between the target object and the object fetching hand in the image, and when the user grabs the target object is judged, the blind assisting task is finished, the step 2 is returned, and the user is waited to send out the object searching voice again; otherwise, returning to the step 4.
the information processing module can also be realized by a mobile device 18, such as a mobile phone, a tablet computer, etc., the information processing capability of the system is better, the camera module 1, the microphone 3 and the earphone 4 on the special glasses 8 can all perform data transmission with the mobile device 18 in a wireless communication mode, and even can be directly replaced by a camera, a microphone and a stereo speaker of the mobile device 18.
in the above situation, the tactile stimulation module 5 may adopt another vibration strategy to reduce the number of required vibrators to 6, that is, only six vibrators with guiding directions of front, rear, left, right, upper and lower are required; after the information processing module 3 obtains the position of the target object through an image processing program and estimates the distance, a space rectangular coordinate system is established by taking the shooting position of the camera module 1 or the center of the fetching hand as an origin, and three axial directions of the coordinate system respectively correspond to three directions from bottom to top, from back to front and from left to right in an actual space; on the basis, a space vector pointing to the center of the target object from the origin of the coordinate system is constructed, orthogonal decomposition is carried out on the space vector in the upper three axial directions, and the decomposed vibrator is provided with a vibrator corresponding to the orthographic projection axial direction, namely the vibrator needing vibration; the vibration intensity depends on the length of the orthographic projection corresponding to the vibration intensity, and the longer the orthographic projection length is, the lower the vibration stimulus intensity is, and conversely, the higher the vibration intensity is.
for example, when the vehicle is to move to the upper right of the front area, orthogonal decomposition causes a certain length of orthogonal projection on the front half axis, the right half axis, and the upper half axis, the front vibrator 14, the rear vibrator 15, and the upper right vibrator 17 vibrate, and the intensity of the vibration depends on the length of the orthogonal projection. Based on the principle of orthogonal decomposition, different guiding directions are generated by utilizing combined vibration of the vibrators, and the number of the vibrators required by vibration stimulation guiding is reduced. However, this tactile stimulation strategy is slightly more complex than example 1, requiring longer pre-training and adaptation procedures by the user.
the foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Claims (10)
1. An intelligent blind-aiding system based on auditory and tactile guidance, comprising: sound collection module, camera module, information processing module, auditory stimulation module and tactile stimulation module, wherein: the sound collection module collects external sound signals and outputs the external sound signals to the information processing module, the camera module collects external scene image information and outputs the external scene image information to the information processing module, the information processing module analyzes and processes the external sound signals and the scene image information and respectively outputs processing results to the auditory stimulation module and the tactile stimulation module through a listening tactile information distribution algorithm, the auditory stimulation module outputs voice prompts according to the processing results, and the tactile stimulation module implements tactile stimulation according to the processing results;
the auditory tactile information distribution algorithm is as follows: outputting the names, the directions, the relative distances, the shapes, the colors and the numerical information of the target objects and the obstacles identified in the scene images to an auditory stimulation module for users to know the directions and the specific information of the target objects and the obstacles; outputting the planning path direction, the remaining path distance and the guiding information of the user for grabbing the target object by the fetching hand to the tactile stimulation module for guiding the user to safely approach the target object and completing the action of grabbing the target object;
The form of the tactile stimulation comprises: vibration stimulation, pressure stimulation, electrical stimulation, cold and hot stimulation.
2. the intelligent blind assisting system as claimed in claim 1, wherein the information processing module comprises: the device comprises a memory for storing the names, the directions, the relative distances, the shapes, the colors and the quantity information of the objects and the obstacles identified in the scene image, a processor for executing a voice processing algorithm, an image processing algorithm, a path planning algorithm and an auditory tactile information distribution algorithm, and a communication module for outputting the planned path direction, the remaining path distance and the guidance information for completing the grabbing action of the fetching hand to the tactile stimulation module.
3. the intelligent blind assistant system as claimed in claim 1, wherein the voice prompt contains the names, directions, relative distances, shapes, colors and numerical information of recognized objects and obstacles, and has a virtual three-dimensional stereo sound effect.
4. the intelligent blind assisting system as claimed in claim 1, wherein the camera module and the microphone as the sound collecting module send the collected scene images and the external sounds to the information processing module, and after the information processing module analyzes and processes the information from the camera module and the microphone, on one hand, the information processing module sends the prompt information to the earphone as the auditory stimulation module; on the other hand, the control instruction is sent to the touch stimulation module through Bluetooth; the camera module, the information processing module, the microphone, the earphone, the fixed connecting ring and the power supply module are all arranged on the special glasses.
5. The intelligent blind assisting system as claimed in claim 1, wherein the touch stimulation module is provided with an elastic material, and the elastic material is provided with a bluetooth microcontroller, touch stimulation units representing different guiding directions and a power supply battery; when vibration stimulation is adopted as tactile stimulation, the vibrator embedded with the elastic material is used as a tactile stimulation unit; when pressure stimulation is adopted as tactile stimulation, a pressure block embedded with elastic materials is used as a tactile stimulation unit; when the electric stimulation is used as the tactile stimulation, the stimulation electrode embedded in the elastic material is used as a tactile stimulation unit; when the cold and hot stimulation is adopted as the tactile stimulation, the cold and hot temperature block embedded with the elastic material is used as the tactile stimulation unit.
6. The intelligent blind assistant system of claim 1, wherein the tactile stimulation module comprises: a front tactile stimulation unit, a rear tactile stimulation unit and each horizontal tactile stimulation unit; the corresponding touch stimulation module controls the front direction touch stimulation unit or the rear direction touch stimulation unit to vibrate when the corresponding touch stimulation module needs to move to the front area or the rear area according to the planned path, and in each horizontal direction vibrator, the touch stimulation unit corresponding to the direction of the target object also vibrates; according to the planned path, when the target object is in the left/right area, controlling the front tactile stimulation unit and the rear tactile stimulation unit not to vibrate, and only the vibrator corresponding to the direction of the target object in the vibrators in the horizontal direction vibrates; the intensity of the vibrator vibration depends on the estimated remnant path distance, and the smaller the distance, the greater the vibration intensity, and vice versa.
7. A control method based on the system of any preceding claim, comprising the steps of:
Step 1: when the voice recognition device is used for the first time, a voice sample of a user is recorded in a quiet environment through the voice collection module, and after voice characteristics of the user are extracted and analyzed through the information processing module, a user voice characteristic model is established and stored in a voice characteristic model library;
step 2: in the object searching process, the sound acquisition module and the camera module continuously acquire external sound and images of scenes around the user and send the external sound and the images to the information processing module;
And step 3: the information processing module compares the external sound signal with the user voice feature model, and when the external sound signal is judged to be absent, the information processing module does not generate any output information or instruction and continues to analyze and process the external sound signal; when the voice signal of the user exists, the information processing module extracts the voice signal, converts the voice signal into characters and generates the name of the target object and the feature description of the name;
And 4, step 4: the information processing module identifies, detects and estimates depth information of a target object in a scene image according to the name of the target object and the characteristic description of the target object, judges whether the target object exists in the image through a target object existence judgment algorithm, generates a processing result without the target object in the image and sends a prompt instruction to the auditory stimulation module when the information processing module judges that the target object does not exist in the image, the auditory stimulation module informs a user that the target object is not found and asks the user to adjust the direction in a voice prompt mode, and then the step 3 is repeated; otherwise, entering step 5;
And 5: the information processing module converts the direction and space estimated distance information of the target relative to the camera module into corresponding prompt voice signals, performs virtual three-dimensional stereo modification processing and sends the prompt voice signals to the auditory stimulation module;
Step 6: the user receives the corresponding voice prompt, when the number of the prompted target objects is only one, the user sends voice to confirm whether the target objects are the required target objects, the information processing module obtains the voice information of the user according to the method in the step 3, if the user confirms that the object identified by the information processing module is the required target objects, the step 7 is carried out, and if not, the step 3 is returned; when the number of the prompted target objects exceeds one, the user sends out voice to describe or select the target objects more specifically, and the step 3 is returned;
And 7: the information processing module presets a target object distance threshold, after the position calibration, tracking and distance estimation are carried out on the target object, whether the distance between the target object and the camera module is greater than the target object distance threshold or not is judged, if so, the step 8 is carried out, and if not, the step 9 is carried out;
and 8: the information processing module identifies and detects the obstacles in the image, plans a user fetching path, transmits the obtained target object and obstacle information and path guiding information to the auditory sense information distribution program, transmits the processed corresponding information to the auditory sense stimulation module and the tactile sense stimulation module respectively, and enters step 11
And step 9: the information processing module judges whether an object taking hand exists in the image, and when the judgment result is that the object taking hand does not exist in the image, the auditory stimulation module prompts the user to 'please stretch the hand to take the object', and the step 4 is returned; otherwise, entering step 10;
step 10: the information processing module identifies and detects obstacles in the image after carrying out position calibration, tracking and distance estimation on a target object and a user fetching hand, plans a fetching path for the fetching hand of the user by combining the obtained direction and space distance information of the target object relative to the fetching hand, transmits the obtained target object, obstacle information and path guide information to an auditory sense information distribution program, and transmits the processed corresponding information to the auditory sense stimulation module and the tactile sense stimulation module respectively;
Step 11: the auditory stimulation module gives a voice prompt to the user, and the tactile stimulation module gives tactile stimulation to the object-fetching hand of the user to cooperatively guide the user to seek and fetch objects; returning to the step 4;
Step 12: the information processing module analyzes and judges whether the user grabs the target object according to the relative position and the spatial distance relationship between the target object and the object fetching hand in the image, and when the user grabs the target object is judged, the blind assisting task is finished, the step 2 is returned, and the user is waited to send out the object searching voice again; otherwise, returning to the step 4.
8. the control method according to claim 7, wherein the target presence determination algorithm is for determining a cause of the target being undetectable, and includes: firstly, a target object does not appear in an image; secondly, when a target object appears in the image, the camera module 1 cannot acquire the image information of the target object due to the change of the camera view field or the movement of the target object; and thirdly, in the process of finding and fetching objects by a user, the object fetching hand shields the target object, so that the information processing module cannot identify the target object.
9. The control method according to claim 7 or 8, wherein the target object presence determination algorithm specifically includes the steps of:
1) Judging whether a target object appears or not, if so, judging that the target object does not exist in the image and ending the judging program, wherein the judgment is of a first reason;
2) acquiring the positions of the target objects in the first frames of images;
2) Analyzing the change path of the position of the target object, if the target object is continuously moved from the image to the edge of the image and finally the target object is not detected, judging that the second reason is that the target object does not exist in the image, and ending the judgment program; and if not, the third reason is that the target object is still in the image, the position of the target object in the image detected last time is taken as the current position of the target object, subsequent analysis processing is carried out, after the target object is identified and detected again, the position information of the target object is updated, and the judgment program is ended.
10. The control method according to claim 7, wherein the target object identification and the position calibration are realized by a target detection algorithm, which specifically comprises: adopting a YOLO, SSD and RCNN algorithm based on a convolutional neural network; the position tracking of the target object is realized through a target tracking algorithm, and specifically comprises the following steps: adopting KCF, CSK and other traditional tracking algorithms, or adopting GOTURN, Siamese RPN and other tracking algorithms based on a convolutional neural network; the spatial estimation distance information is realized by an image depth information estimation algorithm, and specifically comprises the following steps: estimating the depth information of each pixel point in the image by adopting a monocular image depth information estimation algorithm based on a convolutional neural network and through a single-angle two-dimensional image; the training set can adopt image data sets with huge data size, pixel-level Depth information marks, semantic labels and other rich information, such as NYU Depth, KITTI and the like.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910793409.6A CN110559127A (en) | 2019-08-27 | 2019-08-27 | intelligent blind assisting system and method based on auditory sense and tactile sense guide |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910793409.6A CN110559127A (en) | 2019-08-27 | 2019-08-27 | intelligent blind assisting system and method based on auditory sense and tactile sense guide |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110559127A true CN110559127A (en) | 2019-12-13 |
Family
ID=68776169
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910793409.6A Pending CN110559127A (en) | 2019-08-27 | 2019-08-27 | intelligent blind assisting system and method based on auditory sense and tactile sense guide |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110559127A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111109786A (en) * | 2019-12-25 | 2020-05-08 | 李嘉伦 | Intelligent obstacle early warning crutch based on deep learning and early warning method thereof |
| RU2727558C1 (en) * | 2019-12-25 | 2020-07-22 | Федеральное государственное бюджетное образовательное учреждение высшего образования "Сибирский государственный университет геосистем и технологий" | Method for creating an audio-tactile source of cartographic information using digital information and nanotechnologies and using it in an active mode by a blind or visually impaired people |
| CN111759075A (en) * | 2020-08-07 | 2020-10-13 | 闫然 | Student's safety trip schoolbag |
| CN112515928A (en) * | 2020-11-26 | 2021-03-19 | 苏州中科先进技术研究院有限公司 | Intelligent blind assisting system, method, computer equipment and storage medium |
| CN113196390A (en) * | 2021-03-09 | 2021-07-30 | 曹庆恒 | Perception system based on hearing and use method thereof |
| CN113180894A (en) * | 2021-04-27 | 2021-07-30 | 浙江大学 | Visual intelligence-based hand-eye coordination method and device for multiple-obstacle person |
| CN113191333A (en) * | 2021-05-29 | 2021-07-30 | 山东新一代信息产业技术研究院有限公司 | Blind guiding method, system and blind guiding equipment based on artificial intelligence |
| CN113526348A (en) * | 2021-06-30 | 2021-10-22 | 中国人民解放军空军工程大学 | Lifting appliance system with auditory and tactile prompt functions and use method thereof |
| CN113840588A (en) * | 2021-08-15 | 2021-12-24 | 曹庆恒 | Touch sensing system and use method thereof |
| CN114257920A (en) * | 2022-02-25 | 2022-03-29 | 荣耀终端有限公司 | Audio playing method and system and electronic equipment |
| CN114652513A (en) * | 2022-03-16 | 2022-06-24 | 陈智谋 | Touch technology |
| CN115356736A (en) * | 2022-07-01 | 2022-11-18 | 德州学院 | Unrestrained animal tracking system based on thermal imaging and sound wave technology |
| CN115907804A (en) * | 2023-03-03 | 2023-04-04 | 腾讯科技（深圳）有限公司 | Business processing method and device, computer equipment and storage medium |
| CN117001715A (en) * | 2023-08-30 | 2023-11-07 | 哈尔滨工业大学 | Intelligent auxiliary system and method for visually impaired people |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002065721A (en) * | 2000-08-29 | 2002-03-05 | Komatsu Ltd | Device and method for supporting environmental recognition for visually handicapped |
| CN102217990A (en) * | 2011-04-27 | 2011-10-19 | 南京航空航天大学 | Environment information sensor based on vibration touch |
| CN102609724A (en) * | 2012-02-16 | 2012-07-25 | 北京航空航天大学 | Method for prompting ambient environment information by using two cameras |
| JP2015092887A (en) * | 2013-11-08 | 2015-05-18 | セイコーエプソン株式会社 | Object recognition device |
| CN105640748A (en) * | 2016-03-16 | 2016-06-08 | 宁波市江东精诚自动化设备有限公司 | Vibration blind-guiding clothing |
| CN106038183A (en) * | 2016-06-29 | 2016-10-26 | 冯伟林 | Wearable device and navigation system for blind people |
| CN107007437A (en) * | 2017-03-31 | 2017-08-04 | 北京邮电大学 | Interactive blind person's householder method and equipment |
| CN107278301A (en) * | 2016-12-30 | 2017-10-20 | 深圳前海达闼云端智能科技有限公司 | A kind of method and device for aiding in user to look for something |
| CN109645633A (en) * | 2019-01-21 | 2019-04-19 | 珠海格力电器股份有限公司 | A kind of walking stick, air navigation aid, device |
-
2019
- 2019-08-27 CN CN201910793409.6A patent/CN110559127A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002065721A (en) * | 2000-08-29 | 2002-03-05 | Komatsu Ltd | Device and method for supporting environmental recognition for visually handicapped |
| CN102217990A (en) * | 2011-04-27 | 2011-10-19 | 南京航空航天大学 | Environment information sensor based on vibration touch |
| CN102609724A (en) * | 2012-02-16 | 2012-07-25 | 北京航空航天大学 | Method for prompting ambient environment information by using two cameras |
| JP2015092887A (en) * | 2013-11-08 | 2015-05-18 | セイコーエプソン株式会社 | Object recognition device |
| CN105640748A (en) * | 2016-03-16 | 2016-06-08 | 宁波市江东精诚自动化设备有限公司 | Vibration blind-guiding clothing |
| CN106038183A (en) * | 2016-06-29 | 2016-10-26 | 冯伟林 | Wearable device and navigation system for blind people |
| CN107278301A (en) * | 2016-12-30 | 2017-10-20 | 深圳前海达闼云端智能科技有限公司 | A kind of method and device for aiding in user to look for something |
| CN107007437A (en) * | 2017-03-31 | 2017-08-04 | 北京邮电大学 | Interactive blind person's householder method and equipment |
| CN109645633A (en) * | 2019-01-21 | 2019-04-19 | 珠海格力电器股份有限公司 | A kind of walking stick, air navigation aid, device |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2727558C1 (en) * | 2019-12-25 | 2020-07-22 | Федеральное государственное бюджетное образовательное учреждение высшего образования "Сибирский государственный университет геосистем и технологий" | Method for creating an audio-tactile source of cartographic information using digital information and nanotechnologies and using it in an active mode by a blind or visually impaired people |
| CN111109786A (en) * | 2019-12-25 | 2020-05-08 | 李嘉伦 | Intelligent obstacle early warning crutch based on deep learning and early warning method thereof |
| CN111759075A (en) * | 2020-08-07 | 2020-10-13 | 闫然 | Student's safety trip schoolbag |
| CN112515928A (en) * | 2020-11-26 | 2021-03-19 | 苏州中科先进技术研究院有限公司 | Intelligent blind assisting system, method, computer equipment and storage medium |
| WO2022111443A1 (en) * | 2020-11-26 | 2022-06-02 | 苏州中科先进技术研究院有限公司 | Intelligent blind assisting system and method, computer device, and storage medium |
| CN113196390B (en) * | 2021-03-09 | 2024-04-05 | 曹庆恒 | Auditory sense system and application method thereof |
| CN113196390A (en) * | 2021-03-09 | 2021-07-30 | 曹庆恒 | Perception system based on hearing and use method thereof |
| WO2022188022A1 (en) * | 2021-03-09 | 2022-09-15 | 曹庆恒 | Hearing-based perception system and method for using same |
| CN113180894B (en) * | 2021-04-27 | 2022-02-11 | 浙江大学 | Visual intelligence-based hand-eye coordination method and device for multiple-obstacle person |
| CN113180894A (en) * | 2021-04-27 | 2021-07-30 | 浙江大学 | Visual intelligence-based hand-eye coordination method and device for multiple-obstacle person |
| CN113191333A (en) * | 2021-05-29 | 2021-07-30 | 山东新一代信息产业技术研究院有限公司 | Blind guiding method, system and blind guiding equipment based on artificial intelligence |
| CN113526348A (en) * | 2021-06-30 | 2021-10-22 | 中国人民解放军空军工程大学 | Lifting appliance system with auditory and tactile prompt functions and use method thereof |
| CN113526348B (en) * | 2021-06-30 | 2024-01-30 | 中国人民解放军空军工程大学 | Lifting appliance system with hearing and touch prompt functions and application method thereof |
| WO2023019376A1 (en) * | 2021-08-15 | 2023-02-23 | 曹庆恒 | Tactile sensing system and method for using same |
| CN113840588A (en) * | 2021-08-15 | 2021-12-24 | 曹庆恒 | Touch sensing system and use method thereof |
| CN114257920A (en) * | 2022-02-25 | 2022-03-29 | 荣耀终端有限公司 | Audio playing method and system and electronic equipment |
| CN114652513A (en) * | 2022-03-16 | 2022-06-24 | 陈智谋 | Touch technology |
| CN115356736A (en) * | 2022-07-01 | 2022-11-18 | 德州学院 | Unrestrained animal tracking system based on thermal imaging and sound wave technology |
| CN115907804A (en) * | 2023-03-03 | 2023-04-04 | 腾讯科技（深圳）有限公司 | Business processing method and device, computer equipment and storage medium |
| CN117001715A (en) * | 2023-08-30 | 2023-11-07 | 哈尔滨工业大学 | Intelligent auxiliary system and method for visually impaired people |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110559127A (en) | intelligent blind assisting system and method based on auditory sense and tactile sense guide | |
| US11523218B2 (en) | Multifunctional earphone system for sports activities | |
| US20240121548A1 (en) | Stand-alone multifunctional earphone for sports activities | |
| US11037561B2 (en) | Method and apparatus for voice interaction control of smart device | |
| US20240171930A1 (en) | User Experience Localizing Binaural Sound During a Telephone Call | |
| CN109410285B (en) | Calibration method, calibration device, terminal equipment and storage medium | |
| US10015579B2 (en) | Audio accelerometric feedback through bilateral ear worn device system and method | |
| CN107278301B (en) | Method and device for assisting user in finding object | |
| CN113196390B (en) | Auditory sense system and application method thereof | |
| US20210337300A1 (en) | Adjustment mechanism for tissue transducer | |
| CN109272473B (en) | Image processing method and mobile terminal | |
| CN109348359B (en) | Sound equipment and sound effect adjusting method, device, equipment and medium thereof | |
| McLachlan et al. | Towards modelling active sound localisation based on Bayesian inference in a static environment | |
| KR20220066911A (en) | Method and system for adjusting the level of tactile content when providing audio content | |
| JP2022542755A (en) | Method and system for selecting a subset of acoustic sensors of a sensor array | |
| CN111437157A (en) | Intelligent wearable communication equipment and use method thereof | |
| CN112104965B (en) | Sound amplification method and sound amplification system | |
| CN112104962B (en) | Following type robot sound amplification method and sound amplification system based on image recognition | |
| JP2020126195A (en) | Voice interactive device, control device for voice interactive device and control program | |
| CN111752522A (en) | Accelerometer-based selection of audio sources for hearing devices | |
| CN112368768A (en) | Information processing apparatus, information processing method, and acoustic system | |
| WO2019119290A1 (en) | Method and apparatus for determining prompt information, and electronic device and computer program product | |
| Bellotto | A multimodal smartphone interface for active perception by visually impaired | |
| CN112104964B (en) | Control method and control system of following type sound amplification robot | |
| CN113050917A (en) | Intelligent blind-aiding glasses system capable of sensing environment three-dimensionally |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |
|
| RJ01 | Rejection of invention patent application after publication |