WO2022188022A1

WO2022188022A1 - Hearing-based perception system and method for using same

Info

Publication number: WO2022188022A1
Application number: PCT/CN2021/079689
Authority: WO
Inventors: 曹庆恒
Original assignee: 曹庆恒
Priority date: 2021-03-09
Filing date: 2021-03-09
Publication date: 2022-09-15
Also published as: CN113196390B; CN113196390A

Abstract

A hearing-based perception system and method for using same. The system comprises: a user interaction module (10), an information acquisition module (20), and an analysis and processing module (30). The method comprises: receiving a user instruction (S1); acquiring information for analysis and processing in combination with the instruction (S2); performing analysis and calculation according to the instruction and the information, executing the instruction and/or obtaining feedback information (S3); and if there is feedback information, converting the feedback information into an auditory perception signal and feeding same back to a user (S4). The hearing-based perception system and the method for using same can help people better use hearing to perceive information from the outside world, enhances the perception effect, and can increase the efficiency of walking, looking for an object, using a computer and smart device/smart system, etc. when a user is in conditions such as low light, poor light, too strong light, amblyopia, myopia, hyperopia, presbyopia, and eye fatigue, or when it is inconvenient for the user to look carefully, such as during driving.

Description

An auditory-based perception system and method of using the same

technical field

The present invention relates to the technical field of information and communication, in particular to an auditory-based perception system and a method for using the same.

Background technique

Hearing is the most important way for humans to perceive external information besides vision. After hearing the sound waves, the human auditory system uses the information in it to perceive the sound source, information, space, location and environment. Especially for blind people, because they cannot use vision to perceive external information, the use of hearing to perceive external information has become an important way for them to obtain information.

Therefore, there is an urgent need for a perception system based on hearing, which can help people better use hearing to perceive external information and enhance the effect and efficiency of perception.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a perception system based on hearing and a method of using the same, which can help people better use hearing to perceive external information, enhance the effect of perception, and can help blind people or normal people in low light conditions. Improve the efficiency of walking, finding objects, using computers, and smart devices/smart systems in an environment with

In order to achieve the above object, the present invention provides an auditory-based perception system, the system includes: a user interaction module, an information acquisition module and an analysis and processing module,

The user interaction module is used for receiving an instruction and feeding back the feedback information to the user as an auditory perception signal;

The information acquisition module is used to acquire information, and the information is used for analysis and processing by the analysis processing module in combination with the instruction;

The analysis and processing module is configured to perform analysis and calculation according to the instruction and the information, execute the instruction and/or obtain feedback information.

In the above-mentioned auditory-based perception system, the conversion of feedback information into auditory perception signals is completed by the user interaction module or the analysis processing module.

In the above-mentioned auditory-based perception system, the auditory perception signal represents information through at least one of the frequency, rhythm, melody, interval, orientation, distance, size, height, length, and timbre of the sound.

In the auditory-based perception system as described above, the auditory perception signal includes a speech signal.

In the above-mentioned hearing-based perception system, the user interaction module includes an instruction acquisition module and an auditory perception signal output module, and the auditory perception signal output module includes at least one of an earphone, a bone conduction earphone, a speaker, a hearing aid, and a brain-computer interface. one.

In the above-mentioned hearing-based perception system, the instruction acquisition module includes a voice recognition device, a sound recognition device, a gesture recognition device, a body motion recognition device, an expression recognition device, a body signal recognition device, a smart wearable device, a smart tablet, At least one of the mobile phone, mouse, keyboard, smart handle, smart cane, smart ring, and smart bracelet.

In the above-mentioned hearing-based perception system, the information acquisition module includes an image sensor, a radar device, a radio frequency identification device, a positioning device, an audio acquisition device, an infrared device, an ultraviolet device, a laser scanner, a metal detector, a temperature sensor device, light sensing device, touch sensing device, air pressure sensor, water pressure sensor, olfactory recognition device, magnetic field detection device, wind detection device, humidity detection device, electric power detection device, speed detection device, altitude detection device, chemical analysis device, radiation detection at least one of the devices.

With the above-mentioned auditory-based perception system, the system can also feed back information to the user through non-auditory signals, or feed back information to the user through a combination of auditory signals and non-auditory signals.

The above-mentioned auditory-based perception system, the auditory-based perception system further includes a data transmission module, and the data transmission module converts the instruction accepted by the user interaction module and the information acquired by the information acquisition module or analyzed. The instructions and/or information processed by the processing module are sent to the network/system/server/smart device, and the server/network/system/smart device performs analysis and calculation according to the instruction and the information, executes the instruction and/or converts the The result is transmitted to the data transmission module.

In the above-mentioned auditory-based perception system, the auditory-based perception system can also obtain information for analysis and processing in combination with instructions from the network/system/server/smart device through the data transmission module or the information acquisition module.

The auditory-based perception system as described above, which is used for assisting walking, assisting movement, sports training, navigation, assisting driving, assisting parking, positioning, location guidance, finding targets, reflecting pictures, reflecting objects, Detection, reconnaissance, exploration, design, maintenance, equipment use, device use, learning, teaching, shopping, office, social, games, entertainment, film and television, computer, health testing, disease diagnosis, surgical treatment, virtual concerts, virtual reality technology at least one of.

The above-mentioned auditory-based perception system, as an auditory-based operating system, can be used alone or in combination with other systems to operate computers, artificial intelligence, smart devices, and virtual reality devices.

The present invention also provides a method for using an auditory-based perception system, the method comprising:

receive user instructions;

Obtain information for analysis and processing combined with instructions;

Perform analysis and calculation according to the instruction and the information, execute the instruction and/or obtain feedback information;

If there is feedback information, the feedback information is converted into an auditory perception signal and fed back to the user.

An auditory-based perception system and its using method of the present invention include: a user interaction module, an information acquisition module and an analysis and processing module. The method includes: receiving a user instruction; acquiring information for analyzing and processing the instruction; performing analysis and calculation according to the instruction and the information, executing the instruction and/or obtaining feedback information; if there is feedback information , which converts the feedback information into auditory perception signals and feeds them back to the user. The auditory-based perception system and its using method of the present invention can help people better use their hearing to perceive external information, enhance the effect of perception, and can help blind people or normal people when there is insufficient light or poor light. , too strong light, amblyopia, myopia, hyperopia, presbyopia, eye fatigue and other environments, or when it is inconvenient to watch carefully, such as when driving, to improve the efficiency of walking, finding objects, using computers, and smart devices/intelligent systems.

Description of drawings

FIG. 1 is a schematic diagram of an auditory-based perception system according to the first embodiment of the present invention.

FIG. 2 is a method flowchart of a method for using an auditory-based perception system according to a second embodiment of the present invention.

Detailed ways

In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings and embodiments.

Refer to FIG. 1 for the first embodiment of the present invention. FIG. 1 is a schematic diagram of an auditory-based perception system according to the first embodiment of the present invention. As shown in the figure, the auditory-based perception system of the present invention includes: a user interaction module 10 , an information acquisition module 20 and an analysis processing module 30 .

The user interaction module 10 is used to receive the instruction and feedback the feedback information to the user as an auditory perception signal; the information acquisition module 20 is used to acquire information, and the information is used for the analysis processing module 30 to analyze and process the instruction; the analysis processing module 30 It is used for analyzing and calculating according to the instruction and the information, executing the instruction and/or obtaining feedback information.

The working flow of the auditory-based perception system of the present invention is:

First, the user interaction module 10 receives user instructions. In the present invention, the user interaction module 10 includes an instruction acquisition module and an auditory perception signal output module. The instruction obtaining module is used to obtain the instruction issued by the user. Users can convert relevant information through voice, gestures, movements, expressions, body signals such as body temperature, heartbeat, blood pressure, breathing, etc. as instructions, or operate tablet computers, mobile phones, mice, keyboards, smart handles, smart canes, smart canes, etc. Wearable devices, etc., or other suitable ways to issue instructions, therefore, the instruction acquisition module may include a voice recognition device, a voice recognition device, a gesture recognition device, a body motion recognition device, an expression recognition device, a body signal recognition device, smart wearable devices, One of a smart tablet, a mobile phone, a mouse, a keyboard, a smart handle, a smart walking stick, a smart finger ring, and a smart wristband, and may also include other devices suitable for receiving user instructions. Commands can also be issued at timed, periodically, or when certain conditions are triggered, depending on the settings. The auditory perception signal output module is used to feed back information to the user through the auditory perception signal. The auditory perception signal output module may include at least one of earphones, bone conduction earphones, speakers, hearing aids, and brain-computer interfaces, or other suitable devices. or a combination of related devices. The auditory-based perception system of the present invention can also feed back information to the user through non-auditory signals, for example, it can feed back information to the user through a blind tablet, an intelligent blind handle, an intelligent blind mouse, etc., or through a combination of auditory signals and non-auditory signals. Feedback information to users.

In the present invention, the auditory perception signal refers to representing information through at least one of the characteristics of sound frequency, rhythm, melody, interval, orientation, distance, size, height, length, and timbre, etc., so that the user can use the signal to perceptual information. For example, spatial orientation information can be conveyed to the user through sound. Since the transmission characteristics of sound waves transmitted by a sound source to a specific orientation can be expressed as a function data set, this function data set representing the sound wave transmission characteristics can be used to process audio signals, so that the audio signals reflect the sound waves transmitted by the sound source to The transmission characteristics of this azimuth. When the processed audio signal is converted into sound through the playback device, the sound shows the transmission characteristics of the sound source transmitting the sound wave to the azimuth, so that the user can feel the virtual sound source spatial azimuth. The specific orientation may include the direction, location, height, and the like of the sound.

Next, it is necessary to obtain information for analyzing and processing in conjunction with the instruction. Some information may be used in the analysis and processing of user instructions. For example, if the user's instruction is for a blind person to walk to a certain place, then, first of all, it is necessary to obtain the user's current position and destination position, and also need to obtain the obstacle information on the road. With Only after sufficient information is available can appropriate feedback be obtained for the analysis and processing of user instructions. In the hearing-based perception system of the present invention, the information acquisition module 20 may include an image sensor, a radar device, a radio frequency identification device, a positioning device, an audio acquisition device, an infrared device, an ultraviolet device, a laser scanner, a metal detector, and a temperature sensing device. , light sensing device, touch sensing device, air pressure sensor, water pressure sensor, olfactory recognition device, magnetic field detection device, wind detection device, humidity detection device, power detection device, speed detection device, altitude detection device, chemical analysis device, radiation detection device At least one of them may also be used by other suitable devices to obtain relevant information.

After that, the analysis and processing module 30 performs analysis and calculation according to the instruction and the information, executes the instruction and/or obtains feedback information. For example: a three-dimensional space model is established for the blind walking instruction and the obtained information such as the user's current position, destination position, and obstacles. The space model may also include information related to the time dimension. In the present invention, the model may be a mapping entity A model of a scene/object; it can also be a model that maps a virtual scene/object, such as an operating system, an operation interface, a game, a virtual system, etc.; it can also be a combination of entity and virtual. After the model is established, the relevant objects instructed by the user are substituted into the model to plan a travel route suitable for the blind, and in the process of traveling for the blind, a prompt is given according to the actual situation or the user is guided to follow the planned route.

Finally, if there is feedback information, the feedback information is converted into an auditory perception signal, which is then fed back to the user by the user interaction module 10 . Converting the feedback information into auditory perception signals is completed by the user interaction module 10 or the analysis processing module 30, and may also be completed by other devices or devices. For example, the route of the blind person walking and the guidance and reminders given according to the actual situation are converted into auditory perception signals and fed back to the user. When the user needs to be guided to go to a certain position, an auditory perception signal that the sound source is at the target position can be sent out, so that the user can perceive the location information through the auditory perception signal, so as to walk to the location. During the actual walking process of the user, the auditory perception signal is adjusted as the distance between the user and the target position changes, so that the user can walk to the position. It can be that every time the user moves the position, according to the change of the user's current position and the target position, the sound source sends out an auditory perception signal at the target position that is transmitted to the user's current position, so that the user can continuously perceive the target position during the movement process. Finally successfully reached the target position. If there are obstacles in the route, you can use other sound signals with different frequencies, rhythm, melody, interval, bearing, distance, size, height, length and timbre to send out the auditory perception of information such as obstacle location, distance, height, and danger. signal, so that the user can perceive the position, distance, height, danger and other information of the obstacle through the auditory perception signal, so as to bypass the obstacle.

The meaning of sound signals and their combinations of different frequencies, rhythms, melody, intervals, orientations, distances, sizes, heights, lengths and timbres used to represent destinations and obstacles, targets, objects, content, and their combinations can be preset, And it is a signal that the user has been trained to distinguish and understand the meaning of, and it can also be an existing sound signal that can contain information, such as a speech signal of an existing language or other regular sound signals. For example: an auditory-based perception system can set and/or train the definitions of sound signals of different frequencies, rhythms, melody, intervals, orientations, distances, sizes, pitches, lengths, and timbres, such as object definition, target definition, orientation Definition, definition of distance, definition of color, definition of temperature, definition of high and low, definition of warning/danger signal, definition of operation signal, definition of operation result, etc. After setting or training, after hearing the sound signal set as a representative meaning, the user can quickly realize the meaning represented by the sound signal or a combination of sound signals. The auditory-based perception system of the present invention can also feed back information to the user by combining the auditory perception signal with other signals, for example, the auditory perception signal can be combined with the Braille signal to feed back information.

The auditory-based perception system of the present invention can be used for various purposes, such as walking, finding objects, games, computers, virtual concerts, virtual reality technology, etc. In the case of amblyopia, myopia, hyperopia, presbyopia, eye fatigue, etc., or when it is inconvenient to watch carefully, such as driving, you can easily use the computer based on the auditory perception system to paint, play, compose, write, work, and study. Etc., enhance the importance of hearing in these fields, enhance the effect of people's use of auditory perception information, so that humans, especially the blind, can live more conveniently.

The following will take the hearing-based perception system of the present invention to help blind people find objects as an example for specific description.

Step 1: The user issues an instruction to find the object. The user may issue an instruction through voice, and the system acquires the user's voice instruction through the microphone worn on the user and recognizes the instruction. Users can also issue commands through gestures, actions, or other means. At this time, the user's instruction can be obtained through a related device such as a camera.

Step 2: After the system obtains the user instruction, it obtains the relevant information through the information obtaining module 20 . First, find the item you are looking for, and determine the location of the item; second, determine the current location of the user; and then obtain the surrounding environment information.

The image data in the space can be obtained from different angles through multiple image sensors for spatial modeling and position calculation. The image sensor can be set in a suitable position in the room, or can be worn on the user's body, for example: the image can be The sensor, the microphone for obtaining instructions, and the earphone for feeding back the auditory perception signal to the user are integrated into a head-worn portable device, which is worn on the user's head.

It is also possible to obtain position information in space through a positioning device, or obtain information such as the position and size of each item through the Internet of Things.

Step 3: Perform analysis and calculation according to the instruction and the information to obtain feedback information. First, a spatial model is established through the acquired information, mainly through multi-angle spatial image data, as well as the position information and related size of the objects in the space. It is also possible to obtain the established spatial model through the network or other means, or to modify the existing spatial model to obtain a new spatial model that conforms to the actual situation. After that, use the space model and the location information of users and items to plan a suitable fetching path.

It is also possible to plan a method for getting the item according to the space/equipment/furniture/device where the item is stored, and the finding and retrieving the item can also be implemented in steps. For example: 1, go to the door of the storage room; 2, open the door to enter the storage room; 3, go to the locker; 4, open the second cabinet door; 5, get the item you are looking for.

Then, the acoustic model is established according to the spatial model, and the correlation function of the acoustic wave transmission is obtained, which is used to calculate the auditory perception signal. To establish the acoustic model, the beam tracking algorithm can be used to establish the acoustic model, calculate the intersection of the relevant beam and the space, and obtain the correlation function of the sound wave transmission. When the audio signal processed by this correlation function is converted into sound through the playback device, the sound is The transmission characteristics of the sound wave transmitted by the sound source to the azimuth are displayed, so that the user can feel the virtual sound source spatial azimuth.

In the fourth step, the feedback information is converted into auditory perception signals and fed back to the user. A virtual sound source is produced at the location of the item, and after processing the correlation function of sound wave transmission, the auditory perception signal is obtained, which is output to the user through the earphone. If the planned fetching path is not a straight path, the path can be divided into multiple straight segments, and then, a sound source is virtualized at the end point of the first straight segment, and the auditory perception signal is calculated and fed back to the user. After the end point of each straight line segment, enter the second straight line segment. At this time, a sound source is virtualized at the end point of the second straight line segment, and the auditory perception signal is calculated and fed back to the user to guide the user to the point, and so on until reaching the The location of the item being sought. According to the user's moving route and planned route, direction, height deviation, and time limit/urgency, the corresponding auditory perception signal can be set and fed back to the user, so that the user can be continuously corrected and reminded.

The auditory-based perception system of the present invention can also be used for virtual concerts. First, the scene of the virtual concert venue, the acoustic model is established. Then, each instrument, part, etc. is virtualized into different sound sources, and these sound sources can be located in different positions. The correlation functions of the sound wave transmission from different sound sources to the user's location are calculated separately. The music generated by each virtual sound source is processed by the correlation function of sound wave transmission and then superimposed to obtain the final auditory perception signal of the concert and output to the user.

In the present invention, the correlation function for calculating acoustic wave transmission may be a set of head related transfer function data (Head Related Transfer Function, HRTF), a set of interaural time difference data (Interaural Time Difference, ITD), and a set of interaural intensity difference data (IID) Any appropriate set of data that can characterize the transmission characteristics of sound waves emitted by a sound source to a certain azimuth. ITD refers to the time difference between the sound signal reaching both ears due to the distance difference between the sound source and the left and right ears. IID refers to the difference in intensity of the acoustic signal when it reaches both ears due to the difference in the distance between the sound source and the left and right ears. Both ITD and IID are functions of sound source location and sound wave frequency. When the sound source localization data set is the ITD and IID data sets, the user can distinguish whether the sound source is located on its left or right. HRTF is the acoustic transfer function from the sound source to both ears in the free field, which is used to describe the characteristic changes that occur when the sound wave emitted by the sound source in the free sound field is incident at a certain point in the ear canal at a certain angle. HRTF is a function of the location of the sound source, the frequency of the sound wave, and the shape and properties of the body surface. The unit impulse response from the sound source to the anthropometric point is called the Head Related Impulse Response (HRIR). HRTF is the Fourier transform of HRIR. When the sound source localization data set is the HRTF data set, the user can distinguish whether the sound source is located in front, behind, above, below, left or right.

If multiple function data sets are used to represent the transmission characteristics of the sound waves transmitted by the sound source to multiple specific azimuths respectively, and the multiple function data sets are used to process the audio signals respectively, the audio signals can respectively represent the sound waves transmitted by the sound source to multiple specific azimuths. Azimuth transmission characteristics. With this solution, a virtual auditory environment can be constructed. On this basis, if the user's real physical orientation is projected as a specific orientation in the virtual auditory environment, the difference between the user's different real physical orientations and the different specific orientations in the virtual auditory environment will occur. By establishing a corresponding relationship between them, users can hear sound effects that are consistent with their own physical orientations according to their own physical orientations.

It can be set to allow the user to move the position. When the user's position changes or the direction of the user's ear changes, the correlation function of the sound wave transmission of different sound sources is recalculated. The user can really enjoy the concert on the spot as if he were there. Users can also issue instructions to adjust the sounding position, volume, etc. of certain instruments and parts, and enjoy their own concerts at will.

The auditory-based perception system as described above, which is used for assisting walking, assisting movement, sports training, navigation, assisting driving, assisting parking, positioning, location guidance, finding targets, reflecting pictures, reflecting objects, Detection, reconnaissance, exploration, design, maintenance, equipment use, device use, learning, teaching, shopping, office, social, games, entertainment, film and television, computer, health testing, disease diagnosis, surgical treatment, virtual concerts, virtual reality technology at least one of. For example: in the process of walking, remind the user of the route, the change of road conditions and the specific location of the obstacle through the position, distance, signal type, etc. of the auditory signal; in sports training, through the auditory signal to remind the athlete whether the angle and distance of the action meet the training requirements Or guide the athlete’s actions; in assisted driving, the driver/pilot’s route or route and the location and distance of related objects can be reminded through auditory signals; position guidance can enable users to accurately grasp the target position through auditory signals, such as when inserting a key into the keyhole , the auditory signal can reflect the relative orientation and distance between the keyhole and the key, and it can be quickly aligned even if it is invisible/invisible; the target can be found by radar or infrared equipment. The auditory signal feeds back the relative orientation and distance of the target/person to the user; when it is used to reflect the picture, the picture to be reflected can be obtained through the image sensor, and then the image recognition software parses the picture content into images such as points, lines, graphics, colors, etc. The system converts the relevant information elements and their position information into auditory signals, so that the user can receive the relevant information of the screen according to the auditory signal, or the system can feedback the screen information of the position through the auditory information according to the position specified by the user. For the user, with the movement of the user-specified position in different positions on the screen, the system can transmit the screen information to the user through auditory information. The screen here can be a real screen or a virtual screen stored in the system; When designing, it can increase the information feedback of the designer in terms of space, and have a more three-dimensional and intuitive feeling about the scheme involved. The above application can be achieved by the auditory-based perception system of the present invention alone or in combination with other systems and devices.

The auditory-based perception system of the present invention, the above-mentioned auditory-based perception system, as an auditory-based operating system, the auditory-based perception system can be used alone or in combination with other systems for operating computers, artificial intelligence, and smart devices. , virtual reality device or other suitable device. Existing computer systems usually use a video operation interface. For blind people or people with poor eyesight, or when ordinary people use it at a distance, the information fed back by the computer system cannot be well transmitted to users. In addition, the existing visual-based interface is limited in the amount of information and the form of information, and many times, it cannot fully, vividly and accurately accept instructions and feedback information. By adopting the auditory-based perception system of the present invention, the feedback information is converted into auditory perception signals and fed back to the user, and the user's instructions can be received in multiple dimensions, so that the blind or poor-sighted people, or ordinary people can easily obtain the information from a long distance. The information fed back by the computer system can also increase the way of existing computer interaction and enhance the effect of interaction, reduce the difficulty of relevant personnel using the computer system and control equipment, and improve the effect of computer use.

The computer system based on the auditory perception system can realize the operation of position, route, quantity, size, temperature, time, degree, shape, state, and object, as well as the existing computer system, and can also include object recognition/discrimination/expansion, object Movement of virtual locations, object modification/deletion, generation, alteration, etc. It is also possible to use the auditory-based perception system of the present invention to control the device to complete the operation of the target object, for example: realize the operation of manipulators, robots, smart furniture, unmanned vehicles, drones through the auditory-based perception system of the present invention. , electronic paper books, etc. A computer system combined with an auditory-based perception system can increase the spatial dimension of information and other dimensions of information that can be carried by hearing on the basis of existing computer applications, greatly improving the application efficiency and use experience of computers.

The auditory-based perception system of the present invention may further include a data transmission module, the data transmission module transmits the instructions received by the user interaction module and the information acquired by the information acquisition module or the instructions processed by the analysis processing module and/or Or the information is sent to the server/network/system/smart device, the server/network/system/smart device performs analysis and calculation according to the instruction and the information, executes the instruction and/or transmits the result to the data transmission module. Specific networks/systems/smart devices include: Internet, Internet of Things, satellite networks, local area networks, smart office systems, smart home systems, smart phones, smart TVs, smart cars, smart roads, smart cities, drones, smart robots, smart Kitchen, smart clothing, etc. Through the data transmission module, the data is transmitted to the server/network/system/smart device for analysis and calculation, which can increase the data processing capability of the auditory-based perception system of the present invention, and can also expand the application range of the auditory-based perception system of the present invention. At the same time, the calculation amount of the analysis processing module can be reduced, the hardware requirement for the analysis processing module can be reduced, and the cost and weight of the auditory-based perception system of the present invention can be reduced.

The auditory-based perception system of the present invention can also obtain information from the Internet, the Internet of Things, or other information systems, servers, and smart devices through a data transmission module or an information acquisition module, for analysis and calculation in combination with instructions. Specific sources of information can include: Internet, Internet of Things, satellite networks, local area networks, smart office systems, smart home systems, smart phones, smart speakers, smart cars, smart roads, smart cities, drones, smart robots, smart kitchens, smart Clothing, smart glasses, etc. By acquiring information in combination with the above network, system, server, and smart device, richer and more comprehensive information can be acquired, and the hardware requirements for the information acquisition module 20 can be reduced, thereby reducing the cost and weight of the auditory-based perception system of the present invention.

In the auditory-based perception system of the present invention, the model can be established by establishing a complete model by a single system, or by establishing a part of the model by multiple systems and devices based on unified signal/information standards, and then by one or more of the systems. Or servers can be integrated into a complete set of models. The information needed to build the model can be obtained through smart appliances, smart furniture, smart houses (homes, wards, hospitals, schools, factories), smart roads, smart phones, smart speakers, smart cars, smart city systems, image sensors, A positioning device and an audio acquisition device are used to obtain it.

Refer to FIG. 2 for the second embodiment of the present invention. FIG. 2 is a method flowchart of a method for using an auditory-based perception system according to a second embodiment of the present invention. As shown in the figure, the use method of the auditory-based perception system of the present invention includes:

S1: Receive user instructions;

S2: Obtain information for analyzing and processing in combination with the instruction;

S3: perform analysis and calculation according to the instruction and the information, execute the instruction and/or obtain feedback information;

S4: If there is feedback information, convert the feedback information into an auditory perception signal and feed it back to the user.

The method of using an auditory-based perception system of the present invention corresponds to the technical features of an auditory-based perception system of the present invention. Reference can be made to the foregoing description of the auditory-based perception system, which will not be repeated here.

To sum up, an auditory-based perception system and a method for using the same of the present invention include: a user interaction module, an information acquisition module, and an analysis and processing module. The method includes: receiving a user instruction; acquiring information for analyzing and processing the instruction; performing analysis and calculation according to the instruction and the information, executing the instruction and/or obtaining feedback information; if there is feedback information , which converts the feedback information into auditory perception signals and feeds them back to the user. The auditory-based perception system and its using method of the present invention can help people better use their hearing to perceive external information, enhance the effect of perception, and can help blind people or normal people when there is insufficient light or poor light. , too strong light, amblyopia, myopia, hyperopia, presbyopia, eye fatigue and other environments, or when it is inconvenient to watch carefully, such as when driving, to improve the efficiency of walking, finding objects, using computers, and smart devices/intelligent systems.

The specific embodiments described above further describe the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

An auditory-based perception system, characterized in that the system comprises: a user interaction module, an information acquisition module and an analysis and processing module,

The user interaction module is used for receiving an instruction and feeding back the feedback information to the user as an auditory perception signal;

The information acquisition module is used to acquire information, and the information is used for analysis and processing by the analysis processing module in combination with the instruction;

The analysis and processing module is configured to perform analysis and calculation according to the instruction and the information, execute the instruction and/or obtain feedback information.
The auditory-based perception system according to claim 1, wherein the conversion of feedback information into auditory perception signals is completed by a user interaction module or an analysis and processing module.
The auditory-based perception system according to claim 1, wherein the auditory perception signal is at least one of frequency, rhythm, melody, interval, orientation, distance, size, height, length and timbre of the sound to represent information.
The auditory-based perception system of claim 3, wherein the auditory perception signal comprises a speech signal.
The auditory-based perception system according to claim 1, wherein the user interaction module comprises an instruction acquisition module and an auditory perception signal output module, and the auditory perception signal output module is an earphone, a speaker, a hearing aid, a brain-computer interface at least one of.
The auditory-based perception system according to claim 3, wherein the instruction acquisition module comprises a voice recognition device, a voice recognition device, a gesture recognition device, a body motion recognition device, an expression recognition device, a body signal recognition device, an intelligent At least one of wearable devices, smart tablets, mobile phones, mice, keyboards, smart handles, smart canes, smart rings, and smart wristbands.
The auditory-based perception system according to claim 1, wherein the information acquisition module comprises an image sensor, a radar device, a radio frequency identification device, a positioning device, an audio acquisition device, an infrared device, an ultraviolet device, and a laser scanner , metal detector, temperature sensing device, light sensing device, touch sensing device, air pressure sensor, water pressure sensor, olfactory recognition device, magnetic field detection device, wind detection device, humidity detection device, power detection device, speed detection device, altitude detection device , at least one of a chemical analysis device and a radiation detection device.
The auditory-based perception system according to claim 1, wherein the auditory-based perception system can also feed back information to the user through non-auditory signals, or feedback information to the user through a combination of auditory signals and non-auditory signals.
The auditory-based perception system according to claim 1, characterized in that: the auditory-based perception system further comprises a data transmission module, and the data transmission module transfers the instruction accepted by the user interaction module to the information acquisition module. The acquired information or the instructions and/or information processed by the analysis processing module are sent to the network/system/server/smart device, and the server/network/system/smart device performs analysis and calculation according to the instruction and the information, and executes the The instructions and/or results are transmitted to the data transmission module.
The auditory-based perception system according to claim 9, characterized in that: the auditory-based perception system can also obtain instructions for combining from a network/system/server/smart device through the data transmission module or the information acquisition module Information for analytical processing.
The auditory-based perception system according to any one of claims 1-10, wherein the auditory-based perception system is used for assisting walking, assisting movement, sports training, navigation, assisting driving, assisting parking, Positioning, location guidance, finding targets, reflecting images, reflecting objects, detection, reconnaissance, exploration, design, maintenance, equipment use, device use, learning, teaching, shopping, office, social interaction, games, entertainment, film and television, computers, health testing , at least one of disease diagnosis, surgical treatment, virtual concert, and virtual reality technology.
The auditory-based perception system according to claim 1, wherein the auditory-based perception system can be used alone or in combination with other systems as an auditory-based operating system for operating computers, artificial intelligence, smart devices, virtual Realistic device.
A method of using an auditory-based perception system according to any one of claims 1-12, wherein the method comprises:

receive user instructions;

Obtain information for analysis and processing combined with instructions;

Perform analysis and calculation according to the instruction and the information, execute the instruction and/or obtain feedback information;

If there is feedback information, the feedback information is converted into auditory perception signals and fed back to the user.