CN112185415A - Sound visualization method and device, storage medium and MR mixed reality equipment - Google Patents
Sound visualization method and device, storage medium and MR mixed reality equipment Download PDFInfo
- Publication number
- CN112185415A CN112185415A CN202010948238.2A CN202010948238A CN112185415A CN 112185415 A CN112185415 A CN 112185415A CN 202010948238 A CN202010948238 A CN 202010948238A CN 112185415 A CN112185415 A CN 112185415A
- Authority
- CN
- China
- Prior art keywords
- sound
- monitored
- sound signals
- mixed reality
- converted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/04—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
- G08B21/0438—Sensor means for detecting
- G08B21/0469—Presence detectors to detect unsafe condition, e.g. infrared sensor, microphone
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B7/00—Signalling systems according to more than one of groups G08B3/00 - G08B6/00; Personal calling systems according to more than one of groups G08B3/00 - G08B6/00
- G08B7/06—Signalling systems according to more than one of groups G08B3/00 - G08B6/00; Personal calling systems according to more than one of groups G08B3/00 - G08B6/00 using electric transmission, e.g. involving audible and visible signalling through the use of sound and light sources
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
- G10L21/10—Transforming into visible information
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
- G10L2021/065—Aids for the handicapped in understanding
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Gerontology & Geriatric Medicine (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a sound visualization method and device, a storage medium and MR mixed reality equipment, wherein the method comprises the following steps: monitoring sound signals in a target environment in real time, wherein the target environment is an environment where MR mixed reality equipment is located; comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result; screening out a sound signal to be converted from the monitored sound signals; according to the type of the monitored sound signal, the sound signal to be converted is converted into visual information corresponding to the content of the sound signal, the visual information is fused with a real scene to be displayed, the sound visualization and MR mixed reality technology is combined, the visual information can be presented in the real scene, and the hearing-impaired people can be helped to feel the sound of the surrounding environment on the premise of not influencing the real life of the hearing-impaired people.
Description
Technical Field
The invention belongs to the technical field of voice recognition and conversion application, and particularly relates to a voice visualization method and device, a storage medium and MR mixed reality equipment.
Background
The MR mixed reality technology is a technology that mixes physical reality with a digital world, and realizes capturing of position data in the real world of a person, and includes position, illumination, sound, localization, and object recognition, etc., of boundaries and surfaces thereof through spatial mapping and understanding. By combining the three aspects of computer processing, human input, and environmental input, a mixed world can then be created and the immersive mixed reality experience experienced by the outsiders.
The patent with publication number CN109308282A belongs to the invention in this field after search. The invention discloses a parallel architecture method and a parallel architecture device applied to MR mixed reality equipment, which adopt a heterogeneous parallel form of a central processing unit, a graphic image processor and a reconfigurable processor combination, and utilize the characteristics of high concurrency, low power consumption and reconfigurability of the reconfigurable processor to realize high-speed processing of data streams, so that pictures displayed by the MR mixed reality equipment can be more real-time, and user experience is improved.
Currently, there are MR mixed reality devices on the market, such as MR wearable devices, and the wearer can feel the projection of virtual images in real life.
At present, auditory handicapped people cannot hear sound, cannot know the discussion content of people around, and cannot feel the atmosphere of the environment around, so that the auditory handicapped people cannot blend into the environment around. Even in special cases, there may be a danger of not being able to react in a timely manner.
There is a need for a sound visualization method and apparatus, a storage medium, and an MR mixed reality device.
Disclosure of Invention
The technical problem to be solved by the invention is that the auditory handicapped person cannot hear the sound, so that the auditory handicapped person cannot blend into the surrounding environment.
In order to solve the problems, the invention provides a sound visualization method and device, a storage medium and an MR mixed reality device.
In a first aspect, the present invention provides a sound visualization method applied to an MR mixed reality device, including the following steps:
monitoring sound signals in a target environment in real time, wherein the target environment is an environment where MR mixed reality equipment is located;
comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result;
screening out a sound signal to be converted from the monitored sound signals;
and converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and the real scene.
According to an embodiment of the present invention, preferably, the step of screening the monitored sound signals for conversion comprises the steps of:
and screening out the sound signals with the volume exceeding a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
According to an embodiment of the present invention, preferably, the step of screening the monitored sound signals for conversion comprises the steps of:
judging whether a sound for prompting a dangerous condition exists in the monitored sound signal or not according to the type of the monitored sound signal;
when the monitored sound signals have sounds for prompting the dangerous conditions, the sounds for prompting the dangerous conditions are used as the sound signals to be converted;
and when no sound for prompting the dangerous condition exists in the monitored sound signals, screening out the sound signals of which the volume exceeds a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
According to an embodiment of the present invention, preferably, when the type of the monitored sound signal is music, the visual information is a video with lyrics and/or with music waveform rhythm beats.
According to an embodiment of the present invention, preferably, when the type of the monitored sound signal is a language, the visual information is a text and/or a gesture.
According to an embodiment of the present invention, it is preferable that the visual information is a warning image indicating different levels of danger when the type of the monitored sound signal is a vehicle siren.
According to an embodiment of the present invention, preferably, the risk level is determined by:
calculating the speed of the vehicle according to the position change of the vehicle at two adjacent moments;
the hazard level of the vehicle is determined from the speed of the vehicle and the distance between the vehicle and the MR mixed reality device.
In a second aspect, the present invention provides a sound visualization apparatus applied to an MR mixed reality device, including:
the monitoring module is used for monitoring sound signals in a target environment in real time, wherein the target environment is the environment where the MR mixed reality equipment is located;
the determining module is used for comparing the monitored sound signals with various pre-stored reference sounds and determining the type of the monitored sound signals according to the comparison result;
the screening module is used for screening out the sound signals to be converted from the monitored sound signals;
and the conversion module is used for converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and the real scene.
In a third aspect, the invention provides a storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.
In a fourth aspect, the invention provides an MR mixed reality device comprising a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, carries out the steps of the method described above.
Compared with the prior art, one or more embodiments in the above scheme can have the following advantages or beneficial effects:
the sound visualization method is applied to monitor sound signals in a target environment in real time, wherein the target environment is the environment where the MR mixed reality equipment is located; comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result; screening out a sound signal to be converted from the monitored sound signals; according to the type of the monitored sound signal, the sound signal to be converted is converted into visual information corresponding to the content of the sound signal, the visual information is fused with a real scene to be displayed, the sound visualization and MR mixed reality technology is combined, the visual information can be presented in the real scene, and the hearing-impaired people can be helped to feel the sound of the surrounding environment on the premise of not influencing the real life of the hearing-impaired people.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 shows a flow chart of a sound visualization method according to an embodiment of the invention;
FIG. 2 shows a flow chart of a sound visualization method according to an embodiment of the invention;
fig. 3 shows a flow chart of a method for visualizing three sounds according to an embodiment of the present invention.
Detailed Description
The following detailed description of the embodiments of the present invention will be provided with reference to the drawings and examples, so that how to apply the technical means to solve the technical problems and achieve the technical effects can be fully understood and implemented. It should be noted that, as long as there is no conflict, the embodiments and the features of the embodiments of the present invention may be combined with each other, and the technical solutions formed are within the scope of the present invention.
Example one
In order to solve the technical problems in the prior art, an embodiment of the present invention provides a sound visualization method, and in this embodiment, a sound signal to be converted is screened from monitored sound signals according to a volume.
Referring to fig. 1, the sound visualization method of the present embodiment is applied to an MR mixed reality device, and includes the following steps:
s110, monitoring sound signals in a target environment in real time, wherein the target environment is an environment where the MR mixed reality equipment is located;
s120, comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result;
s130, judging whether the monitored sound signals have sound signals with the volume exceeding a preset threshold value:
if yes, go to step S140;
if not, no response is given;
s140, taking the sound signal with the volume exceeding a preset threshold value as a sound signal to be converted;
s150, converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and a real scene.
In step S120, the pre-stored reference sounds are pre-established sound banks according to the difference of frequency and amplitude of each sound, for example, a section of car whistling sound is input, and when sounds with similar frequencies are acquired, it can be determined as whistling sound.
In step S150, the conversion form of the sound signal is different for different types of the monitored sound signal. The method comprises the following specific steps:
when the type of the monitored sound signal is music, the sound signal to be converted is converted into a video with lyrics and/or with music waveform rhythm beats.
And when the type of the monitored sound signal is a language, converting the sound signal to be converted into characters and/or gestures.
When the type of the monitored sound signal is a vehicle whistling sound, the sound signal to be converted is converted into a warning image representing different levels of danger.
Wherein the hazard level is determined by:
calculating the speed of the vehicle according to the position change of the vehicle at two adjacent moments;
the hazard level of the vehicle is determined from the speed of the vehicle and the distance between the vehicle and the MR mixed reality device.
Specifically, when the speed of the vehicle is calculated, the MR mixed reality device, similar to a tachometer for a vehicle, captures the position of the same vehicle by using a dynamic image tracking recognition technology at two moments, and can calculate the speed according to a distance and time formula. The ranging principle can be referred to as a rangefinder of an iPhone device.
The sound visualization method of the embodiment screens the sound signal to be converted from the monitored sound signals according to the volume, performs sound visualization on the sound signal with the volume exceeding the preset threshold value, and enables a user to visually feel the sound related to the user in the surrounding environment.
The sound visualization method of the present embodiment converts music into video with lyrics and/or with music waveform rhythm beats, thereby allowing a user to visually perceive the music.
The sound visualization method of the present embodiment converts the language into text and/or gestures, thereby allowing the user to visually perceive the language.
The sound visualization method of the present embodiment converts the vehicle whistle into a warning image representing different levels of danger, thereby allowing the user to visually perceive the vehicle danger.
The sound visualization method of the embodiment judges the danger of the vehicle to the user according to the speed of the vehicle and the distance between the vehicle and the user, so that the user can react in time according to the danger of the vehicle.
The sound visualization method of the embodiment utilizes the MR mixed reality technology to fuse and display the visualization information and the real scene, can utilize the MR mixed reality technology to help hearing-impaired people understand the sound, brings living convenience to the hearing-impaired people, and helps the hearing-impaired people to blend into the environment nearby; weakening the subjective missense of auditory handicapped people and increasing courage for the daily life of the auditory handicapped people.
Example two
In order to solve the technical problems in the prior art, an embodiment of the present invention provides a sound visualization method, and in this embodiment, a sound signal to be converted is screened from monitored sound signals in combination with a dangerous condition and a volume.
Referring to fig. 2, the sound visualization method of the present embodiment is applied to an MR mixed reality device, and includes the following steps:
s210, monitoring sound signals in a target environment in real time, wherein the target environment is an environment where MR mixed reality equipment is located;
s220, comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result;
s230, judging whether the monitored sound signals have sounds for prompting dangerous conditions according to the types of the monitored sound signals:
if yes, go to S240;
if not, executing S250;
s240, using the sound for prompting the dangerous situation as a sound signal to be converted;
s250, judging whether the monitored sound signals have sound signals with the volume exceeding a preset threshold value:
if yes, go to step S260;
if not, no response is given;
s260, taking the sound signal with the volume exceeding a preset threshold value as a sound signal to be converted;
and S270, converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and a real scene.
In step S270, the conversion form of the sound signal is different for different types of the monitored sound signal. The method comprises the following specific steps:
when the type of the monitored sound signal is music, the sound signal to be converted is converted into a video with lyrics and/or with music waveform rhythm beats.
And when the type of the monitored sound signal is a language, converting the sound signal to be converted into characters and/or gestures.
When the type of the monitored sound signal is a vehicle whistling sound, the sound signal to be converted is converted into a warning image representing different levels of danger.
Wherein the hazard level is determined by:
calculating the speed of the vehicle according to the position change of the vehicle at two adjacent moments;
the hazard level of the vehicle is determined from the speed of the vehicle and the distance between the vehicle and the MR mixed reality device.
The sound visualization method of the embodiment combines the dangerous condition prompting and the volume to screen the sound signal to be converted from the monitored sound signal, firstly judges whether the monitored sound signal is the sound for prompting the dangerous condition, and then screens the sound signal to be converted according to the volume of the monitored sound signal, so that the sound for prompting the dangerous condition is preferentially presented, and the safety of a user is guaranteed.
The sound visualization method of the embodiment combines the MR with the voice input and the voice analysis by utilizing the MR mixed reality technology, visualizes the sound in the visual field of the hearing impaired person, helps the hearing impaired person to blend into the surrounding environment, and timely reacts to other people or critical conditions
EXAMPLE III
In order to solve the technical problems in the prior art, an embodiment of the present invention further provides an example of applying the sound visualization method of the first embodiment to an MR mixed reality device worn by a hearing-impaired person.
Referring to fig. 3, the sound visualization method of the present embodiment includes the following steps:
s310, wearing the MR mixed reality equipment by the hearing-impaired person;
s320, receiving ambient environment sound by the MR mixed reality equipment;
s330, intelligently judging the sound type, such as whistling, language, music and the like;
and S340, combining a plurality of technologies (image recognition, semantic analysis and the like) to text or graph the sound, assisting with the action, presenting the sound in the visual field range of the MR mixed reality device wearer, and fusing the sound with the real scene.
The sound visualization method of the present embodiment is described below in a specific application scenario.
Scene one: the hearing-impaired person A wears the MR wearable device, walks on a road, and has a vehicle in a long distance to drive at high speed and continuously whistle. At the moment, the MR wearable device catches the whistle sound, the speed of the MR wearable device is calculated by combining the driving position and the position change situation in the real world according to the continuous situation of the sound, the real-time distance between the MR wearable device and the A is also calculated, the danger level is judged, and the danger warning with the corresponding level is sent out: therefore, the driving speed is slow, the whistle sound is short, and a certain distance is left between the dangerous level and the A, the dangerous level is judged to be low level, and character prompt information (which can also be combined with a graph) which is light in color and does not block the main realization is projected in the visual field of the A; on the contrary, if the system determines that the current danger level is higher, an alarm with heavier color (such as red or by using a contrast color) is projected in the visual field of the A and flickers, so that the system plays a role of strong prompt.
Scene two: the hearing-impaired person B wears the MR wearable device to walk into a restaurant, and the restaurant plays relaxing music. The MR wearable device captures the rhythm and melody of the current music and generates the dynamic effect corresponding to the music rhythm, such as beating note and rhythm light, in the visual field of B by combining the lyric content (the lyric is searched after real-time listening and translating or on-line listening to the song and recognizing the song). When B sits down to order, the waiter goes to introduce dishes to the waiter and order the dishes, at the moment, the MR wearable device identifies the content spoken by the waiter and translates the content in real time, and the content is displayed in the visual field range of B in the form of characters and virtual character stroke sign language.
The sound visualization method of the embodiment visualizes the sound of the automobile whistle in the first scene, combines the prompt message with the real scene, and directly presents the sound in front of eyes, so that the sound visualization method can enable the hearing-impaired people to know the sound at the first time and react in time compared with other alarm devices, and safety guarantee is brought to the hearing-impaired people.
The sound visualization method of the embodiment visualizes the restaurant music in the scene two, so that the hearing impaired people can feel the atmosphere of the current environment through another sense organ, meanwhile, the sound visualization method of the embodiment can help the hearing impaired people to communicate with other people more conveniently, great convenience is brought to the daily social life of the hearing impaired people, and the subjective feeling of discrimination of the hearing impaired people is effectively weakened.
The sound visualization method of the embodiment helps hearing-impaired people to realize sound visualization based on the MR mixed reality technology, and only a voice module is required to be arranged on the current MR wearable equipment and used for capturing and analyzing environmental sound and understanding semantics, so that the hearing-impaired people can be helped to feel the sound of the surrounding environment.
Example four
In order to solve the technical problems in the prior art, the embodiment of the invention also provides a sound visualization device.
The sound visualization apparatus of the embodiment is applied to an MR mixed reality device, and includes:
the monitoring module is used for monitoring sound signals in a target environment in real time, wherein the target environment is the environment where the MR mixed reality equipment is located;
the determining module is used for comparing the monitored sound signals with various pre-stored reference sounds and determining the type of the monitored sound signals according to the comparison result;
the screening module is used for screening out the sound signals to be converted from the monitored sound signals;
and the conversion module is used for converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and the real scene.
The screening module is further used for screening out the sound signals with the volume exceeding a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
The screening module is further used for judging whether sounds for prompting dangerous conditions exist in the monitored sound signals or not according to the types of the monitored sound signals;
when the monitored sound signals have sounds for prompting the dangerous conditions, the sounds for prompting the dangerous conditions are used as the sound signals to be converted;
and when no sound for prompting the dangerous condition exists in the monitored sound signals, screening out the sound signals of which the volume exceeds a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
In the conversion module, when the type of the monitored sound signal is music, the visual information is video with lyrics and/or with music waveform rhythm beats.
In the conversion module, when the type of the monitored sound signal is a language, the visual information is a character and/or a gesture.
In the conversion module, when the type of the monitored sound signal is a vehicle whistling sound, the visual information is a warning image representing different danger levels.
In the conversion module, the hazard level is determined by:
calculating the speed of the vehicle according to the position change of the vehicle at two adjacent moments;
the hazard level of the vehicle is determined from the speed of the vehicle and the distance between the vehicle and the MR mixed reality device.
The sound visualization device of this embodiment combines the sound visualization with the MR mixed reality technology, and it presents visual information and real scene fusion through the MR mixed reality technology, when supplying the hearing impaired personage to watch, can be under the prerequisite that does not influence the real life of hearing impaired personage, help the sound of hearing impaired personage experience surrounding environment makes things convenient for the hearing impaired personage to adapt to surrounding environment better.
EXAMPLE five
In order to solve the above technical problems in the prior art, an embodiment of the present invention further provides a storage medium.
The storage medium of the present embodiment has stored thereon a computer program which, when executed by a processor, implements the steps of the method in the above-described embodiments.
EXAMPLE six
In order to solve the technical problems in the prior art, an embodiment of the present invention further provides an MR mixed reality device.
The MR mixed reality device of the present embodiment includes a storage medium and a processor, the memory having stored thereon a computer program, which when executed by the processor implements the steps of the above method.
Although the embodiments of the present invention have been described above, the above description is only for the convenience of understanding the present invention, and is not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A sound visualization method is applied to an MR mixed reality device, and is characterized by comprising the following steps:
monitoring sound signals in a target environment in real time, wherein the target environment is an environment where MR mixed reality equipment is located;
comparing the monitored sound signals with various pre-stored reference sounds, and determining the type of the monitored sound signals according to the comparison result;
screening out a sound signal to be converted from the monitored sound signals;
and converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and the real scene.
2. The method of claim 1, wherein the step of screening the monitored acoustic signals for conversion comprises the steps of:
and screening out the sound signals with the volume exceeding a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
3. The method of claim 1, wherein the step of screening the monitored acoustic signals for conversion comprises the steps of:
judging whether a sound for prompting a dangerous condition exists in the monitored sound signal or not according to the type of the monitored sound signal;
when the monitored sound signals have sounds for prompting the dangerous conditions, the sounds for prompting the dangerous conditions are used as the sound signals to be converted;
and when no sound for prompting the dangerous condition exists in the monitored sound signals, screening out the sound signals of which the volume exceeds a preset threshold value from the monitored sound signals to serve as the sound signals to be converted.
4. The method according to claim 1, characterized in that the visual information is a video with lyrics and/or with rhythmic beats of the musical waveform when the type of the monitored sound signal is music.
5. The method of claim 1, wherein the visual information is text and/or gestures when the type of the monitored acoustic signal is a language.
6. The method of claim 1, wherein the visual information is a warning image indicating different levels of danger when the type of the monitored acoustic signal is a vehicle siren.
7. The method of claim 6, wherein the risk level is determined by:
calculating the speed of the vehicle according to the position change of the vehicle at two adjacent moments;
the hazard level of the vehicle is determined from the speed of the vehicle and the distance between the vehicle and the MR mixed reality device.
8. A sound visualization device applied to an MR mixed reality device is characterized by comprising:
the monitoring module is used for monitoring sound signals in a target environment in real time, wherein the target environment is the environment where the MR mixed reality equipment is located;
the determining module is used for comparing the monitored sound signals with various pre-stored reference sounds and determining the type of the monitored sound signals according to the comparison result;
the screening module is used for screening out the sound signals to be converted from the monitored sound signals;
and the conversion module is used for converting the sound signal to be converted into visual information corresponding to the content of the sound signal according to the type of the monitored sound signal, and fusing and displaying the visual information and the real scene.
9. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
10. MR mixed reality device comprising a memory and a processor, characterized in that the memory has stored thereon a computer program which, when being executed by the processor, carries out the steps of the method according to one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010948238.2A CN112185415A (en) | 2020-09-10 | 2020-09-10 | Sound visualization method and device, storage medium and MR mixed reality equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010948238.2A CN112185415A (en) | 2020-09-10 | 2020-09-10 | Sound visualization method and device, storage medium and MR mixed reality equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112185415A true CN112185415A (en) | 2021-01-05 |
Family
ID=73921717
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010948238.2A Pending CN112185415A (en) | 2020-09-10 | 2020-09-10 | Sound visualization method and device, storage medium and MR mixed reality equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112185415A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112714355A (en) * | 2021-03-29 | 2021-04-27 | 深圳市火乐科技发展有限公司 | Audio visualization method and device, projection equipment and storage medium |
| CN113314141A (en) * | 2021-05-21 | 2021-08-27 | 江苏第二师范学院 | Vocal music singing volume and pitch quantification method and quantification equipment |
| GB2616648A (en) * | 2022-03-16 | 2023-09-20 | Wayray Ag | System and method |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107024982A (en) * | 2016-12-09 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method for safety monitoring of virtual reality device, device and virtual reality device |
| TWM547728U (en) * | 2017-06-01 | 2017-08-21 | Nat Taichung Univ Of Science And Tech | Intelligent real-time traffic system dedicated for the hearing-impaired |
| CN107223277A (en) * | 2016-12-16 | 2017-09-29 | 深圳前海达闼云端智能科技有限公司 | A kind of deaf-mute's householder method, device and electronic equipment |
| CN107703738A (en) * | 2016-08-09 | 2018-02-16 | 湖北文理学院 | The patient that becomes deaf helps sense wrist-watch |
| CN108511002A (en) * | 2018-01-23 | 2018-09-07 | 努比亚技术有限公司 | The recognition methods of hazard event voice signal, terminal and computer readable storage medium |
| CN108962254A (en) * | 2018-06-11 | 2018-12-07 | 北京佳珥医学科技有限公司 | For assisting the methods, devices and systems and augmented reality glasses of hearing-impaired people |
| CN108983965A (en) * | 2018-06-11 | 2018-12-11 | 北京佳珥医学科技有限公司 | For alerting the method and apparatus and augmented reality glasses of abnormal sound source |
| CN109065055A (en) * | 2018-09-13 | 2018-12-21 | 三星电子（中国）研发中心 | Method, storage medium and the device of AR content are generated based on sound |
| CN109784128A (en) * | 2017-11-14 | 2019-05-21 | 幻视互动（北京）科技有限公司 | Mixed reality intelligent glasses with text and language process function |
-
2020
- 2020-09-10 CN CN202010948238.2A patent/CN112185415A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107703738A (en) * | 2016-08-09 | 2018-02-16 | 湖北文理学院 | The patient that becomes deaf helps sense wrist-watch |
| CN107024982A (en) * | 2016-12-09 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method for safety monitoring of virtual reality device, device and virtual reality device |
| CN107223277A (en) * | 2016-12-16 | 2017-09-29 | 深圳前海达闼云端智能科技有限公司 | A kind of deaf-mute's householder method, device and electronic equipment |
| TWM547728U (en) * | 2017-06-01 | 2017-08-21 | Nat Taichung Univ Of Science And Tech | Intelligent real-time traffic system dedicated for the hearing-impaired |
| CN109784128A (en) * | 2017-11-14 | 2019-05-21 | 幻视互动（北京）科技有限公司 | Mixed reality intelligent glasses with text and language process function |
| CN108511002A (en) * | 2018-01-23 | 2018-09-07 | 努比亚技术有限公司 | The recognition methods of hazard event voice signal, terminal and computer readable storage medium |
| CN108962254A (en) * | 2018-06-11 | 2018-12-07 | 北京佳珥医学科技有限公司 | For assisting the methods, devices and systems and augmented reality glasses of hearing-impaired people |
| CN108983965A (en) * | 2018-06-11 | 2018-12-11 | 北京佳珥医学科技有限公司 | For alerting the method and apparatus and augmented reality glasses of abnormal sound source |
| CN109065055A (en) * | 2018-09-13 | 2018-12-21 | 三星电子（中国）研发中心 | Method, storage medium and the device of AR content are generated based on sound |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112714355A (en) * | 2021-03-29 | 2021-04-27 | 深圳市火乐科技发展有限公司 | Audio visualization method and device, projection equipment and storage medium |
| CN113314141A (en) * | 2021-05-21 | 2021-08-27 | 江苏第二师范学院 | Vocal music singing volume and pitch quantification method and quantification equipment |
| GB2616648A (en) * | 2022-03-16 | 2023-09-20 | Wayray Ag | System and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112185415A (en) | Sound visualization method and device, storage medium and MR mixed reality equipment | |
| McKeown et al. | Mapping candidate within-vehicle auditory displays to their referents | |
| JP5881263B2 (en) | Display of sound status on wearable computer system | |
| US20110218953A1 (en) | Design of systems for improved human interaction | |
| Ribeiro et al. | Auditory augmented reality: Object sonification for the visually impaired | |
| WO2016199248A1 (en) | Information presentation system and information presentation method | |
| CN107945284B (en) | Scene display method for automatic driving vehicle and intelligent glasses | |
| Wang et al. | Using advisory 3D sound cues to improve drivers' performance and situation awareness | |
| JP2021177390A (en) | Content output system, terminal device, content output method, information processing method, and program | |
| WO2017169273A1 (en) | Information processing device, information processing method, and program | |
| JP2018189720A (en) | Information output control device, information output control method, information output system and program | |
| JP2019135505A (en) | Program and train operation simulator | |
| US10978072B2 (en) | Display control system, display control method, and computer program product using display sectors in one revolution of clockwise or counterclockwise direction | |
| CN113452985A (en) | Device and head-mounted kit | |
| WO2020234939A1 (en) | Information processing device, information processing method, and program | |
| WO2017169272A1 (en) | Information processing device, information processing method, and program | |
| Haverkamp | Advanced description of noise perception by analysis of cross-sensory interactions within soundscapes | |
| US20210097727A1 (en) | Computer apparatus and method implementing sound detection and responses thereto | |
| JPH1165814A (en) | Interactive system and image display method | |
| WO2023176145A1 (en) | Information processing device, information processing method, and program | |
| Sappa et al. | Coloresia: An Interactive Colour Perception Device for the Visually Impaired | |
| Haverkamp | Essentials for description of crosssensory interaction during perception of a complex environment | |
| Graakjær | Audi (o) Branding and Object Sounds in an Audio-Visual Setting: The Case of the Car | |
| RU2679866C1 (en) | Visual-sound system for the blind and partially sighted people | |
| CN116860196A (en) | System and method for interaction according to noise information, electronic equipment and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |