WO2019238017A1

WO2019238017A1 - Method and device for providing anomalous sound source alert and augmented reality glasses

Info

Publication number: WO2019238017A1
Application number: PCT/CN2019/090607
Authority: WO
Inventors: 苏进; 张志扬; 李琦
Original assignee: 北京佳珥医学科技有限公司
Priority date: 2018-06-11
Filing date: 2019-06-11
Publication date: 2019-12-19
Also published as: CN108983965A

Abstract

The present invention relates to the technical field of augmented reality, and provides a method and device for providing an anomalous sound source alert and augmented reality glasses. The method comprises: receiving sounds from one or more sound sources (S10); determining the direction of each sound source of the one or more sound sources on the basis of the sound received from each sound source of the one or more sound sources (S11); on the basis of the sound of each sound source of the one or more sound sources and a preset alert condition, determining whether an alert is required for any of the one or more sound sources (S12); and if so, displaying information corresponding to the sound source for which an alert is required (S13). The device comprises a sound receiving module (1), a determination module (2), a processing module (3) and a display module (4). The augmented reality glasses comprise the above described device. The present invention enables a hearing-impaired person to become aware of an anomalous sound source in a timely manner and so avoid safety hazards.

Description

Method and device for alerting abnormal sound source and augmented reality glasses

Technical field

The present invention relates to the field of augmented reality technology, and in particular, to a method and device for alerting abnormal sound sources, and augmented reality glasses.

Background technique

Augmented Reality (AR) technology is a technology that calculates the position and angle of an image in real time, superimposes the corresponding image, video, and 3D model on the image, and then fuses the virtual world with the real world. The AR client can combine real-time image recognition of the offline environment of the user with the pictures stored directly in its local image recognition, and according to the pre-configured display of the identified offline targets in the real scene The effect is enhanced to display the corresponding display data. With the development of technology, the application of augmented reality technology is widespread, but for the hearing impaired, augmented reality technology has not helped them well.

Due to hearing limitations, hearing impaired people usually cannot find sound sources such as whistle cars, shouting pedestrians, falling glasses and other abnormal warning information in a timely manner. This situation is certain for hearing impaired people. Hidden dangers.

Summary of the Invention

The object of the present invention is to provide a method and device for alerting abnormal sound sources and augmented reality glasses, which can enable hearing-impaired people to discover abnormal sound sources in time and avoid hidden safety hazards.

In order to achieve the above object, one aspect of the present invention provides a method for alerting an abnormal sound source, the method comprising: receiving sound of at least one sound source; and determining based on the sound of each of the received at least one sound source, determining The position of each sound source in the at least one sound source; determining whether there is a sound source that needs to be warned in the at least one sound source based on the sound of each sound source in the at least one sound source and a preset alarm condition, The preset alarm conditions include one or more of the following: the volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source is greater than the background volume, and the difference from the background volume is greater than the preset difference The volume and the waveform of the sound of the sound source conform to a predetermined waveform; and when it is determined that there is a sound source requiring an alarm in the at least one sound source, displaying information corresponding to the sound source requiring an alarm, wherein the and The information corresponding to the sound source to be alarmed includes the position of the sound source to be alarmed.

Optionally, the information corresponding to the sound source that needs to be alarmed further includes the volume of the sound of the sound source that needs to be alarmed and / or the alarm symbol.

Optionally, when it is determined that there is a sound source that needs to be alerted in the at least one sound source, the method further includes: comparing the characteristics of the sound of the sound source that needs to be alerted with the characteristics of the preset sound waveform data To determine the type of sound of the sound source that needs to be alerted, and the information corresponding to the sound source that needs to be alerted also includes the determined type.

Accordingly, another aspect of the present invention provides an apparatus for alerting an abnormal sound source, the apparatus comprising: a sound receiving module for receiving sound of at least one sound source; and a determining module for receiving the at least one sound source based on the received The sound of each sound source in a sound source, determining the position of each sound source in the at least one sound source; a processing module, configured to be based on the sound of each sound source in the at least one sound source and a preset alarm condition Determine whether there is a sound source in the at least one sound source that needs to be alarmed, wherein the preset alarm condition includes one or more of the following: the volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source A volume that is greater than the background volume and a difference from the background volume is greater than a preset difference volume, and that the waveform of the sound of the sound source conforms to the preset waveform; and a display module, configured to determine that an alarm needs to be performed in the at least one sound source In the case of a sound source, the information corresponding to the sound source that needs to be alarmed is displayed, wherein the information corresponding to the sound source that needs to be alarmed includes the need to be alarmed The orientation of the sound source.

Optionally, when it is determined that there is a sound source that needs to be alerted in the at least one sound source, the processing module is further configured to: By comparison, the type of sound of the sound source that needs to be alerted is determined, and the information corresponding to the sound source that needs to be alerted also includes the determined type.

Optionally, the display module is a near-eye display.

Optionally, the near-eye display is a see-through near-eye display.

In addition, another aspect of the present invention provides an augmented reality glasses, which includes the above-mentioned device.

In addition, another aspect of the present invention provides a machine-readable storage medium, where the machine-readable storage medium stores instructions, and the instructions are used to cause a machine to execute the foregoing method.

According to the above technical solution, when it is determined that there is a sound source that needs to be warned in at least one of the received sound sources, the position of the sound source that needs to be warned is displayed. In this way, the hearing-impaired person is informed in a simple and intuitive manner The abnormal sound source enables the hearing impaired person to find the abnormal sound source and understand its location in time, so that the hearing impaired person can take measures against the abnormal sound source, thereby avoiding potential safety hazards.

Other features and advantages of the present invention will be described in detail in the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are used to provide a further understanding of the embodiments of the present invention, and constitute a part of the description. Together with the following specific implementations, the drawings are used to explain the embodiments of the present invention, but not to limit the embodiments of the present invention. In the drawings:

FIG. 1 is a flowchart of a method for alerting an abnormal sound source according to an embodiment of the present invention;

FIG. 2 is an exemplary diagram using arrows to indicate directions according to another embodiment of the present invention; FIG.

FIG. 3 is an exemplary diagram of an orientation provided by another embodiment of the present invention; FIG.

FIG. 4 is an exemplary diagram of displaying information corresponding to a sound source requiring an alarm according to another embodiment of the present invention; FIG.

5 is a flowchart of a method for alerting an abnormal sound source according to another embodiment of the present invention; and

FIG. 6 is a structural block diagram of an apparatus for alerting an abnormal sound source according to another embodiment of the present invention.

Reference Signs

1 sound receiving module 2 determining module

3 processing module 4 display module

detailed description

The specific implementations of the embodiments of the present invention will be described in detail below with reference to the drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

An aspect of an embodiment of the present invention provides a method for alerting an abnormal sound source. FIG. 1 is a flowchart of a method for alerting an abnormal sound source according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps.

In step S10, the sound of at least one sound source is received.

In step S11, the position of each sound source in the at least one sound source is determined based on the sound of each sound source in the received at least one sound source. The determination of the position of the sound source may be based on the time when the sound emitted from the sound source is received. For example, the sound receiving module for receiving the sound of at least one sound source includes multiple sound collection modules, the multiple sound collection modules are arranged at different positions, and the multiple sound collection modules receive sound from the same sound source. Time is different. For each sound source in the at least one sound source, the position of the sound source is determined according to the time when the sound arrives at the plurality of sound collection modules, that is, according to the time difference between the sounds arriving at the plurality of sound collection modules. Optionally, in the embodiment of the present invention, the sound collection module may be a microphone, and the sound receiving module may be a microphone array. For example, the microphone array may include 2, 4, 6, 7, or 8 microphones. Optionally, in the embodiment of the present invention, the reference point of the orientation may be set according to an actual situation, for example, it may be a position where the sound receiving module is located. Specifically, it may be any one of a plurality of sound collection modules, or may be an intermediate position of the plurality of sound collection modules. In addition, when the sound receiving module is worn by the hearing-impaired person or is not far from the hearing-impaired person, the sound-receiving module is used as the reference point, which is actually the hearing-impaired person as the reference point. Azimuth knows where the sound source is relative to itself. Optionally, in the embodiment of the present invention, the orientation may include a direction and / or a distance.

In step S12, it is determined whether there is a sound source that needs to be alarmed in the at least one sound source based on the sound of each sound source in the at least one sound source and a preset alarm condition, wherein the preset alarm condition includes one or more of the following: The volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source is greater than the background volume and the difference from the background volume is greater than the preset difference volume, and the waveform of the sound of the sound source conforms to the predetermined waveform. If it is determined that there is a sound source that needs to be alerted, step S13 is performed; otherwise, step S10 is performed, the sound from the sound source is continuously received, and the determination is continued. In this step, the sound of each sound source in at least one sound source is compared with a preset alarm condition, so as to determine whether each sound source in the at least one sound source is a sound source that needs to be warned one by one, thereby determining at least one Are there any sound sources that need to be alerted? Optionally, in the embodiment of the present invention, the preset alarm volume may be 90 dB. Optionally, in the embodiment of the present invention, the preset difference volume may be 30 dB. In addition, in the embodiment of the present invention, the background volume may be set in advance; it may also be entered after the background volume is measured using other devices after reaching an environment; or, there may be a sound detection module that automatically Detect background volume.

In step S13, when it is determined that there is a sound source that needs to be alarmed in at least one sound source, information corresponding to the sound source that needs to be alarmed is displayed, wherein the information corresponding to the sound source that needs to be alarmed includes information that needs to be performed. The position of the sound source of the alarm. Optionally, in the embodiment of the present invention, the orientation may include a direction and / or a distance, and a reference point of the orientation may be set according to an actual situation, as described above. In addition, in the embodiment of the present invention, an arrow may be used to indicate the direction. The arrow is located in an area delimited by a circle. The starting point of the arrow is the origin of the circle. The origin is equivalent to the position of the hearing impaired. The vertical axis of the circle is at an angle, as shown in FIG. 2, and the dotted line in the figure is the vertical axis passing through the circle. In addition, the horizontal axis of the circle is used as a reference, as shown by the horizontal dashed line shown in FIG. 2. When the arrow is located above the horizontal axis, it means that the sound source is in front of the hearing impaired; when the arrow is located below the horizontal axis, Indicates that the sound source is behind the hearing impaired. For example, taking the direction example shown in 2 as an example, the sound source indicated by the arrow is in front of the hearing impaired person. In addition, the way in which the direction of the sound source is indicated by arrows can also be interpreted as the direction is indicated by a clock. The circle represents the dial, and the vertical axis located in the upper half of the horizontal axis of the circle indicates the 12 o'clock direction. According to the angle at which the arrow deviates from 12 o'clock, it is determined that the sound source is about the clock direction. Taking the direction example shown in Figure 2 as an example, the sound source indicated by the arrow is about 10 o'clock. In addition, in a case where the azimuth includes a direction and a distance, the azimuth can be expressed using an example as shown in FIG. 3. It should be noted that the position of the display distance can be set according to the actual situation, and there is no limitation on this. In addition, in a case where the azimuth includes only the distance, only the distance may be displayed. In addition, in the embodiment of the present invention, text may also be used to describe the orientation. For example, taking the orientation shown in FIG. 3 as an example, the text "direction is ten o'clock and distance is 50 cm" may be displayed. In the case where the sound source to be alarmed includes one sound source, the position of the sound source to be alarmed is displayed in the example manner shown in FIG. 3; and when the sound source to be alarmed includes multiple sound sources, For each sound source that needs to be alerted, the position is displayed using the example shown in Figure 3.

When it is determined that there is a sound source that needs to be alarmed in at least one of the received sound sources, the position of the sound source that needs to be alarmed is displayed. In this way, the hearing impaired person is notified in a simple and intuitive manner that there is an abnormal sound source, so that Hearing impaired persons find the abnormal sound source in time and understand its location, so that the hearing impaired can have countermeasures against the abnormal sound source, avoiding hidden safety risks. In addition, it should be noted that the method for alarming abnormal sound sources provided by the embodiments of the present invention can be applied not only to hearing-impaired persons, but also to ordinary persons.

Optionally, in the embodiment of the present invention, the information corresponding to the sound source that needs to be alarmed further includes the volume of the sound of the sound source that needs to be alarmed and / or the alarm symbol. Optionally, the alert symbol may be "!", "*", And the like. Take the sound source that needs to be alarmed to include a sound source and the volume of the sound of the sound source is 100dB, and the information corresponding to the sound source that needs to be alarmed includes the position, volume, and alarm symbol, for example, to display the sound source that needs to be alerted. The corresponding information may be as shown in FIG. 4, wherein “!” Is used in FIG. 4 to represent the alarm symbol. In addition, when the sound source that needs to be alarmed includes multiple sound sources and the information corresponding to the sound source that needs to be alarmed includes the position, volume, and alarm symbol, the information corresponding to each sound source that needs to be alarmed can be used. The example shown in FIG. 4 is displayed. The related information of multiple sound sources can be arranged up and down or left and right, or the information of multiple sound sources that need to be alerted can be arranged in other ways.

Optionally, in the embodiment of the present invention, when it is determined that there is a sound source that needs to be warned in at least one sound source, the method further includes: changing the sound characteristics of the sound source that needs to be warned and the preset sound waveform. The data features are compared to determine the type of sound of the sound source that needs to be alerted, and the information corresponding to the sound source that needs to be alerted also includes the determined type.

FIG. 5 is a flowchart of a method for alerting an abnormal sound source according to another embodiment of the present invention. As shown in FIG. 5, the method shown in FIG. 5 is different from the method shown in FIG. 1 in that the method shown in FIG. 5 further includes the following content.

When it is determined that there is a sound source that needs to be alerted in at least one sound source, step S53 is performed. In step S53, the characteristics of the sound of the sound source that needs to be alarmed are compared with the characteristics of the preset sound waveform data, and the type of sound of the sound source that needs to be alarmed is determined. In addition, in step S54, the displayed information corresponding to the sound source requiring an alarm may include the determined type in addition to the position of the sound source requiring an alarm. In addition, the information corresponding to the sound source that needs to be alarmed may further include the volume of the sound of the sound source that needs to be alarmed and / or the alarm symbol. In this way, the hearing-impaired person is made aware of the possible sound types of sound sources that need to be alerted, so as to be aware of them. Specifically, in the embodiment of the present invention, the characteristics of the sound of the sound source that needs to be warned may be compared with the characteristics of the preset sound waveform data through AI machine learning.

Accordingly, another aspect of the embodiments of the present invention further provides an apparatus for alerting an abnormal sound source. FIG. 6 is a structural block diagram of an apparatus for alerting an abnormal sound source according to another embodiment of the present invention. As shown in FIG. 6, the device includes a sound receiving module 1, a determining module 2, a processing module 3, and a display module 4. The sound receiving module 1 is configured to receive sound from at least one sound source. The determining module 2 is configured to determine the position of each sound source in the at least one sound source based on the sound of each sound source in the received at least one sound source. The processing module 3 is configured to determine whether there is a sound source that needs to be warned in the at least one sound source based on a volume of sound of each sound source in the at least one sound source and a preset alarm condition, where the preset alarm condition includes one or more of the following The volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source is greater than the background volume and the difference from the background volume is greater than the preset difference volume, and the waveform of the sound of the sound source conforms to the preset waveform. The display module 4 is configured to display information corresponding to a sound source requiring an alarm when it is determined that there is a sound source requiring an alarm in at least one sound source, and the information corresponding to the sound source requiring an alarm includes the need to perform an alarm. The orientation of the sound source.

When it is determined that there is a sound source that needs to be alarmed in at least one of the received sound sources, the position of the sound source that needs to be alarmed is displayed. In this way, the hearing impaired person is notified in a simple and intuitive manner that there is an abnormal sound source, so that Hearing impaired persons find the abnormal sound source in time and understand its location, so that the hearing impaired can have countermeasures against the abnormal sound source, avoiding hidden safety risks.

Optionally, in the embodiment of the present invention, the display module may be a near-eye display, so that the position of the sound source that needs to be warned is presented in front of the eyes. Preferably, in the embodiment of the present invention, the near-eye display may be a see-through near-eye display. In this way, without affecting the hearing-impaired person from observing other things, the hearing-impaired person can see the sound source that needs to be alerted in real time. Orientation.

The specific working principle and benefits of the device for alarming abnormal sound sources provided by the embodiments of the present invention are similar to the specific working principles and benefits of the method for alarming abnormal sound sources provided by the embodiments of the present invention, and will not be repeated here.

In addition, an embodiment of the present invention further provides augmented reality glasses, which include the devices described in the foregoing embodiments. The augmented reality glasses include an electronic circuit system that supports the operation of the device described in the foregoing embodiment. The electronic circuit system includes a power supply, a processor, a network connection module, and a sound receiving module. In addition, the electronic circuit system may further include an externally visible human-machine interface module and buttons and / or a touch control panel. The processor includes the processing module and the voice recognition module described in the foregoing embodiments. The human-machine interface module includes a display module. The processor can also perform offline voice recognition locally, or online voice recognition in the cloud via a network connection.

Optionally, in the embodiment of the present invention, the touch control panel, buttons, and / or sound receiving module described in the foregoing embodiments may be provided on the glasses or glasses accessories of the augmented reality glasses, for example, on the temples, On the frame or lens. Optionally, in the embodiment of the present invention, the sound receiving module may be disposed on a mirror frame, on the same temple, or on a different temple. For example, when the sound receiving module is a microphone array and the microphone sub-array includes two microphones, the two microphones are respectively arranged on two frames, or are disposed on different positions of the same temple, or are respectively disposed on two On the temples. In addition, when the display module is a near-eye display, the orientation of the sound source that needs to be warned is presented in front of the eyes. Among them, the near-eye display may be transparent or non-transparent. Further, in the case where the near-eye display is a see-through near-eye display, it realizes that the hearing-impaired person can see the position of the sound source that needs to be alerted in real time without affecting the viewing scene of the hearing-impaired person.

In addition, another aspect of the embodiments of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores instructions, and the instructions are used to cause a machine to execute the method described in the foregoing embodiments.

In summary, when it is determined that there is a sound source that needs to be warned in at least one of the received sound sources, the position of the sound source that needs to be warned is displayed. In this way, the hearing-impaired person is notified in a simple and intuitive way of existence The abnormal sound source enables the hearing impaired person to find the abnormal sound source and understand its location in time, so that the hearing impaired person can take measures against the abnormal sound source, thereby avoiding potential safety hazards. In addition, the displayed information corresponding to the sound source that needs to be alerted may further include the type of the sound source that is determined to require an alert. In this way, the hearing-impaired person is made aware of the possible sound types of sound sources that need to be alerted, so as to be aware of them.

The optional implementations of the embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the foregoing implementations. Within the scope of the technical concept of the embodiments of the present invention, the embodiments of the present invention may be implemented. The technical solution of the present invention performs various simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

In addition, it should be noted that the specific technical features described in the foregoing specific embodiments can be combined in any suitable manner without conflict. In order to avoid unnecessary repetition, the embodiments of the present invention do not separately describe various possible combinations.

Those skilled in the art can understand that all or part of the steps in the method of the above embodiments can be completed by a program instructing related hardware. The program is stored in a storage medium and includes a number of instructions to enable a microcontroller, a chip, or a processor. (processor) executes all or part of the steps of the method described in each embodiment of the present application. The foregoing storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes .

In addition, various combinations of the embodiments of the present invention can also be arbitrarily combined, as long as it does not violate the idea of the embodiment of the present invention, it should also be regarded as the content disclosed in the embodiment of the present invention.

Claims

A method for alerting an abnormal sound source, characterized in that the method includes:

Receiving sound from at least one sound source;

Determining the position of each sound source in the at least one sound source based on the received sound of each of the at least one sound source;

Determine whether there is a sound source that needs to be warned in the at least one sound source based on the sound of each sound source in the at least one sound source and a preset alarm condition, wherein the preset alarm condition includes one or more of the following : The volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source is greater than the background volume and the difference from the background volume is greater than the preset difference volume, and the waveform of the sound of the sound source conforms to a predetermined waveform; and

When it is determined that there is a sound source requiring alarm in the at least one sound source, information corresponding to the sound source requiring alarm is displayed, wherein the information corresponding to the sound source requiring alarm includes information requiring alarm. The position of the sound source.
The method according to claim 1, wherein the information corresponding to the sound source that needs to be alarmed further comprises a volume and / or an alarm symbol of a sound of the sound source that needs to be alarmed.
The method according to claim 1 or 2, wherein when it is determined that there is a sound source that needs to be warned in the at least one sound source, the method further comprises: The feature is compared with the preset sound waveform data feature to determine the type of sound of the sound source that needs to be alerted,

The information corresponding to the sound source that needs to be alerted also includes the determined type.
A device for alarming abnormal sound sources, characterized in that the device includes:

A sound receiving module, configured to receive sound from at least one sound source;

A determining module, configured to determine the position of each sound source in the at least one sound source based on the sound of each sound source in the received at least one sound source;

A processing module, configured to determine whether there is a sound source that needs to be alarmed in the at least one sound source based on the sound of each sound source in the at least one sound source and a preset alarm condition, wherein the preset alarm condition includes the following One or more: the volume of the sound of the sound source is greater than the preset alarm volume, the volume of the sound of the sound source is greater than the background volume and the difference from the background volume is greater than the preset difference volume, and the waveform of the sound of the sound source Conforms to a preset waveform; and

A display module, configured to display information corresponding to a sound source requiring an alarm when it is determined that there is a sound source requiring an alarm in the at least one sound source, wherein the information corresponding to the sound source requiring an alarm is displayed Include the position of the sound source that needs to be alerted.
The device according to claim 4, wherein the information corresponding to the sound source that needs to be alarmed further comprises a volume of sound of the sound source that needs to be alarmed and / or an alarm symbol.
The device according to claim 4, characterized in that, in a case where it is determined that there is a sound source that needs to be warned in the at least one sound source, the processing module is further configured to use the sound of the sound source that needs to be warned. The feature is compared with the preset sound waveform data feature to determine the type of sound of the sound source that needs to be alerted,

The information corresponding to the sound source that needs to be alerted also includes the determined type.
The device according to any one of claims 4-6, wherein the display module is a near-eye display.
The device according to claim 7, wherein the near-eye display is a see-through near-eye display.
An augmented reality glasses, characterized in that the augmented reality glasses include: the device according to any one of claims 4-8.
A machine-readable storage medium has instructions stored on the machine-readable storage medium, which are used to cause a machine to perform the method according to any one of claims 1-3.