CN110767226A - Sound source localization method, device, speech recognition method, system, storage device and terminal with high accuracy - Google Patents

Abstract

The invention discloses a sound source positioning method and a device with high accuracy, which comprises the following steps: collecting a sound signal; judging whether a voice signal exists in the sound signal or not; extracting all voice signals and acquiring the sound source position of each voice signal; performing voiceprint recognition on each voice signal one by one; judging whether the identified voiceprint features are stored in a voiceprint database or not; acquiring image information of a sound source position where a voice signal corresponding to the voiceprint feature is located; performing model training by using a machine self-learning method, determining a speaker corresponding to the voiceprint characteristics and identity information thereof, and storing the corresponding voiceprint characteristics and the identity information of the speaker in a voiceprint database; and displaying the sound source position information of the voice signal corresponding to the voiceprint characteristics and the identity information of the corresponding speaker. The invention can position the speaker in preparation, match the identity of the speaker and the content of speaking; the method is suitable for the field of voice recognition.

Description

Sound source localization method, device, speech recognition method, system, storage device and terminal with high accuracy

technical field

The present invention relates to the technical field of speech recognition, in particular to a sound source localization method and device with high accuracy, and a speech recognition method, system, storage device and terminal with high pertinence.

Background technique

As one of the ten important scientific and technological development technologies in the field of information technology, speech recognition technology has been widely developed in various fields such as industry, home appliances, communications, automotive electronics, medical care, home services, and consumer electronics products. When performing speech recognition, some algorithms based on microphone arrays are usually used for sound source localization.

Generally speaking, sound source localization algorithms based on microphone arrays are divided into three categories: one is the method based on beamforming, the other is the method based on high-resolution spectral estimation, and the third is the method based on the delay difference of arrival (TDOA). (1) Beamforming is a controllable beamforming technology based on maximum output power. Its basic idea is to weight and sum the signals collected by each array element to form a beam, and to guide the beam by searching for the possible position of the sound source, and modify the weights. Maximize the output signal power of the microphone array. This method can be used both in the time domain and in the frequency domain. Its time shift in the time domain is equivalent to a phase delay in the frequency domain. In frequency domain processing, a matrix containing auto-spectrum and cross-spectrum is first used, which is called Cross-Spectral Matrix (CSM). At each frequency of interest, the processing of the array signal gives the energy level at each given spatial scan grid point or each direction of arrival (DOA) of the signal. Thus, the array represents a summed quantity of responses associated with the sound source distribution. This method is suitable for large microphone arrays and is highly adaptable to the test environment. (2) Methods based on high-resolution spectral estimation include autoregressive AR model, minimum variance spectral estimation (MV), and eigenvalue decomposition methods (such as Music algorithm), all of which are calculated by acquiring the signals of the microphone array. Correlation matrix of the spatial spectrum. In theory, the direction of the sound source can be effectively estimated. In practice, to obtain ideal accuracy, it is necessary to pay a large computational cost, and more assumptions are required. When the array is large, this kind of spectral estimation The method requires a large amount of computation, is sensitive to environmental noise, and can easily lead to inaccurate positioning, so it is rarely used in modern large-scale sound source localization systems. (3) The localization technology of time difference of arrival (TDOA), this kind of sound source localization method is generally divided into two steps, first to estimate the time difference of arrival, and obtain the acoustic delay (TDOA) between the elements in the microphone array; Then, the obtained sound arrival time difference is combined with the known spatial position of the microphone array to further determine the position of the sound source. The calculation amount of this method is generally smaller than the first two, which is more conducive to real-time processing, but the positioning accuracy and anti-interference ability are weak, and it is suitable for near-field, single audio source, and non-repetitive signals, such as voice signals, Microsoft XBOX360 The kinect's microphone array (4 one-dimensional arrays with unequal spacing) is a typical TDOA algorithm application.

At present, these traditional sound source localization algorithms are highly dependent on the number of microphone arrays, and their ability to localize sound signals with small signal-to-noise ratios is also poor. In the face of multi-person meetings, cocktail parties and other scenarios, traditional sound source localization is almost impossible. work, not to mention accurately locating the speaker's location, matching the speaker's identity and the content of the speech.

SUMMARY OF THE INVENTION

In view of the deficiencies in the related art, the technical problem to be solved by the present invention is to provide a sound source localization method and device with high accuracy that can prepare the location of the speaker, match the identity of the speaker and the content of the speech. , Speech recognition method, system, storage device and terminal.

In order to solve the above technical problems, the present invention provides a sound source localization method with high accuracy, comprising the following steps: S101, collecting sound signals in the environment in real time; S102, judging whether there is a voice signal in the collected sound signals, If there is, go to step S103, otherwise, go back to step S101; S103, extract all the voice signals, and obtain the sound source position information where each voice signal is located; S104, carry out the sound of each extracted voice signal one by one S105, determine whether the currently recognized voiceprint features have been stored in the voiceprint database, if so, then go to step S108, otherwise, go to step S106; wherein, the voiceprint database stores multiple groups of voiceprint features and each The identity information of the speaker uniquely corresponding to a set of voiceprint features; S106, obtain the image information at the position of the sound source where the voice signal corresponding to the voiceprint feature is located; S107, use the machine self-learning method to perform model training, and determine the voice The speaker and the identity information corresponding to the voiceprint feature are stored, and the corresponding voiceprint feature and the identity information of the speaker are stored in the voiceprint database; S108. Display the sound source position information where the voice signal corresponding to the voiceprint feature is located and the corresponding The identity of the speaker.

Preferably, after performing step S103, first perform steps S103-1 to S103-3, and then perform step S104; S103-1, collect image information at the position of the sound source where each voice signal is located; S103-2, one by one Determine whether someone exists at the corresponding sound source position, if so, execute step S103-3, otherwise, return to step S101; S103-3, determine that the corresponding voice signal at the sound source position where someone exists is a valid voice signal, and then execute Step S104; the voice signal described in step S104 is a valid voice signal.

Preferably, the image information described in step S106 includes: human face feature information and lip movement feature information; after performing step S106, first perform steps S106-1 to S106-4, and then perform steps S107; S106-1 , according to the acquired image information, carry out face recognition and lip movement recognition, determine the current number of voices at this sound source position; S106-2, judge whether the current number of voices is only 1, if so, then execute step S107, Otherwise, execute step S106-3; S106-3, continue to collect the sound signal at the sound source position; S106-4, judge whether there are other voice signals at the sound source position, if so, return to execute step S104, otherwise, S106-2 is repeatedly executed.

The present invention also provides a sound source localization device with high accuracy, comprising: a first sound collection unit: used to collect sound signals in the environment in real time; a first judgment unit: used to judge whether the collected sound signals There is a voice signal; voice extraction and positioning unit: when there is a voice signal in the collected voice signal, it is used to extract all voice signals, and obtain the position information of the sound source where each voice signal is located; voiceprint recognition unit: use Perform voiceprint recognition on each of the extracted voice signals one by one; the second judgment unit: for judging whether the currently identified voiceprint features have been stored in the voiceprint database; wherein, the voiceprint database stores multiple groups of The voiceprint feature and the identity information of the speaker uniquely corresponding to each group of voiceprint features; image acquisition unit: used to obtain the corresponding voiceprint feature when the currently recognized voiceprint feature is not stored in the voiceprint database. Image information at the location of the sound source where the speech signal is located; machine learning unit: used for model training using machine self-learning methods, to determine the speaker and its identity information corresponding to the voiceprint feature, and to compare the corresponding voiceprint feature and utterance. The personal identity information is stored in the voiceprint database; the display unit is used to display the sound source position information of the voice signal corresponding to the voiceprint feature and the corresponding voiceprint when the currently recognized voiceprint feature has been stored in the voiceprint database. The identity of the speaker.

Preferably, it also includes: an image acquisition unit: used to collect the image information at the sound source location where each speech signal is located after extracting all the speech signals and acquiring the sound source location information where each speech signal is located; third Judging unit: used to judge whether there is someone at the corresponding sound source position one by one; valid voice judgment unit: used to determine that the corresponding voice signal at the sound source position where someone exists is valid when there is someone at the corresponding sound source position voice signal, and then perform voiceprint recognition on all valid voice signals one by one.

Preferably, the image information in the image acquisition unit includes: human face feature information and lip movement feature information; the sound source localization device with high accuracy includes: a sounding number counting unit: used for when When the currently recognized voiceprint feature is not stored in the voiceprint database, after obtaining the image information at the sound source position where the voice signal corresponding to the voiceprint feature is located, perform face recognition and lip movement recognition according to the obtained image information , determine the current number of voices at the sound source position; the fourth judgment unit: used to judge whether the current number of voices is only one, if so, then use the machine self-learning method for model training to determine the corresponding voiceprint feature. Speakers and their identity information, and store the corresponding voiceprint features and speaker identity information in the voiceprint database; the second sound collection unit: used to continue to collect the sound when the current number of speakers is more than one. The sound signal at the position of the sound source; the fifth judgment unit: used to judge whether there are other voice signals at the position of the sound source, if so, perform voiceprint recognition on each of the extracted voice signals, otherwise, repeat the judgment Whether the current number of voices is only 1.

The present invention also provides a highly targeted speech recognition method, comprising the following steps: S10, adopting a sound source localization method to determine the location information of the sound source where the speech signal is located and the identity information of the corresponding speaker; S20, responding to The voice recognition command converts the voice content corresponding to the designated speaker into text content and displays it; the sound source localization method is the above-mentioned sound source localization method with high accuracy.

The present invention also provides a highly targeted speech recognition system, comprising: a sound source localization device: used for using a sound source localization method to determine the location information of the sound source where the speech signal is located and the identity information of the corresponding speaker; Conversion module: used to convert the voice content corresponding to the designated speaker into text content in response to the voice recognition command, and display it; the sound source localization device is the above-mentioned sound source with high accuracy Positioning means.

The present invention also provides a storage device, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor and executing the above-mentioned highly targeted speech recognition method.

The present invention also provides a terminal, comprising: a processor, adapted to implement various instructions; and a storage device, adapted to store a plurality of instructions, the instructions are adapted to be loaded by the processor and execute the above-mentioned highly targeted speech recognition method.

The beneficial technical effect of the present invention is:

1. The present invention first collects various sound signals in the environment through the first sound acquisition unit in real time, and then judges whether there are voice signals in these collected sound signals, if not, then explain that these sound signals are not the desired target signals, No processing is performed; if so, on the one hand, the position of the sound source of these voice signals is obtained, and on the other hand, voiceprint recognition is performed on these voice signals one by one. Since the vocal organs (such as tongue, teeth, larynx, lungs, nasal cavity, etc.) used by each vocal body when speaking are different in size and shape, the voiceprint characteristics of any two vocal bodies are different. Therefore, for each voice signal in the present invention, as long as they come from different speakers, they uniquely correspond to a set of voiceprint features. When the voiceprint feature corresponding to a voice signal is obtained, first determine whether the voiceprint feature has been stored in the voiceprint database. The identity of the speaker of the voiceprint feature has also been confirmed, then the current location information of the corresponding speaker and his/her identity information are displayed directly through the display unit; if not, it can only be collected through on-site methods. To confirm and match the position and identity of the speaker, first obtain the image information at the sound source position where the voice signal is located through the image acquisition unit, and then use the machine self-learning method to train the speaker model, and finally determine the voiceprint feature Corresponding speaker and its identity information, and store the corresponding voiceprint features and speaker identity information in the voiceprint database, and then display the current location information of the corresponding speaker and his/her identity information through the display unit . In this way, it is possible to know exactly who is speaking and where. If you still want to know who said what, you only need to send a request to the voice conversion module, and the voice conversion module can convert the voice content corresponding to the specified speaker into text content and display it to the requester. The present invention adds voiceprint recognition and image recognition on the basis of traditional sound source localization, which can accurately locate the speaker's position, match the speaker's identity and the content of the speech, and is less dependent on the number of microphones and less sensitive to low The signal-to-noise ratio of ambient sound source localization is accurate, and it can fully adapt to environments with complex sound sources such as multi-person meetings and cocktail parties.

2. In the present invention, after it is determined that there is a voice signal in the collected sound signal, the image information at the position of the sound source where each voice signal is located can be collected first through the image acquisition unit, and then it is determined whether there are people at these sound source positions. If there is someone, follow-up processing is performed on these voice signals, which eliminates the interference of sound sources that do not come from the human body (such as radio sound, TV sound, etc.), and improves the accuracy and recognition efficiency.

3. In the present invention, when the location and identity of the speaker need to be collected on-site, the image information obtained by the image acquisition unit on the spot includes the person's facial feature information and lip movement feature information. In this way, according to the obtained face Use facial feature information and lip motion feature information to perform face recognition and lip motion recognition, determine the current number of speakers at the sound source location, and then determine whether the current number of speakers is only 1, if it is 1, then directly carry out the machine Self-learning; if there are more than one, first judge whether there are other voice signals at the sound source position, if so, perform voiceprint recognition directly, if not, it will loop to judge the number of voices until the number of voices is 1. Machine self-learning can finally uniquely determine the identity and position of the current speaker, thus avoiding the interference of multiple people speaking at the same or adjacent positions at the same time but unable to distinguish who the currently collected voiceprint belongs to. .

Description of drawings

The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings. The same reference numerals refer to the same parts throughout the drawings. The drawings are not intentionally scaled to actual size, and the emphasis is on illustrating the gist of the present invention.

1 is a schematic flowchart of a sound source localization method with high accuracy provided in Embodiment 1 of the present invention;

2 is a schematic structural diagram of a sound source localization device with high accuracy according to Embodiment 1 of the present invention;

3 is a schematic flowchart of a highly targeted speech recognition method provided by Embodiment 1 of the present invention;

4 is a schematic structural diagram of a highly targeted speech recognition system provided by Embodiment 1 of the present invention;

5 is a schematic flowchart of a sound source localization method with high accuracy provided in Embodiment 2 of the present invention;

6 is a schematic structural diagram of a sound source localization device with high accuracy according to Embodiment 2 of the present invention;

7 is a schematic flowchart of a sound source localization method with high accuracy provided by Embodiment 3 of the present invention;

8 is a schematic structural diagram of a sound source localization device with high accuracy provided by Embodiment 3 of the present invention;

9 is a schematic flowchart of a sound source localization method with high accuracy provided in Embodiment 4 of the present invention;

10 is a schematic structural diagram of a sound source localization device with high accuracy provided in Embodiment 4 of the present invention;

In the figure: 101 is the first sound acquisition unit, 102 is the first judgment unit, 103 is the voice extraction and positioning unit, 104 is the voiceprint recognition unit, 105 is the second judgment unit, 106 is the image acquisition unit, 107 is the machine learning unit unit, 108 is a display unit, 109 is an identity marking unit, 103-1 is an image acquisition unit, 103-2 is a third judgment unit, 103-3 is a valid voice judgment unit, 106-1 is a voice counting unit, 106- 2 is a fourth judgment unit, 106-3 is a second sound collection unit, 106-4 is a fifth judgment unit, 10 is a sound source localization device, and 20 is a voice conversion module.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all of the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention .

Next, the present invention is described in detail with reference to the schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of description, the schematic diagrams are only examples, which should not limit the protection scope of the present invention.

An embodiment of the high-accuracy sound source localization method, device, speech recognition method, system, storage device, and terminal will be described in detail below with reference to the accompanying drawings.

Example 1

FIG. 1 is a schematic flowchart of a sound source localization method with high accuracy provided by Embodiment 1 of the present invention. As shown in FIG. 1 , the sound source localization method with high accuracy may include the following steps:

S101. Collect sound signals in the environment in real time.

S102 , judging whether there is a voice signal in the collected sound signal, if so, execute step S103 , otherwise, return to step S101 .

S103 , extracting all the voice signals, and acquiring the position information of the sound source where each voice signal is located.

S104: Perform voiceprint recognition on each of the extracted voice signals one by one.

S105, determine whether the currently recognized voiceprint features have been stored in the voiceprint database, if so, go to step S108, otherwise, go to step S106; wherein, there are multiple groups of voiceprint features and each group of voiceprint features stored in the voiceprint database The identity information of the speaker uniquely corresponding to the pattern feature.

S106: Acquire image information at the sound source position where the voice signal corresponding to the voiceprint feature is located.

S107. Use a machine self-learning method to perform model training, determine a speaker corresponding to the voiceprint feature and its identity information, and store the corresponding voiceprint feature and speaker identity information in a voiceprint database.

S108: Display the sound source position information where the voice signal corresponding to the voiceprint feature is located and the identity information of the corresponding speaker.

Correspondingly, this embodiment also provides a sound source localization device with high accuracy. FIG. 2 is a schematic structural diagram of the sound source localization device with high accuracy provided in Embodiment 1 of the present invention, as shown in FIG. 2 , A sound source localization device with high accuracy, which can include:

The first sound collection unit 101: used to collect sound signals in the environment in real time.

The first judging unit 102: for judging whether there is a voice signal in the collected sound signal.

Voice extraction and positioning unit 103 : for extracting all the voice signals when there are voice signals in the collected voice signals, and acquiring the position information of the sound source where each voice signal is located.

Voiceprint recognition unit 104: for performing voiceprint recognition on each extracted voice signal one by one.

Second judging unit 105: for judging whether the currently recognized voiceprint feature has been stored in the voiceprint database; wherein, the voiceprint database stores multiple sets of voiceprint features and the uniquely corresponding utterance of each set of voiceprint features person's identifiable information.

The image acquisition unit 106 is configured to acquire image information at the position of the sound source where the voice signal corresponding to the voiceprint feature is located when the currently recognized voiceprint feature is not stored in the voiceprint database.

The machine learning unit 107 is used to perform model training by using the machine self-learning method, determine the speaker corresponding to the voiceprint feature and its identity information, and store the corresponding voiceprint feature and speaker identity information in the voiceprint database.

Display unit 108: used to display the sound source location information of the voice signal corresponding to the voiceprint feature and the identity information of the corresponding speaker when the currently recognized voiceprint feature has been stored in the voiceprint database.

In addition, this embodiment also provides a highly targeted speech recognition method. FIG. 3 is a schematic flowchart of a highly targeted speech recognition method provided in Embodiment 1 of the present invention. As shown in FIG. A highly targeted speech recognition method may include the following steps:

S10. Using a sound source localization method, determine the position information of the sound source where the voice signal is located and the identity information of the corresponding speaker.

S20. In response to the voice recognition command, convert the voice content corresponding to the designated speaker into text content and display it.

The sound source localization method is the above-mentioned sound source localization method with high accuracy.

Correspondingly, this embodiment also provides a highly targeted speech recognition system, and FIG. 4 is a schematic structural diagram of a highly targeted speech recognition system provided in Embodiment 1 of the present invention, as shown in FIG. 4 , Highly targeted speech recognition systems that can include:

The sound source localization device 10 is used for determining the position information of the sound source where the voice signal is located and the identity information of the corresponding speaker by adopting the sound source localization method.

Voice conversion module 20: used to convert the voice content corresponding to the specified speaker into text content in response to the voice recognition command, and display it.

The sound source localization device 10 is the above-mentioned sound source localization device with high accuracy.

In this embodiment, the first sound collecting unit 101 first collects various sound signals in the environment in real time, and then judges whether there are voice signals in the collected sound signals, if not, it means that these sound signals are not the intended target signals, No processing is performed; if so, on the one hand, the position of the sound source of these voice signals is obtained, and on the other hand, voiceprint recognition is performed on these voice signals one by one. Since the vocal organs (such as tongue, teeth, larynx, lungs, nasal cavity, etc.) used by each vocal body when speaking are different in size and shape, the voiceprint characteristics of any two vocal bodies are different. Therefore, for each voice signal in the present invention, as long as they come from different speakers, they uniquely correspond to a set of voiceprint features. When the voiceprint feature corresponding to a voice signal is obtained, first determine whether the voiceprint feature has been stored in the voiceprint database. The identity of the speaker of the voiceprint feature has also been confirmed, then the current location information of the corresponding speaker and his/her identity information are directly displayed through the display unit 108; way to confirm and match the position and identity of the speaker, first obtain the image information at the sound source position where the voice signal is located through the image acquisition unit 106, and then use the machine self-learning method to perform model training on the speaker, and finally determine the sound source. The speaker and the identity information corresponding to the fingerprint feature are stored, and the corresponding voiceprint feature and speaker identity information are stored in the voiceprint database, and then the display unit 108 is used to display the current location information of the corresponding speaker and his/her identity information. In this way, it is possible to know exactly who is speaking and where. If you still want to know who said what, you only need to send a request to the voice conversion module 20, and the voice conversion module 20 can convert the voice content corresponding to the specified speaker into text and display it to the requester. The present invention adds voiceprint recognition and image recognition on the basis of traditional sound source localization, which can accurately locate the speaker's position, match the speaker's identity and the content of the speech, and is less dependent on the number of microphones and less sensitive to low The signal-to-noise ratio of ambient sound source localization is accurate, and it can fully adapt to environments with complex sound sources such as multi-person meetings and cocktail parties.

During specific implementation, the display of the location information of the speaker described in the present invention can be displayed in the form of a map, and the core geographic location of the map can be the location of the user wearing the device/device of the present invention. All speakers The location of the user is marked one by one around the user's location on the map for real-time display. Specifically, the identity information of the speaker described in the present invention may be the avatar of the speaker or the name of the speaker. Before using the device/device of the present invention, the voiceprint information and identity information of the pre-speaker can be collected in advance, so that matching can be performed directly during use, saving a lot of time for on-site collection and matching.

Embodiment 2

FIG. 5 is a schematic flowchart of a sound source localization method with high accuracy provided by Embodiment 2 of the present invention. As shown in FIG. 5 , on the basis of Embodiment 1, the present invention may first perform step S103 Steps S103-1 to S103-3 are executed, and then step S104 is executed.

S103-1: Collect image information at the sound source position where each voice signal is located.

S103-2: Determine whether there is a person at the corresponding sound source position one by one, if yes, execute step S103-3, otherwise, return to step S101.

S103-3, determine that the voice signal corresponding to the sound source position where someone exists is a valid voice signal, and then execute step S104;

The voice signal described in step S104 is a valid voice signal.

Correspondingly, this embodiment also provides a sound source localization device with high accuracy. FIG. 6 is a schematic structural diagram of a sound source localization device with high accuracy provided in Embodiment 2 of the present invention, as shown in FIG. 6 . As shown, the sound source localization device with high accuracy may further include:

Image acquisition unit 103-1: used to collect image information at the sound source location where each speech signal is located after extracting all the speech signals and acquiring the sound source location information where each speech signal is located.

The third judging unit 103-2 is used for judging whether there is a person at the corresponding sound source position one by one.

Valid voice determination unit 103-3: used to determine that the corresponding voice signal at the sound source position where someone exists is a valid voice signal when someone exists at the corresponding sound source position, and then all valid voice signals are sounded one by one. Pattern recognition.

In this embodiment, after it is determined that there are voice signals in the collected sound signals, the image acquisition unit 103-1 may first collect image information at the sound source positions where each voice signal is located, and then determine the positions of these sound source positions. Whether someone exists, and if someone exists, these voice signals are then processed, which eliminates the interference of sound sources that do not come from the human body (such as radio sound, TV sound, etc.), and improves accuracy and recognition efficiency.

Embodiment 3

FIG. 7 is a schematic flowchart of a sound source localization method with high accuracy provided by Embodiment 3 of the present invention. As shown in FIG. 7 , on the basis of Embodiment 1, the image information described in step S106 may include: Human facial feature information and lip movement feature information.

In the present invention, after step S106 is performed, steps S106-1 to S106-4 are performed first, and then step S107 is performed:

S106-1. Perform face recognition and lip movement recognition according to the acquired image information, and determine the current number of speakers at the sound source position.

S106-2, judging whether the current number of speakers is only one, if so, go to step S107; otherwise, go to step S106-3.

S106-3. Continue to collect the sound signal at the sound source position.

S106-4: Determine whether there are other voice signals at the sound source position, if so, return to step S104, otherwise, repeat S106-2.

Correspondingly, this embodiment also provides a sound source localization device with high accuracy. FIG. 8 is a schematic structural diagram of a sound source localization device with high accuracy provided in Embodiment 3 of the present invention, as shown in FIG. 8 . As shown, on the basis of Embodiment 1, the image information described in the image acquisition unit 106 may include: human facial feature information and lip movement feature information; the sound source localization device with high accuracy may include:

The number of voice counting unit 106-1 is used to obtain the image information at the position of the sound source where the voice signal corresponding to the voiceprint feature is located when the currently recognized voiceprint feature is not stored in the voiceprint database. Image information, perform face recognition and lip movement recognition, and determine the current number of speakers at the sound source location.

The fourth judgment unit 106-2 is used to judge whether the current number of speakers is only one, and if so, then use the machine self-learning method to perform model training, determine the speaker corresponding to the voiceprint feature and its identity information, and use the machine self-learning method for model training. Corresponding voiceprint features and speaker identity information are stored in the voiceprint database.

The second sound collection unit 106-3: configured to continue to collect the sound signal at the sound source position when the current number of speakers is more than one.

The fifth judgment unit 106-4 is used for judging whether there are other voice signals at the sound source position. If so, each voice signal extracted will be subjected to voiceprint recognition, otherwise, it will be repeatedly judged whether the current number of voices is Only 1.

In this embodiment, when the location and identity of the speaker needs to be collected on-site, the image information obtained by the image obtaining unit 106 on the spot includes the facial feature information and lip movement feature information of the person. Use facial feature information and lip motion feature information to perform face recognition and lip motion recognition, determine the current number of speakers at the sound source location, and then determine whether the current number of speakers is only 1, if it is 1, then directly carry out the machine Self-learning; if there are more than one, first judge whether there are other voice signals at the sound source position, if so, perform voiceprint recognition directly, if not, it will loop to judge the number of voices until the number of voices is 1. Machine self-learning can finally uniquely determine the identity and position of the current speaker, thus avoiding the interference of multiple people speaking at the same or adjacent positions at the same time but unable to distinguish who the currently collected voiceprint belongs to. .

Embodiment 4

FIG. 9 is a schematic flowchart of a sound source localization method with high accuracy provided by Embodiment 4 of the present invention. As shown in FIG. 9 , on the basis of Embodiment 1, the sound source localization method with high accuracy is further Can include the following steps:

S109. In response to the speaker identity marking command, mark and display the identity information of the designated speaker.

Correspondingly, this embodiment also provides a sound source localization device with high accuracy. FIG. 10 is a schematic structural diagram of a sound source localization device with high accuracy provided in Embodiment 4 of the present invention, as shown in FIG. 10 . As shown, on the basis of Example 1,

A sound source localization device with high accuracy, further comprising:

Identity marking unit 109: used to mark and display the identity information of the designated speaker in response to the speaker's identity marking command.

After the location and identity of the speaker are determined, the identity of the speaker can be marked according to actual needs for easy identification.

Correspondingly, the present invention also provides a storage device in which a plurality of instructions are stored, and the instructions are suitable for being loaded by the processor and executing the above-mentioned highly targeted speech recognition method.

The storage device may be a computer-readable storage medium, which may include: ROM, RAM, magnetic disk or optical disk, and the like.

Correspondingly, the present invention also provides a terminal, which may include:

a processor adapted to implement the instructions; and

A storage device adapted to store a plurality of instructions adapted to be loaded by a processor and execute the highly targeted speech recognition method as described above.

The terminal can be any device capable of sound source localization and speech recognition, and the device can be various wearable terminal devices, such as glasses, bracelets, etc., which can be specifically implemented by software and/or hardware. The processor can be a multi-core processor with GPU or NPU, such as Qualcomm, Rockchip, MediaTek and other processors.

During specific implementation, the first sound collection unit 101 and the second sound collection unit 106-3 may be linear arrays, annular arrays, cardioid arrays, spiral arrays, irregular arrays, etc. based on electret microphones, or may be Linear arrays, annular arrays, cardioid arrays, helical arrays, irregular arrays, etc. based on MEMS microphones. The image acquisition unit 106 and the image acquisition unit 103-1 can be a monocular camera, a binocular camera, a multi-eye camera, a depth camera, or the like. The display unit 108 can be a display screen such as an OLED, an LCD, etc., a display module such as a prism module, a free-form surface, an optical waveguide, etc., or an LED lamp or other display modules can be adopted.

For a noisy environment where many people are talking at the same time, or an environment where the distance between the speaker and the listener is relatively far, compared with the traditional sound source localization, the present invention has greater advantages, especially for the disabled person with hearing impairment. , the advantages of the present invention are particularly prominent. Based on machine vision and voiceprint recognition technology, the present invention combines the audio data and image data collected on the spot to perform sound source localization, voiceprint recognition, face recognition and lip movement recognition, and determine how many corresponding sound source positions there are. Individuals, how many individuals are speaking, and who are speaking, and then accurately obtain the location information of multiple sound sources and the identity information of the speaker. The invention solves the problems of inaccurate positioning and inability to obtain desired human voices in the traditional sound source localization algorithm for multi-person conferences, cocktail parties and other application environments. Accurate sound source localization with outstanding substantive features and significant progress.

In the description of the present invention, it can be understood that the related features in the above-mentioned methods, apparatuses and systems may refer to each other. In addition, "first", "second", etc. in the above-mentioned embodiments are used for distinguishing each embodiment or for description purpose, and do not represent the advantages and disadvantages of each embodiment, nor should they be interpreted as indicating or implying that they are relatively important. nature or implicitly indicate the number of technical features indicated. Thus, features defined as "first", "second" etc. may expressly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and modules described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a device is apparent from the above description. Furthermore, the present invention is not directed to any particular programming language. It should be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the embodiments provided in this application, it should be understood that the disclosed systems and methods may be implemented in other manners. The system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (10)

1. A sound source localization method with high accuracy, characterized by: the method comprises the following steps:

s101, collecting sound signals in an environment in real time;

s102, judging whether a voice signal exists in the collected sound signal, if so, executing a step S103, otherwise, returning to the step S101;

s103, extracting all voice signals and acquiring sound source position information of each voice signal;

s104, performing voiceprint recognition on each extracted voice signal one by one;

s105, judging whether the currently identified voiceprint features are stored in a voiceprint database, if so, executing a step S108, otherwise, executing a step S106; the voiceprint database stores a plurality of groups of voiceprint characteristics and the identity information of the speaker uniquely corresponding to each group of voiceprint characteristics;

s106, acquiring image information of a sound source position where the voice signal corresponding to the voiceprint feature is located;

s107, performing model training by using a machine self-learning method, determining a speaker corresponding to the voiceprint characteristics and identity information thereof, and storing the corresponding voiceprint characteristics and the identity information of the speaker in a voiceprint database;

and S108, displaying the sound source position information of the voice signal corresponding to the voiceprint characteristic and the identity information of the corresponding speaker.

2. The sound source localization method with high accuracy according to claim 1, characterized in that: after the step S103 is completed, the steps S103-1 to S103-3 are executed, and then the step S104 is executed;

s103-1, collecting image information of a sound source position where each voice signal is located;

s103-2, judging whether people exist at the corresponding sound source positions one by one, if so, executing the step S103-3, otherwise, returning to the step S101;

s103-3, judging that the voice signal corresponding to the sound source position with the existence of people is an effective voice signal, and then executing the step S104;

the voice signal described in step S104 is an effective voice signal.

3. The sound source localization method with high accuracy according to claim 1, characterized in that: the image information in step S106 includes: face feature information and lip movement feature information of the person;

after the step S106 is executed, the steps S106-1 to S106-4 are executed, and then the step S107 is executed;

s106-1, carrying out face recognition and lip movement recognition according to the acquired image information, and determining the current number of the sounders at the sound source position;

s106-2, judging whether the number of the current sounders is only 1, if so, executing a step S107, otherwise, executing a step S106-3;

s106-3, continuously collecting the sound signals at the sound source position;

s106-4, judging whether other voice signals exist at the sound source position, if so, returning to the step S104, otherwise, repeatedly executing the step S106-2.

4. Sound source localization apparatus with high accuracy, characterized by: the method comprises the following steps:

first sound collection unit (101): the system is used for acquiring sound signals in the environment in real time;

first judgment unit (102): the voice recognition device is used for judging whether a voice signal exists in the collected voice signals;

speech extraction and localization unit (103): the voice recognition system is used for extracting all voice signals and acquiring the position information of a sound source where each voice signal is located when the voice signals exist in the acquired voice signals;

voiceprint recognition unit (104): the voice print recognition module is used for carrying out voice print recognition on each extracted voice signal one by one;

second determination unit (105): the voice print database is used for judging whether the currently identified voice print characteristics are stored in the voice print database; the voiceprint database stores a plurality of groups of voiceprint characteristics and the identity information of the speaker uniquely corresponding to each group of voiceprint characteristics;

image acquisition unit (106): the voice recognition method comprises the steps that when the currently recognized voiceprint features are not stored in a voiceprint database, image information of a sound source position where a voice signal corresponding to the voiceprint features is located is obtained;

machine learning unit (107): the system is used for carrying out model training by utilizing a machine self-learning method, determining a speaker corresponding to the voiceprint characteristics and identity information thereof, and storing the corresponding voiceprint characteristics and the identity information of the speaker in a voiceprint database;

display unit (108): and the voice recognition module is used for displaying the sound source position information of the voice signal corresponding to the voiceprint characteristics and the identity information of the corresponding speaker when the currently recognized voiceprint characteristics are stored in the voiceprint database.

5. The sound source localization device with high accuracy according to claim 4, wherein: further comprising:

image acquisition unit (103-1): the voice recognition system is used for acquiring image information of a sound source position where each voice signal is located after all voice signals are extracted and sound source position information where each voice signal is located is acquired;

third judging unit (103-2): the system is used for judging whether people exist at the corresponding sound source positions one by one;

valid speech determination unit (103-3): and when a person exists at the corresponding sound source position, judging the voice signal corresponding to the sound source position where the person exists as an effective voice signal, and then carrying out voiceprint recognition on all the effective voice signals one by one.

6. The sound source localization device with high accuracy according to claim 4, wherein: the image information in the image acquisition unit (106) includes: face feature information and lip movement feature information of the person; the sound source localization apparatus with high accuracy includes:

an utterance people counting unit (106-1): the voice recognition system is used for carrying out face recognition and lip movement recognition according to the acquired image information after acquiring the image information of the sound source position where the voice signal corresponding to the voiceprint feature is located when the currently recognized voiceprint feature is not stored in the voiceprint database, and determining the current number of the voice producing people at the sound source position;

fourth judging unit (106-2): the voice print recognition system is used for judging whether the number of the current voice speakers is only 1, if so, performing model training by using a machine self-learning method, determining the voice speakers corresponding to the voice print characteristics and identity information thereof, and storing the corresponding voice print characteristics and the identity information of the voice speakers in a voice print database;

a second sound collection unit (106-3): when the number of the current sounders is not more than 1, continuously acquiring the sound signals at the sound source position;

fifth judging unit (106-4): and the voice recognition module is used for judging whether other voice signals exist at the position of the sound source, if so, performing voiceprint recognition on each extracted voice signal one by one, and otherwise, repeatedly judging whether the number of the current uttered people is only 1.

7. A speech recognition method with high pertinence, characterized by: the method comprises the following steps:

determining the sound source position information of the voice signal and the identity information of a corresponding speaker by adopting a sound source positioning method;

responding to the voice recognition command, converting the voice content corresponding to the appointed speaker into character content, and displaying the character content;

the sound source localization method according to any one of claims 1 to 3, wherein the sound source localization method is a sound source localization method with high accuracy.

8. A speech recognition system with high pertinence, characterized by: the method comprises the following steps:

sound source localization device (10): the method comprises the steps of determining sound source position information of a voice signal and identity information of a corresponding speaker by adopting a sound source positioning method;

speech conversion module (20): the voice recognition device is used for responding to a voice recognition command, converting voice content corresponding to a specified speaker into character content and displaying the character content;

the sound source localization apparatus (10) is the sound source localization apparatus with high accuracy according to any one of claims 4 to 6.

9. A storage device having a plurality of instructions stored therein, characterized in that: the instructions are adapted to be loaded by a processor and to perform a sound source localization method with high accuracy as claimed in any of claims 1-3.

10. A terminal, characterized in that: the method comprises the following steps:

a processor adapted to implement instructions; and

a storage device adapted to store a plurality of instructions adapted to be loaded by a processor and to perform a sound source localization method with high accuracy as claimed in any of claims 1-3.

Images