CN106714064B

CN106714064B - Real-time processing method for cochlear prosthesis audio

Info

Publication number: CN106714064B
Application number: CN201710111151.8A
Authority: CN
Inventors: 王宁远; 田春; 李方波; 孙晓安; 黄穗; 李晓波
Original assignee: Zhejiang Nurotron Biotechnology Co ltd
Current assignee: Zhejiang Nurotron Biotechnology Co ltd
Priority date: 2017-02-28
Filing date: 2017-02-28
Publication date: 2022-06-17
Anticipated expiration: 2037-02-28
Also published as: CN106714064A

Abstract

The invention discloses a real-time audio processing system and a real-time audio processing method for a cochlear implant, wherein the system comprises a sound signal acquisition unit, a DSP processing unit, a DSP signal output unit, a DSP-to-PC data conversion unit, a PC end signal real-time acquisition unit, a PC end real-time processing unit and a PC end real-time debugging unit, wherein the sound signal acquisition unit is connected with the DSP processing unit and comprises an acquisition module and an analog-to-digital conversion module; the DSP processing unit is connected with the DSP signal output unit, at least comprises a fast Fourier transform module, a frequency division processing module and an envelope extraction module, buffers the processed signals to the DSP signal output unit, and outputs the signals in a serial I2S format. The invention can output, debug and demonstrate the processing result and process of the DSP in real time.

Description

Real-time processing method for cochlear prosthesis audio

Technical Field

The invention belongs to the field of signal processing, and particularly relates to a real-time processing method for cochlear prosthesis audio.

Background

The artificial cochlea is mainly a hearing aid device for helping patients with severe and extremely severe deafness to regain hearing, and consists of an external mechanical speech processor and an internal implant. The speech processor collects external voice signals through a microphone, converts the voice signals into pulse signals of each frequency band through voice preprocessing, voice coding strategies and the like, and finally transmits the coded signals in a radio frequency mode; after the implant decoding chip collects the radio frequency signals, the implant decoding chip bypasses outer hair cells of a cochlea to directly stimulate auditory nerves through an electrode array implanted into the cochlea, so that a patient regains hearing.

In the signal processing system of the artificial cochlea, the quality of sound preprocessing and the sound coding algorithm directly determine the listening quality of a patient, so that the research and the optimization of the sound processing algorithm with higher quality are the main means for improving the life quality of an artificial cochlea implant. However, no algorithm development system suitable for the cochlear implant exists at present, which brings great inconvenience to the work of developers. Therefore, an algorithm research and development system suitable for the cochlear implant is provided, the technical development of related industries is promoted, and the life quality of the hearing-impaired people can be improved better.

An algorithm research and development system of an existing artificial cochlear speech processor mainly depends on PC (personal computer) simulation software, a section of collected sound signals are required to be input as signals of an algorithm and are led into the algorithm simulation software, then the whole process is carried out through the simulation software, and a processing result can be detected and demonstrated after the process is finished. The method cannot effectively reflect the influence of the input signal and the change of the surrounding environment on the algorithm output in time, and has no real-time property.

Meanwhile, in the algorithm DSP development and implementation stage of the artificial cochlea speech processor, a particularly convenient DSP algorithm debugging system is unavailable. In order to determine whether the intermediate process of the implementation of the debugging algorithm is correct and to check whether the variation process of some special signals is consistent in the DSP with the simulation stage of the PC algorithm, the signals to be observed are stored in the memory of the DSP to be observed. But due to the limitation of the memory storage capacity, only signals of limited length can be observed, and the real-time performance is not provided.

Disclosure of Invention

In view of this, the present invention provides a real-time cochlear implant audio processing method, which can output, debug and demonstrate the processing results and processes of PC-side simulation software and algorithms in the DSP in real time.

In order to achieve the above object, the present invention provides a cochlear implant audio real-time processing system, which comprises a sound signal acquisition unit, a DSP processing unit, a DSP signal output unit, a DSP-to-PC data conversion unit, a PC end signal real-time acquisition unit, a PC end real-time processing unit and a PC end real-time debugging unit, wherein,

the sound signal acquisition unit is connected with the DSP processing unit and comprises an acquisition module and an analog-to-digital conversion module, and after the acquisition module acquires external sound, a digital signal is output through the analog-to-digital conversion module;

the DSP processing unit is connected with the DSP signal output unit, at least comprises a fast Fourier transform module, a frequency division processing module and an envelope extraction module, caches the processed signals to the DSP signal output unit, and outputs the processed signals in a serial I2S format;

the input of the DSP to PC data conversion unit is connected with the output of the DSP signal output unit, the output of the DSP to PC data conversion unit is connected with the PC end signal real-time acquisition unit, and the DSP to PC data conversion unit at least comprises an I2S-to-USB hardware module;

the real-time signal acquisition unit of the PC end, the real-time processing unit of the PC end and the real-time debugging unit of the PC end are sequentially connected, wherein the real-time signal acquisition unit of the PC end at least comprises a continuous framing sampling module, syllable segments of collected voice signals are output to the real-time processing unit of the PC end to be processed by a voice algorithm, and the real-time debugging unit of the PC end is used for debugging in real time after the processing.

Preferably, the system further comprises a DSP demonstration unit connected to the DSP signal output unit for synchronously demonstrating the sound signal buffered in the DSP signal output unit.

Preferably, the system further comprises a PC end real-time demonstration unit which is connected with the PC end real-time processing unit and synchronously demonstrates the real-time processing result or the processing intermediate process.

Preferably, the real-time signal acquisition unit at the PC end comprises a USB module, a cache module and a PC framing acquisition module which are connected in sequence, the USB module puts the transmitted parallel data stream into the cache module, forms a queue with data in the cache module, sets the length of the queue to change the cache capacity of the cache module, and finally, the PC framing acquisition module performs acquisition.

Based on the above purpose, the invention also provides a real-time cochlear implant audio processing method adopting the system, which comprises the following steps:

after the sound signal acquisition unit acquires a sound signal, the sound signal is converted into a digital signal through the analog-to-digital conversion module and output to the DSP processing unit;

the data required by debugging are extracted and detected, and are transmitted to a PC end through a DSP to PC data conversion unit;

carrying out continuous frame sampling on a PC end to acquire data of a required time length;

and after the voice algorithm processing is carried out, algorithm debugging is carried out.

Preferably, after the sound signal is processed by the DSP processing unit and transmitted to the DSP signal output unit, the sound signal is played and demonstrated by the DSP demonstration unit.

Preferably, after the speech algorithm processing, the processing result or the intermediate process of the processing is synchronously demonstrated.

Preferably, the data of the time length required for continuous framing sampling acquisition at the PC end is acquired by framing a voice signal in real time by setting a sampling rate, the number of sampling points of each frame and the number of channels of voice, wherein the data is transmitted by the USB module and put in the buffer module, then the data in the buffer module forms a queue, the buffer capacity of the buffer module is changed by setting the size of the queue.

The invention has the beneficial effects that: the problem that a method of adopting pre-recorded sound as an algorithm input signal does not have real-time performance when being researched and developed on PC simulation software can be effectively solved; in the process of realizing the digital signal processing system of the algorithm, the problem that only very limited data signals can be observed during algorithm debugging due to insufficient memory of DSP storage data is solved; and the problem of no real-time property exists when the algorithm evaluates and demonstrates.

Drawings

In order to make the object, technical scheme and beneficial effect of the invention more clear, the invention provides the following drawings for explanation:

fig. 1 is a schematic structural diagram of a cochlear implant audio real-time processing system according to embodiment 1 of the present invention;

fig. 2 is a schematic structural diagram of a cochlear implant audio real-time processing system according to embodiment 2 of the present invention;

FIG. 3 is a flowchart illustrating steps of a real-time cochlear implant audio processing method according to an embodiment of the present invention;

FIG. 4 is a waveform diagram of a 1kHz pure tone acquired by a real-time cochlear implant audio processing system according to an embodiment of the present invention;

fig. 5 is a waveform diagram of a vowel "a" collected at the PC side of a cochlear implant audio real-time processing system according to an embodiment of the present invention.

Detailed Description

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Example 1

Referring to fig. 1, a cochlear implant audio real-time processing system according to embodiment 1 of the present invention includes a sound signal collecting unit 10, a DSP processing unit 20, a DSP signal output unit 30, a DSP-to-PC data converting unit 40, a PC-side signal real-time collecting unit 50, a PC-side real-time processing unit 60, and a PC-side real-time debugging unit 70, wherein,

the sound signal acquisition unit 10 is connected with the DSP processing unit 20 and comprises an acquisition module and an analog-to-digital conversion module, wherein the acquisition module acquires external sound and outputs a digital signal through the analog-to-digital conversion module;

the DSP processing unit 20 is connected to the DSP signal output unit 30, and at least includes a fast fourier transform module, a frequency division processing module, and an envelope extraction module, and buffers the processed signal to the DSP signal output unit 30, and the DSP signal output unit 30 outputs the processed signal in a serial I2S format;

the input of the DSP to PC data conversion unit 40 is connected with the output of the DSP signal output unit 30, the output is connected with the PC end signal real-time acquisition unit 50, and the DSP to PC data conversion unit at least comprises an I2S-USB hardware module;

the PC end signal real-time acquisition unit 50, the PC end real-time processing unit 60 and the PC end real-time debugging unit 70 are sequentially connected, wherein the PC end signal real-time acquisition unit 50 at least comprises a continuous sampling module and a framing sampling module, syllable segments of acquired voice signals are output to the PC end real-time processing unit 60 to be processed by a voice algorithm, and the real-time debugging unit 70 is used for real-time debugging after the processing.

Example 2

On the basis of embodiment 1, referring to fig. 2, there is shown a cochlear implant audio real-time processing system according to embodiment 2 of the present invention, further comprising a DSP demonstration unit 80 connected to the DSP signal output unit 30 for demonstrating the sound signal buffered in the DSP signal output unit 30.

And the PC end real-time demonstration unit 90 is connected with the PC end real-time processing unit 60 and synchronously demonstrates the real-time processing result or the intermediate processing process.

In a specific embodiment, the PC-side signal real-time acquisition unit 50 includes a USB module, a cache module, and a PC framing acquisition module, which are connected in sequence, where the USB module puts the transmitted parallel data stream into the cache module, forms a queue with data in the cache module, sets the length of the queue to change the cache capacity of the cache module, and finally, the PC framing acquisition module performs acquisition.

The environmental sound signal is collected by the sound signal collecting unit 10 and transmitted to the DSP, where it may be processed by the DSP processing unit 20, or the original sound signal may be directly transmitted to the DSP signal output unit 30. In the DSP signal output unit 30, the signal may be directly output and demonstrated in the form of audio or electrical stimulation, or the signal may be transmitted to the DSP-to-PC data conversion unit 40 and then collected by the PC-side signal real-time collection unit 50. If the acquired signals are intermediate variables or data processed by a DSP algorithm, real-time algorithm debugging work can be carried out at the PC end; if the collected sound signal is the original sound signal, the real-time processing unit 60 at the PC end can perform simulation processing, and debug and check the data and variables in the processing process, or output and demonstrate the processed data.

A double-microphone array is used in the artificial cochlea speech processor for collecting external environment sound signals, the positions of the double microphones are in tandem relative relation, and the relative horizontal position is 2 cm. The double-microphone array of the voice acquisition unit acquires audio sound signals, converts the audio sound signals into analog electric signals through sound and electricity, and then performs quantitative sampling through the DSP to obtain digital signals which serve as input signals of the whole research and development system.

In the system, a data transmission conversion (I2S-USB) hardware module is set as a master chip, and the DSP is set as a slave chip. Meanwhile, the PC end is provided with the corresponding data transmission bit width and the sampling rate of the recording equipment, and related interface signals are interconnected. Therefore, the voice signals collected by the microphone are output in an I2S format through the DSP digital signal output module, and then converted into USB signals through the voice communication conversion unit and transmitted to the PC terminal. In order to prevent the problem that the rate of reading USB data by the PC is lower than the transmission rate of I2S to cause data loss due to full writing, the I2S end can be provided with cached I2S data with corresponding storage depth; meanwhile, the solution of adjusting the USB transmission rate and accelerating the reading speed can be adopted. It should be noted that the depth of the buffer is related to the problem of voice latency, and the larger the depth is, the longer the latency is, the larger the voice time delay is.

The mode for realizing real-time acquisition of USB digital audio signals at the PC end is divided into: continuous sampling and frame sampling. In order to realize the debugging function of the DSP voice algorithm and solve the problem that the traditional DSP algorithm debugging method only can observe a small amount of data due to the limitation of the storage capacity of the method by adopting the method that the signals needing to be observed are stored in a DSP memory to be derived and observed. In the system, relevant signals inside the DSP are output and a continuous sampling mode is adopted at the PC end, so that whether the change of the relevant signals in the operation process of the DSP is consistent with that in the PC algorithm simulation software or not is observed and judged, and the DSP algorithm debugging function is realized. The mode of realizing continuous sampling from the USB port in PC software is relatively simple, only the corresponding sampling bit width, sampling rate, channel number and equipment ID are needed to be set, then the sampling time length is set, and the sampling data is read, thus completing the continuous acquisition of USB data.

In order to make the PC speech algorithm simulation development have real-time performance, the USB audio data transmitted by the data conversion module needs to be acquired in frames at the PC end. The real-time implementation of speech in speech algorithm processing can only be achieved by dividing a time-continuous speech signal into syllable segments (frames). The specific modes are frame acquisition of voice, frame voice algorithm processing, frame voice effect playing and displaying and the like. Since each frame (syllable segment) is very short in time length, many frames of speech signals are processed and played continuously to be displayed as real-time speech. The parallel data flow transmitted by the USB interface of the physical equipment is firstly put in a cache, and the size of the data volume cached by the physical equipment is set; then, the data in the buffer forms a queue, and the buffer capacity is changed by setting the size of the queue, but it should be noted that the length of the queue directly determines the latency length of the USB voice signal transmitted in real time in the queue, that is, determines whether the voice input in the PC-side voice algorithm is synchronous with the voice generated by the environment; and finally, realizing the framing acquisition of the real-time voice by setting a sampling rate, the number of sampling points of each frame, the number of channels of sound and the like.

In order to realize real-time acquisition, processing and synchronous demonstration, a frame acquisition mode is adopted, syllable segments are composed of multiframe voice signals, and the frame number calculation method of each segment of syllable segment is shown as the following formula:

wherein fs is the sampling rate of the frame-dividing sampling, Period is the execution time length of each syllable segment, and samplespectframe is the number of sampling points of each frame sampling in the frame-dividing sampling.

In the parallel operation, the parallel operation of the acquisition of the voice signals, the processing of the voice algorithm and the real-time synchronous playing of the processed result sound effect is respectively realized by calling related functions. And each function consists of three parts, namely function start, function execution and function stop. The actual work of the function is specifically completed by continuously and repeatedly executing the function execution part. In the PC algorithm, the time synchronization of PC voice sampling, PC voice algorithm processing and PC sound effect playing and displaying is achieved by setting the delay time from the starting function corresponding to each function to the first starting execution of the executed function. The specific parallel pipeline execution flow is as follows: collecting current syllable segments by frames; simultaneously carrying out voice algorithm processing on the voice signal acquired by the previous syllable segment; and simultaneously, the result of the voice signal collected by the previous syllable segment after being processed by the voice algorithm is played and displayed. By the parallel pipeline execution mode, the framing sampling, the voice algorithm processing and the playing display of the short-time syllable segment, the real-time performance of the development of the voice algorithm at the PC end can be realized. The method can not only collect the environmental voice through the microphone array in real time as the input of the algorithm; and the sound effect result of the voice algorithm can be monitored and displayed in real time.

Corresponding to the system, the invention also provides a real-time cochlear implant audio processing method, the flow chart of which is shown in fig. 3, comprising the following steps:

s101, after the sound signal acquisition unit acquires a sound signal, the sound signal is converted into a digital signal through an analog-to-digital conversion module and output to a DSP (digital signal processor);

s102, the data is processed by the DSP processing unit and then transmitted to the DSP signal output unit, the data required by debugging is extracted and detected, and the data is transmitted to the PC end through the DSP-to-PC data conversion unit;

s103, carrying out continuous frame sampling on the PC end to acquire data with required time length;

and S104, carrying out algorithm debugging after carrying out voice algorithm processing.

Furthermore, after the S102 is processed by the DSP processing unit and transmitted to the DSP signal output unit, the DSP demonstration unit plays and demonstrates the sound signal.

And S104, after the voice algorithm processing is carried out, synchronously demonstrating the processing result or the middle process of the processing.

S103, carrying out continuous framing sampling and collecting data with a time length required by the PC end, namely, transmitting parallel data streams by the USB module and placing the data streams in the buffer module, then forming a queue by the data in the buffer module, changing the buffer capacity of the buffer module by setting the size of the queue, and framing and collecting voice signals in real time by setting a sampling rate, the number of sampling points of each frame and the number of channels of voice.

For specific embodiments, reference is made to the above system embodiments, which are not described herein in detail.

To illustrate the beneficial effects of the present invention, it is determined whether it is disturbed by the device intrinsic noise during the transfer conversion of the digital speech signal. Two experimental protocols were designed for validation.

Referring to fig. 4, it is a waveform diagram of signal acquisition and test of a cochlear implant audio real-time processing system according to an embodiment of the present invention; both the solid line and the circle are 1kHz pure tone signals.

Firstly, a standard pure tone sinusoidal signal with the amplitude of 0.3, the frequency of 1kHz and the sampling rate of 96kHz is generated at the PC end and is connected to the DSP audio input interface through an audio line so as to replace the environmental sound input collected by the double microphones. Then, the system collects the transmitted digital voice signal at the other PC terminal.

The solid line is the original input sinusoidal pure tone signal generated at the PC terminal, with a sampling rate of 96kHz, a frequency of 1kHz and an amplitude of 0.3. The circle is a sine signal which is finally collected at the PC end and is output through I2S after being subjected to analog-to-digital conversion sampling of the DSP at the sampling rate of 16 kHz. It can be seen that the sound signal acquisition system is basically correct in the data acquisition and transmission process, and has no frequency drift and distortion.

Fig. 5 is a waveform diagram of a vowel "a" test of a cochlear implant audio real-time processing system according to an embodiment of the present invention.

In order to prevent the microphone from picking up background noise in the environment, a connection to the DSP audio input interface via an audio line is used as a signal input source instead of the microphone. The vowel "a" is played at the PC end, the sampling rate of the original voice is 16kHz, and the duration is about 0.5 s. And finally, outputting the acquired signals through an I2S interface of the DSP, acquiring digital voice signals at a PC end through the system, and comparing and analyzing time domain oscillograms of the original voice and the acquired voice by utilizing PC related software.

Curve 1 is the time domain waveform of the speech signal collected in the PC after the transfer conversion through the system collection at the sampling rate of 16kHz, and curve 2 is the time domain waveform of the speech input signal of the original standard mandarin vowel "a" diphone, with the sampling rate of 16 kHz. Comparing the curves 1 and 2, it can be known that the voice signal collected by the system voice signal real-time collecting and processing system is basically consistent with the original signal, but is affected by very low inherent noise of the circuit and transmission noise of the audio transmission line. These influences are judged acoustically without lowering the rate of Chinese character recognition.

Finally, it is noted that the above-mentioned preferred embodiments illustrate rather than limit the invention, and that, although the invention has been described in detail with reference to the above-mentioned preferred embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the invention as defined by the appended claims.

Claims

1. A real-time processing method of cochlear implant audio is characterized by comprising the following steps:

the data processing method comprises the steps that data to be debugged are transmitted to a DSP signal output unit after being processed by a DSP processing unit, extracted and detected, and transmitted to a PC end through a DSP-to-PC data conversion unit, specifically, a data transmission conversion I2S-to-USB hardware module is set as a main chip, data transmission is converted into I2S-to-USB, the DSP is a slave chip, meanwhile, the corresponding data transmission bit width and the sampling rate of a recording device are set at the PC end, relevant interface signals of the recording device are interconnected, and in order to prevent the problem that the data is fully written and lost due to the fact that the rate of reading USB data by the PC is lower than the I2S transmission rate, cache I2S data with corresponding storage depth is set at the I2S end;

the parallel data flow transmitted by the USB interface of the physical device is firstly put in a buffer memory, the size of the data volume of the buffer memory is set, then the data in the buffer memory forms a queue, the buffer memory capacity is changed by setting the size of the queue, and the data with the time length required by continuous framing sampling acquisition is carried out at the PC end;

after the voice algorithm processing is carried out, algorithm debugging is carried out, specifically, parallel operations of voice signal acquisition, voice algorithm processing and real-time synchronous playing of processed result sound effect are respectively realized by calling related functions at a PC end, and the specific parallel flow execution flow is as follows: acquiring a current syllable segment by frames, simultaneously carrying out voice algorithm processing on a voice signal acquired by a previous syllable segment, and simultaneously carrying out result playing and displaying after the voice algorithm processing on the voice signal acquired by the previous syllable segment;

the cochlear implant audio real-time processing system applied by the method comprises a sound signal acquisition unit, a DSP processing unit, a DSP signal output unit, a DSP-to-PC data conversion unit, a PC end signal real-time acquisition unit, a PC end real-time processing unit and a PC end real-time debugging unit, wherein,

the sound signal acquisition unit is connected with the DSP processing unit and comprises an acquisition module and an analog-to-digital conversion module, the acquisition module acquires external sound and outputs a digital signal through the analog-to-digital conversion module, specifically, a double-microphone array is used for acquiring external environment sound signals, the positions of double microphones of the acquisition module show a front-to-back relative relation, the double-microphone array of the acquisition module acquires audio sound signals and converts the audio sound signals into analog electric signals through sound and electricity, and then the analog-to-digital conversion module performs quantization sampling to obtain the digital signals which serve as input signals of the whole system;

the PC end signal real-time acquisition unit, the PC end real-time processing unit and the PC end real-time debugging unit are sequentially connected, wherein the PC end signal real-time acquisition unit at least comprises a continuous framing sampling module, collects syllable segments of voice signals and outputs the syllable segments to the PC end real-time processing unit for voice algorithm processing, and the PC end real-time debugging unit carries out real-time debugging after the processing;

the DSP demonstration unit is connected with the DSP signal output unit and is used for synchronously demonstrating the sound signals cached in the DSP signal output unit;

the real-time demonstration device also comprises a PC end real-time demonstration unit which is connected with the PC end real-time processing unit and synchronously demonstrates the real-time processing result or the middle processing process;

the environmental sound signals are collected and transmitted to the DSP through the sound signal collecting unit, the environmental sound signals can be processed through the DSP processing unit in the DSP, or original sound signals can be directly transmitted to the DSP signal output unit, the signals can be directly output and demonstrated in an audio or electric stimulation mode in the DSP signal output unit, or the signals are transmitted to the DSP to PC data conversion unit and then collected through the PC end signal real-time collecting unit 50, and if the collected signals are intermediate variables or data processed through DSP algorithm, real-time algorithm debugging work can be carried out at the PC end; if the collected sound signal is the original sound signal, the real-time processing unit at the PC end can be used for carrying out simulation processing, debugging and testing the data and variables in the processing process, or outputting and demonstrating the processed data;

the PC terminal signal real-time acquisition unit comprises a USB module, a cache module and a PC framing acquisition module which are sequentially connected, wherein the USB module is used for putting a transmitted parallel data stream into the cache module, forming a queue by data in the cache module, setting the length of the queue to determine the cache capacity of the cache module, and finally acquiring the data by the PC framing sampling module.