CN115862640A

CN115862640A - Acoustic wave user identification and heartbeat monitoring earphone system and method based on neural network

Info

Publication number: CN115862640A
Application number: CN202211534010.4A
Authority: CN
Inventors: 陈晓江; 孙雪; 卫旭东; 邓文文; 李晓慧; 王安文; 房鼎益
Original assignee: Northwest University
Current assignee: Northwest University
Priority date: 2022-12-01
Filing date: 2022-12-01
Publication date: 2023-03-28

Abstract

The invention discloses a sound wave user identification and heartbeat monitoring earphone system and a method based on a neural network, wherein the system comprises an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device, and a back-end service device at least comprises: the data acquisition module is used for transmitting and receiving sound wave signals; the data processing module is used for obtaining a preprocessed received signal; the human auditory canal feature extraction module is used for extracting MFCC features and carrying out auditory canal features based on the features of the transfer function; the user identity recognition module is used for obtaining a trained neural network model; the heartbeat information extraction module is used for obtaining the processed phase information; obtaining a phase signal without direct path interference from an interference elimination module; the heartbeat frequency calculation module is used for obtaining the heartbeat frequency. The invention can reduce the cost of identity recognition and heartbeat monitoring and has high robustness and robustness.

Description

Acoustic wave user identification and heartbeat monitoring earphone system and method based on neural network

Technical Field

The invention belongs to the technical field of wireless sensing of user identity information and monitoring of user heartbeat, and particularly relates to a sound wave user identification and heartbeat monitoring earphone system and method based on a neural network.

Background

Earphones are one of the most popular wearable devices in our daily lives. It has become an increasing trend to improve user experience and support new functions, such as touch control, to impart intelligence to the headset. In order to improve the recording quality, frebuds is internally provided with a bone vibration sensor. Apple Airpods pro uses an additional inward facing microphone to analyze noise from the outside so that the system can generate an anti-noise waveform to cancel the noise and thus achieve better audio quality. Besides basic music playing and voice communication functions, the earphone can also realize some new functions, such as user identity identification and heartbeat monitoring functions.

At present, the user identification method mainly comprises the following three ways of carrying out identification by using technologies such as user passwords, secret keys and the like, carrying out identification by using technical means such as human body physical features and the like of users, and carrying out identification by using certificates held by the users. The method for identifying the user identity by using the user password and the secret key has the risk of cracking the user password or the secret key and has potential safety hazards. The technical means such as utilizing human body physical characteristics and the like generally has higher safety, but additional equipment is needed to collect the human body physical characteristics such as fingerprints, faces, irises and the like, and the cost is higher. There is also a security implication for identification with a user holding a certificate, such as the certificate being lost or stolen, and requiring the certificate to be carried at any time.

The current heartbeat monitoring method utilizes special medical equipment to monitor, and also uses some intelligent wearable equipment areas to monitor the heartbeat of a user, such as a smart watch, a smart bracelet and the like. The main disadvantages for using professional medical equipment to monitor the heartbeat of a user are the high cost, the relatively complex operation and the poor portability. Although the method for monitoring the mind of the user by using the intelligent wearable device is simple in operation and uses traversal, the cost of installing some sensing elements of some intelligent wearable devices is higher than that of the method provided by the invention.

In summary, the existing user identification method mainly has the problems of potential safety hazard and high cost. The existing heartbeat monitoring method mainly has the problems of high cost, relatively complex operation and poor portability. These problems may result in limited or inconvenient use by the user.

Disclosure of Invention

The invention aims to provide a user identification and heartbeat monitoring earphone system and method based on a neural network and sound waves, and aims to solve the technical problems of complex operation and poor portability existing in the prior art.

In order to realize the technical task, the invention adopts the following technical scheme to realize:

a system for user identity identification and heartbeat monitoring based on a neural network and a sound signal at least comprises an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device and a back-end service device which are sequentially connected; wherein:

the microphone of the in-ear earphone with the microphone is arranged at the front end of the loudspeaker in the earphone earplug;

the acoustic analog-to-digital conversion equipment is used for converting an analog sound wave signal received by the in-ear earphone with the microphone into a digital sound wave signal and segmenting the digital sound wave signal in a period;

the back-end service device at least comprises:

the data acquisition module is used for controlling a loudspeaker and a microphone on the earphone to transmit and receive sound wave signals;

the data processing module is used for cleaning, denoising and dividing the data acquired by the data acquisition module to obtain a preprocessed received signal;

the human auditory canal feature extraction module is used for extracting feature information from the received signals preprocessed by the data processing module, extracting to obtain MFCC features and carrying out auditory canal features on the features based on the transfer function;

the user identity recognition module is used for performing feature splicing on the MFCC features output by the human auditory canal feature extraction module and the features based on the transfer function, inputting the characteristics into the neural network model and then training to obtain a trained neural network model, wherein the trained neural network model is used for realizing user identity recognition;

the heartbeat information extraction module is used for mixing the transmitting signal and the receiving signal to obtain a mixing signal, extracting phase information in the mixing signal, and then processing the phase information in the mixing signal by adopting a VMD algorithm to obtain processed phase information which is used as heartbeat information;

the self-interference elimination module is used for detecting and extracting the maximum peak position of the processed phase information output by the heartbeat information extraction module, aligning the maximum peak position of the processed phase information with the maximum peak position of the phase information of the direct path signal which is prestored and does not contain the reflection path on a time domain, and subtracting the maximum peak position of the phase information to obtain the phase signal which does not contain the direct path interference;

and the heartbeat frequency calculation module is used for detecting the number of peak values of the phase signals output by the self-interference elimination module and dividing the number of peak values by the signal duration to calculate the heartbeat frequency.

The invention also provides a user identity identification method, which is based on the system for user identity identification and heartbeat monitoring based on the neural network and the sound signal, and comprises the following steps:

step 1, controlling a loudspeaker and a microphone of an earphone to transmit and receive sound wave signals through a data acquisition module in rear-end service equipment, and converting the received analog sound wave signals into digital sound wave signals by acoustic analog-to-digital conversion equipment and periodically segmenting the digital sound wave signals;

step 2: the data processing module cleans, denoises and divides the data acquired in the step 1 to obtain a preprocessed received signal;

and 3, step 3: carrying out user auditory canal feature extraction on the preprocessed received signals through a human auditory canal feature extraction module to obtain auditory canal features of the MFCC and auditory canal features based on a transfer function;

and 4, step 4: the method comprises the steps of unfolding the MFCC auditory canal features obtained by a human auditory canal feature extraction module and the auditory canal features based on a transfer function into vectors, then performing feature splicing, inputting the vectors into a neural network model, and then performing training to obtain a trained neural network model, wherein the trained neural network model is used for realizing identity recognition of a user to be detected.

Further, in the step 1, the loudspeaker sends FMCW linear frequency modulation sound wave signals with PN lead codes and 1kHz-21kHz, and the sampling rate of the collected signals is 48kHz.

Further, the step 3 comprises the following steps:

step 31, performing feature extraction on the received signal obtained in the step 2 and subjected to preprocessing by adopting an MFCC algorithm to obtain an MFCC auditory canal feature;

step 32, calculating the frequency responses of the left and right ear canals of the user, wherein the ear canal frequency response H (f) is defined as follows:

where f is the frequency of the transmitted signal, P _xy (f) For pretreatment ofCross-power spectral density, P, between received and transmitted signals _xx (f) Is the self-power spectral density of the transmitted signal;

computing transfer function based ear canal characteristics H _d (f)：

Wherein H _r For the right ear canal frequency response, H _l Left ear canal frequency response.

Further, in the step 4, the neural network model adopts a three-layer MLP network structure.

The invention also provides a heartbeat monitoring method, which is based on the system for user identity identification and heartbeat monitoring based on the neural network and the sound signal, and comprises the following steps:

step 1: controlling a loudspeaker and a microphone of the earphone to transmit and receive sound wave signals through a data acquisition module in the back-end service equipment, and converting the received analog sound wave signals into digital sound wave signals by the acoustic analog-to-digital conversion equipment and segmenting the digital sound wave signals periodically;

step 2: the data processing module is used for cleaning, denoising and dividing the data acquired by the data acquisition module to obtain a preprocessed received signal;

and step 3: the heartbeat information extraction module mixes the transmitting signal with the preprocessed receiving signal to obtain a mixing signal, and extracts phase information in the mixing signal; then, processing the phase information in the mixing signal by adopting a VMD algorithm to obtain the processed phase information;

and 4, step 4: the self-interference elimination module aligns the processed phase information output in the step (3) with the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths on the time domain and then subtracts the phase information from the pre-stored direct path reflection signal to obtain a phase signal without direct path interference;

and 5: and (4) detecting the phase signal output by the step (4) by a heartbeat frequency calculation module to obtain the number of phase signal peaks, and dividing the number by the signal duration corresponding to the phase signal obtained by the step (4) to calculate the heartbeat frequency.

Further, in the step 1, the transmitted signal is an FMCW chirp signal with PN preamble of 16kHz to 21kHz, and the sampling rate of the collected signal is 48kHz.

Compared with the prior art, the invention has the following beneficial effects:

1. the system can be applied to any in-ear earphone integrated with a loudspeaker and a microphone, can realize user identity recognition and heartbeat monitoring by using a cheap earphone with a microphone without additional detection equipment, and reduces the cost of the identity recognition and the heartbeat monitoring.

2. The invention adopts the MFCC characteristics in the acoustic field and the transfer function characteristics based on the human auditory canal structure, realizes the user identity recognition through the neural network, and improves the robustness of the system.

3. The invention adopts the VMD algorithm to remove the motion state interference and provides a method based on the phase maximum peak value alignment to eliminate the hardware self-interference, so that the heartbeat monitoring function is more robust.

Drawings

Fig. 1 is a configuration diagram of a system of the present invention.

Fig. 2 is a diagram of the hardware modification of the earphone in the invention.

FIG. 3 shows the structure and relationship of the user ID modules in the system of the present invention.

Fig. 4 is a schematic diagram of a heartbeat information extraction module in the system of the present invention.

Fig. 5 is a schematic diagram of a self-interference cancellation module in the system of the present invention.

FIG. 6 is a schematic diagram of a module for calculating and extracting the heartbeat frequency of a user in the system of the present invention.

Fig. 7 is a flowchart of a user identification method according to the present invention.

FIG. 8 is a time-frequency diagram of signals of a user identification function in the system of the present invention.

Fig. 9 is a diagram of a neural network model structure of the system subscriber identity module of the present invention.

Fig. 10 is a flowchart of a user heartbeat monitoring method according to the present invention.

FIG. 11 is a time-frequency diagram of a signal of a user's heartbeat monitoring function in the system of the present invention.

Fig. 12 is a diagram showing an experimental result of user identification in the present invention, which shows the identification effect of the user identification method in the present invention.

Fig. 13 is a diagram of an experimental result of user identification in the present invention, which shows that the user identification method of the present invention has strong robustness against different user activities and headset wearing positions.

Fig. 14 is a diagram of an experimental result of user identification in the present invention, which shows that the user identification method of the present invention has strong anti-noise capability.

Fig. 15 is a diagram showing an experimental result of user identification in the present invention, which shows that the user identification method of the present invention has a stable identification capability.

Fig. 16 is an experimental result diagram of the user heartbeat monitoring method in the present invention, which shows that the user heartbeat monitoring method of the present invention has strong robustness for the daily activities of the user.

Fig. 17 is a diagram showing an experimental result of the user heartbeat monitoring method according to the present invention, which shows that the user heartbeat monitoring method according to the present invention is applicable to most users.

Fig. 18 is a diagram showing an experimental result of the method for monitoring heartbeat of a user according to the present invention, which shows that the method for monitoring heartbeat of a user according to the present invention is not affected by music of an earphone.

Fig. 19 is a result diagram of band-pass filtering of received signals with music in the user heartbeat monitoring method of the present invention, which proves that the user heartbeat monitoring method of the present invention can remove the influence of earphone music.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, the system for user identification and heartbeat monitoring based on a neural network and a sound signal at least comprises an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device and a back-end service device which are connected in sequence. Wherein:

see figure 2 for an in-ear headphone retrofit schematic. The original microphone module on the common in-ear earphone with the microphone is removed, the new microphone module is connected with the microphone circuit of the earphone and is packaged, and the new microphone module is arranged at the front end of the loudspeaker in the earphone earplug, so that the microphone can receive the reflected signal of the ear canal. Preferably, the new microphone module is a common microphone.

The acoustic analog-to-digital conversion device is used for converting an analog sound wave signal received by the in-ear earphone into a digital sound wave signal and segmenting the digital sound wave signal in a period.

The back-end service equipment is used for processing the data processed by the acoustic analog-to-digital conversion equipment by using the acoustic propagation characteristics of the human auditory meatus, and completing user identification and heartbeat monitoring. The back-end service equipment at least comprises a data acquisition module, a data processing module, a human ear canal feature extraction module, a user identity identification module, a heartbeat information extraction module, a self-interference elimination module and a heartbeat frequency calculation module.

The data acquisition module is used for controlling a loudspeaker and a microphone of the in-ear earphone to receive and transmit sound wave signals. The earphone is connected with the rear-end service equipment, and the data acquisition module controls a loudspeaker and a microphone on the earphone to transmit and receive sound wave signals by using audio IO of the rear-end service equipment. The data acquisition module collects the sound wave data information returned by the ear canal of the user and sends the sound wave data information to the data processing module.

The back-end service equipment adopts any equipment with audio IO, such as a mobile phone, a computer and the like. The data acquisition module, the data processing module, the human auditory canal feature extraction module, the user identity identification module, the heartbeat information extraction module, the self-interference elimination module and the heartbeat frequency calculation module are loaded in the back-end service equipment as the control module.

The data processing module is used for cleaning, denoising and segmenting the data acquired by the data acquisition module to obtain a preprocessed receiving signal so as to remove errors and useless data, provide clean and representative data, reduce the influence of noise and multipath effect existing in the data and facilitate subsequent feature extraction.

The human auditory canal feature extraction module is used for extracting feature information from the receiving signals preprocessed by the data processing module, extracting and obtaining MFCC features and carrying out auditory canal features based on the structure and outline features of the human auditory canal, and features based on a transfer function, so that the human auditory canal feature extraction module is used for a user identity identification module.

Referring to fig. 3, the user identity recognition module is configured to perform feature concatenation on the MFCC features output by the human ear canal feature extraction module and the transfer function-based features, and then input the MFCC features and the transfer function-based features into the neural network model for training to obtain a trained neural network model, where the trained neural network model is used to implement user identity recognition.

Referring to fig. 4, the heartbeat information extraction module is configured to mix the transmit signal and the receive signal to obtain a mixed signal, extract phase information in the mixed signal m (t), and process the phase information in the mixed signal m (t) by using a VMD algorithm to obtain processed phase information, which is used as the heartbeat information. The algorithm can eliminate motion interference.

Referring to fig. 5, the self-interference elimination module is configured to detect and extract a maximum peak position of the processed phase information output by the heartbeat information extraction module, align the maximum peak position of the processed phase information with a maximum peak position of phase information of a pre-stored direct path signal without a reflection path in a time domain, and subtract the aligned maximum peak position and the maximum peak position to obtain a phase signal without direct path interference. Thereby eliminating direct path self-interference and obtaining accurate phase signals.

Referring to fig. 6, the heartbeat frequency calculation module is configured to detect the number of peak values of the phase signal output by the self-interference cancellation module, and divide the number by the signal duration to calculate the heartbeat frequency.

Referring to fig. 7, based on the system of the present invention, the method for identifying the user identity provided by the present invention includes the following steps:

step 1, controlling a loudspeaker and a microphone of an earphone to transmit and receive sound wave signals through a data acquisition module in rear-end service equipment, and converting the received analog sound wave signals into digital sound wave signals by acoustic analog-to-digital conversion equipment and periodically segmenting the digital sound wave signals.

Specifically, when data is collected, the ear plugs are plugged into the ear canals of the user, and after receiving a control instruction sent by the data collection module, the speaker uninterruptedly sends FMCW chirped sound wave signals with PN lead codes and 1kHz to 21kHz (see time-frequency spectrum in fig. 8). The preamble is only present once at a time of detection for the synchronization signal. Each FMCW chirped acoustic signal is periodic in 0.05 seconds. The sound waves are reflected by the ear canal structure and contour and then transmitted back to the microphone of the earphone. When the signal is collected, the sampling rate is 48kHz, so that one period is 2400 data points.

The acoustic analog-to-digital conversion equipment converts an analog sound wave signal received by the microphone into a digital sound wave signal, and the data acquisition module stores the digital sound wave signal and segments the digital sound wave signal periodically. And further storing the segmented information in a file in a mat format to finish data acquisition.

Step 2: the data processing module cleans, denoises and segments the data acquired by the data acquisition module to obtain a preprocessed received signal. The method comprises the following specific steps:

firstly, the original data acquired by the data acquisition module, such as a microphone of an earphone, is slowly started, does not start to work, and cannot receive the data, the whole data lags by about half a cycle, and partial data are incomplete, which belongs to error data, has a great influence on identification, and the data should be discarded or supplemented.

Secondly, the observation of each group of data shows that the data is very noisy in the early stage of collection and has very obvious changes, and the changes are caused by a system and are not suitable for identification. After the data are stable, the difference between adjacent periods is very small, and the contained characteristic information is insufficient. Based on the two reasons, the method adopts the steps of eliminating direct current components, performing band-pass filtering processing, and selecting a middle data section with small data noise and stable change as a preprocessed received signal.

And step 3: and carrying out user auditory canal feature extraction on the preprocessed received signals through a human auditory canal feature extraction module, and obtaining auditory canal features of the MFCC and auditory canal features based on a transfer function by utilizing the uniqueness of the auditory canals and the outline structures of the users.

Because the difference between the preprocessed data is not obvious and can not be directly used for recognition, further processing and feature extraction are needed, the difference between the data is amplified, a recognition module can capture features conveniently, and correct classification is realized. Therefore, the invention adopts the MFCC characteristics and the transfer function characteristics based on the human auditory canal structure to further amplify the difference between the data. The ear canal feature extraction method based on the MFCC features and the transfer function features comprises the following steps:

1) And (3) performing feature extraction on the received signal obtained in the step (2) and after preprocessing by adopting an MFCC algorithm to obtain the MFCC auditory canal feature. The MFCC algorithm is an existing algorithm and is not described in detail here, and is mainly to convert a signal frequency into a Mel frequency and then calculate a cepstrum coefficient of the signal under the Mel frequency.

2) And (5) extracting transfer function features. Since the MFCC features alone are inaccurate in user identification and cannot be distinguished by using a single MFCC feature, a transfer function feature is added. The frequency responses of the user's left and right ear canals are first calculated. The ear canal frequency response H (f) is defined as follows:

where f is the frequency of the transmitted signal, P _xy (f) For cross-power spectral density, P, between pre-processed received and transmitted signals _xx (f) Is the self-power spectral density of the transmitted signal.

The frequency response of the right ear is divided by the frequency response of the left ear to obtain a transfer function based ear canal characteristic of the user. Transfer function based ear canal characteristics H _d (f) The definition is as follows:

And 4, step 4: and the user identity recognition module trains the neural network model by adopting the characteristic information output by the human auditory canal characteristic extraction module to obtain the trained neural network model.

Specifically, the MFCC auditory canal features obtained by the human auditory canal feature extraction module and the auditory canal features based on the transfer function are expanded into vectors and then are subjected to feature splicing, and then the vectors are input into a neural network model and trained to obtain a trained neural network model, wherein the trained neural network model is used for realizing identity recognition of a user to be detected.

The neural network model structure of the user identity recognition module is shown in fig. 9, a simple three-layer MLP network structure is adopted, the hidden layer is only one layer, and almost no performance burden is imposed on the back-end processing equipment.

Referring to fig. 10, based on the system of the present invention, the heartbeat monitoring method of the present invention includes the following steps:

step 1: except for sending signals, the step 1 of the user identity identification method of the invention is consistent with the steps, and is not described again.

In the method, the transmission signal is an FMCW chirp sound wave signal with PN preamble and 16kHz-21kHz, which is shown in a time-frequency diagram of FIG. 11. The preamble is only present once at a time of detection for the synchronization signal. Each FMCW chirped acoustic signal is periodic in 0.2 seconds with 0.15 seconds being the interval. When the signal is acquired, the sampling rate is 48kHz, so that one period is 9600 data points.

Step 2: and the data processing module is used for cleaning, denoising and segmenting the data acquired by the data acquisition module to obtain a preprocessed received signal.

And step 3: the heartbeat information extraction module mixes the transmitting signal with the preprocessed receiving signal to obtain a mixing signal m (t), and extracts phase information in the mixing signal m (t); and then, processing the phase information in the mixing signal m (t) by adopting a VMD algorithm to obtain the processed phase information which is used as heartbeat information.

When the user moves, the extracted phase information includes motion information interference, so that the motion interference needs to be eliminated. When the mixed signal m (t) moves, the phase change of the generated signal phase change can generate a frequency multiplication signal, so that the phase change in the mixed signal m (t) is decomposed according to the frequency by adopting a VMD algorithm, the motion interference with the frequency multiplication signal is removed, and the phase signal only containing heartbeat information is extracted. Since the VMD decomposition is an existing algorithm, the principle thereof is not described in detail.

And 4, step 4: and 3, aligning the processed phase information output in the step 3 with the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths on a time domain by the self-interference elimination module, and subtracting the aligned phase information and the pre-stored maximum peak point of the phase of the direct path reflection signal without other reflection paths to obtain the phase signal without direct path interference. Therefore, the self-interference influence can be eliminated to obtain an accurate phase signal.

Referring to fig. 2, the microphone is placed in front of the speaker, so that the direct path signal from the microphone to the speaker is included in the received signal, and the phase peak generated by the heartbeat is drowned out. This step is therefore taken to eliminate the direct path effect.

And 5: and (4) detecting the phase signal output by the step (4) by a heartbeat frequency calculation module to obtain the number of phase signal peaks, and dividing the number by the signal duration corresponding to the phase signal obtained by the step (4) to calculate the heartbeat frequency. The calculation formula is as follows:

wherein peak n is the number of peak values of the phase signal, t is the duration corresponding to the phase signal obtained in step 4, and Freq is the heartbeat frequency.

Experimental part:

in order to embody the convenience and universality of the invention, the embodiments use the PC as the acoustic wave analog-to-digital conversion device and the back-end service device, and it should be understood that the specific embodiments described herein are only used for explaining the invention and are not used for limiting the invention. This experimental application includes user, PC, in-ear earphone. The microphone and the loudspeaker in the earphone form a group of acoustic signal transceiving nodes of the acoustic sensing device. The PC is used as sound wave analog-to-digital conversion equipment and back-end service equipment, configures signals, receives and converts signals collected by the earphone for analysis.

The specific operation process comprises the following steps: the server controls the earphone loudspeaker to send sound signals, the earphone microphone records echo signals, the echo signals are sent to the server through the analog-to-digital conversion equipment to be processed, and preprocessing operations such as filtering and denoising are carried out. For the user identification function, the MFCC and the transfer function characteristics are calculated and spliced to form a characteristic vector, the user identity is identified through a trained model, and the result is displayed on a PC, so that the identification of the user identity is realized. For the user heartbeat monitoring function, firstly extracting signal phase information to eliminate motion interference, then eliminating self-interference influence, finally calculating the user heartbeat frequency according to the number of signal phase peak points detected, and displaying the result on a PC.

1. Performance experiment of user identity recognition method

The experiment identifies the user identity information by a user identity identification method based on a neural network and a sound signal. To this end, 120 participants were recruited, including 53 boys and 67 girls, with the age range of participants between 10 and 90 years. The number of participants is increased from 10 to 120 by making it reasonable to wear headphones in a way that is comfortable to the participants during the identification process. Referring to fig. 12, it is observed that although the overall recognition accuracy is decreasing with the increasing number of users, the overall recognition accuracy can still reach more than 96% when the number of users is 120, and the corresponding F1 score can also reach more than 96%. Therefore, the user identity identification method based on the neural network and the sound signal has high user identification precision.

2. Robustness experiment of user identity identification method for user activities and earphone wearing positions

The user identity identification method provided by the invention is expected to have stronger robustness on user activities and the positions where the earphones are worn, and not only can be used in daily activities of users, but also can be used for considering the positions where the earphones are worn by different users. Therefore, in order to verify that the user can normally use the device in daily life, three activities which have influence on identification are selected in daily life, namely, the identification of the user is carried out under the conditions of head movement, speaking and chewing. In order to verify the wearing positions of the earphones of different users, the user identity is identified under the conditions that the wearing angles of the four earphones and the auditory meatus are respectively 0 degrees, 90 degrees, 180 degrees and 270 degrees, and the user identity is identified under the conditions that the wearing depths of the three earphones are respectively shallow, moderate and deep. Referring to fig. 13, it was observed that the balance accuracy and F1 score of head movements and speech were above 98% for three different activities, and above 90% for chewing activities with greater impact on performance. This is because chewing causes a fine deformation of the ear canal of the user, resulting in a decrease in accuracy. Secondly, it is observed that the wearing angles of different angles have slight influence on the recognition performance, but the F1 scores and the balance accuracy of the four wearing angles are all over 90%. Finally, it was observed that the F1 score and the balance accuracy were above 90% for different degrees of wear depth. The user identity recognition method provided by the invention is proved to have stronger robustness on the daily activities of the user and the wearing position of the earphone.

3. Anti-noise capability experiment of user identity identification method

The user identity identification method hopefully provided by the invention can be used in most scenes in daily life and has higher anti-noise performance. Therefore, the user identity is identified under the conditions that four environments with different noises are selected to be respectively 55dB, 63dB, 75dB and 84 dB. Referring to fig. 14, it is observed that when the ambient noise is below 80dB, the error acceptance rate and the error recognition rate are below 3%. When the noise increases to 84dB, the false acceptance rate and false rejection rate increase sharply, but noise exceeding 80dB is rare in daily life. Therefore, the user identity identification method provided by the invention is proved to have stronger anti-noise performance.

4. Stable identification experiment of user identity identification method

The user identity identification method hopefully provided by the invention can realize the stable identification of the user identity for a long time. Therefore, four time dimensions are selected for verification, and the user identity is identified under the conditions of one hour, one day, one week and one month respectively. The specific experimental process was three participants recorded, and data was collected once a day for one month. The user identification performance over time is recorded. Referring to fig. 15, it is observed that the recognition result is slightly degraded in all aspects with time, but the balance accuracy, F1 score, and recognition accuracy of the recognition result are all greater than 95%. Therefore, the user identity identification method provided by the invention has time stability.

5. User heartbeat monitoring method for user activity robustness experiment

The user heartbeat monitoring method hopefully provided by the invention can realize user heartbeat monitoring under the condition of not influencing the daily activities of the user. Therefore, six moving scenes are selected to be sleeping, standing, speaking, head moving, walking and running, and the accuracy of the heartbeat monitoring method provided by the invention is evaluated by 120 participants in the six scenes. The specific experimental process is that a standard heartbeat measuring instrument is selected as a reference, heartbeat data of each user under each activity scene respectively under the standard heartbeat measuring instrument and the user heartbeat measuring method provided by the invention is recorded, and the accuracy of the heartbeat monitoring method provided by the invention is measured by using the absolute error of the standard heartbeat measuring instrument and the user heartbeat measuring method. Referring to FIG. 16, it was observed that the average absolute errors of sleeping, sitting, speaking, head moving, walking and running were 1.04bp, 1.71bpm, 3.42bpm, 4.63bpm, 6.31bpm and 7.28bpm, respectively. It can be seen that the average absolute error of the method is low in an environment where the user is relatively static. Although the average absolute error of the method is high in the user motion state, the error rate is still low because the heartbeat frequency is about 120bpm in the motion state, and the difference value between the actual heartbeat and the measured heartbeat is within 10% of the heartbeat frequency, which is effective data. Therefore, the user heartbeat monitoring method provided by the invention has robustness on user activities.

6. Applicability experiment of user heartbeat monitoring method to different users

The user heartbeat monitoring method hopefully provided by the invention is suitable for most users, namely, the accurate user heartbeat monitoring can be realized for most users. Therefore, under the selected scene that the user sits, the number of the users is gradually increased from 20 to 120, the heartbeat data of each user under the standard heartbeat measuring instrument and the user heartbeat measuring method provided by the invention are recorded, and the average absolute error is calculated. Referring to fig. 17, it is observed that the mean absolute error of the method stabilizes around 1.7 as the number of people increases. Therefore, the user heartbeat monitoring method provided by the invention is suitable for most users.

7. Robustness experiment of user monitoring method on earphone music playing

The user heartbeat monitoring method hopefully provided by the invention can realize the purpose of monitoring the heartbeat of the user on the premise of not influencing the normal use of the earphone by the user. Therefore, the heartbeat data of a user within 20 minutes is recorded by using a standard heartbeat measuring instrument and the user heartbeat measuring method provided by the invention. The data measured by the standard heartbeat measuring instrument is used as a benchmark. In the process of using the method for measuring the heartbeat of the user, the user uses the same earphone to play music. Referring to fig. 18, it is observed that the method and reference of the present invention are very close even in the case of playing music. This is because the frequency of the musical sound is typically below 4kHz, much lower than the frequency of the perceptual signal (16-21 kHz). Referring to fig. 19, therefore, the filter can easily remove the music signal used in the received signal. The result proves that the user heartbeat monitoring method hopefully provided by the invention can realize the monitoring of the user heartbeat on the premise of not influencing the normal use of the earphone by the user.

The present invention is not limited to the above preferred embodiments, and any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sound wave user identification and heartbeat monitoring earphone system based on a neural network is characterized by at least comprising an in-ear earphone with a microphone, an acoustic analog-to-digital conversion device and a back-end service device which are sequentially connected; wherein:

the back-end service equipment at least comprises:

the self-interference elimination module is used for detecting and extracting the maximum peak position of the processed phase information output by the heartbeat information extraction module, aligning the maximum peak position of the processed phase information with the maximum peak position of the phase information of the direct path signal which is prestored and does not contain the reflection path on a time domain, and then subtracting the maximum peak position of the processed phase information from the maximum peak position of the phase information to obtain a phase signal which does not contain the direct path interference;

2. A method for user identification based on the acoustic wave user identification and heartbeat monitoring earphone system based on the neural network of claim 1, comprising the steps of:

and step 3: carrying out user auditory canal feature extraction on the preprocessed received signals through a human auditory canal feature extraction module to obtain auditory canal features of the MFCC and auditory canal features based on a transfer function;

3. The method for identifying a subscriber as claimed in claim 2, wherein in step 1, the speaker transmits FMCW chirp signals of 1kHz to 21kHz with PN preamble, and the sampling rate of the collected signals is 48kHz.

4. The method for identifying the user according to claim 2, wherein the step 3 comprises the steps of:

where f is the frequency of the transmitted signal, P _xy (f) For cross-power spectral density, P, between pre-processed received and transmitted signals _xx (f) Is the self-power spectral density of the transmitted signal;

computing transfer function based ear canal characteristics H _d (f)：

5. The method according to claim 2, wherein in step 4, the neural network model adopts a three-layer MLP network structure.

6. A heartbeat monitoring method based on the acoustic wave user recognition and heartbeat monitoring earphone system based on the neural network as claimed in claim 1, comprising the steps of:

and 2, step: the data processing module is used for cleaning, denoising and dividing the data acquired by the data acquisition module to obtain a preprocessed received signal;

and 3, step 3: the heartbeat information extraction module mixes the transmitting signal with the preprocessed receiving signal to obtain a mixing signal, and extracts phase information in the mixing signal; then, processing the phase information in the mixing signal by adopting a VMD algorithm to obtain processed phase information;

and 4, step 4: the self-interference elimination module aligns the processed phase information output in the step (3) with the maximum peak point of the phase of the pre-stored direct path reflection signal without other reflection paths in the time domain and then subtracts the phase information from the pre-stored direct path reflection signal to obtain a phase signal without direct path interference;

7. The method for monitoring heartbeat according to claim 6 wherein in step 1, the transmitted signal is an FMCW chirp signal with PN preamble at 16kHz-21kHz and the sampling rate of the collected signal is 48kHz.