CN110660201A - Arrival reminding method, device, terminal and storage medium - Google Patents

Abstract

The embodiment of the application discloses a method, a device, a terminal and a storage medium for reminding a user of arriving at a station, and belongs to the field of artificial intelligence. The method comprises the following steps: collecting ambient sounds by a microphone while in a vehicle; identifying the environmental sound; when the environment sound is identified to contain the target alarm ring, adding one to the number of the running stations; and when the number of the running stations reaches the target number of the stations, reminding the station arrival. By adopting the method provided by the embodiment of the application, the number of the running stations is updated when the current environment sound is identified to contain the target alarm ring by acquiring the environment sound in real time and identifying whether the current environment sound contains the target alarm ring, and the station-entering reminding is carried out when the number of the running stations reaches the target number of the stations; because the alarm ring is used for giving out a warning to the passenger, the sound characteristic is obvious and the alarm ring is easy to identify, and therefore the accuracy and the effectiveness of the arrival reminding can be improved by carrying out the arrival reminding based on the alarm ring in the environment sound.

Description

Arrival reminder method, device, terminal and storage medium

technical field

The embodiments of the present application relate to the field of artificial intelligence, and in particular, to an arrival reminder method, device, terminal, and storage medium.

Background technique

When people travel by public transportation such as subways, they need to always pay attention to whether the current stop is their target station, and the arrival reminder function is a function to remind passengers to get off the bus in time when they reach the target station.

In the related art, the terminal usually uses the speech recognition technology to obtain the current station information according to the arrival information broadcast by the subway, and judges whether the current station is the destination station of the passengers.

However, when the above method is used to obtain station information, the voice of passengers and the noise of subway operation will have a greater impact on the results of speech recognition, which may easily lead to delayed or inaccurate reminders.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an arrival reminder method, device, terminal, and storage medium. The technical solution is as follows:

On the one hand, an embodiment of the present application provides an arrival reminder method, the method includes:

When in a vehicle, collect ambient sound through a microphone;

identifying the ambient sound;

When it is recognized that the ambient sound contains a target alarm bell, the operation of adding one to the number of stations that have been traveled is performed, and the target alarm bell is an open-door alarm bell or a door-close alarm bell;

When the number of traveled stations reaches the target number of stations, an arrival reminder is performed, and the target station number is the number of stations between the starting station and the target station, and the target station is a transfer station or a destination station.

On the other hand, an embodiment of the present application provides an arrival reminder device, and the device includes:

The acquisition module is used to collect ambient sound through a microphone when it is in a vehicle;

an identification module for identifying the ambient sound;

a counting module, used for adding one to the number of stations that have been traveled when it is identified that the ambient sound contains a target alarm tone, and the target alarm tone is an open-door alarm tone or a door-closed alarm tone;

Reminder module, used to remind the arrival of the station when the number of the traveled stations reaches the target station number, the target station number is the station number between the starting station and the target station, and the target station is a transfer station or a destination local site.

On the other hand, an embodiment of the present application provides a terminal, where the terminal includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the above aspects. The arrival reminder method described above.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, where the storage medium stores at least one instruction, and the at least one instruction is configured to be executed by a processor to implement the arrival reminder method described in the above aspect .

The beneficial effects of the technical solutions provided by the embodiments of the present application include at least:

In the embodiment of the present application, by collecting ambient sounds in real time and identifying whether the current ambient sound contains the target alarm tone, when it is identified that the target alarm tone is included, the number of traveled stations is updated, and the number of traveled stations reaches the target. When the number of stops is reached, it will remind you to enter the station; because the alarm bell is used to warn passengers, the sound characteristics are more obvious and easy to be identified, so the arrival reminder based on the alarm bell in the ambient sound can improve the accuracy and efficiency of the arrival reminder. effectiveness.

Description of drawings

1 is a flow chart of a method for reminding arrival at a station according to an exemplary embodiment;

2 is a flowchart of a method for reminding arrival at a station according to another exemplary embodiment;

3 is a flowchart of a method for reminding arrival at a station according to another exemplary embodiment;

FIG. 4 is a flowchart of audio data preprocessing according to an exemplary embodiment;

FIG. 5 is a flowchart of a voice recognition process according to another exemplary embodiment;

6 is a flowchart of a method for reminding arrival at a station according to another exemplary embodiment;

FIG. 7 is a flowchart of voice recognition performed by a first-level voice recognition model according to an exemplary embodiment;

FIG. 8 is a spectrogram of an ambient sound according to an exemplary embodiment;

9 is a frame diagram of a secondary voice recognition model shown according to an exemplary embodiment;

10 is a structural block diagram of an arrival reminder device according to an exemplary embodiment;

FIG. 11 is a structural block diagram of a terminal according to an exemplary embodiment.

The above-mentioned drawings have shown clear embodiments of the present disclosure, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the disclosed concepts to those skilled in the art by referring to specific embodiments.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

The "module" mentioned in this article generally refers to a program or instruction stored in a memory that can realize certain functions; the "unit" mentioned in this article generally refers to a functional structure divided by logic, the "unit" It can be realized by pure hardware, or a combination of software and hardware.

As used herein, "plurality" refers to two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship.

The arrival reminder method provided by each embodiment of the present application is used for a terminal with audio collection and processing functions, and the terminal may be a smart phone, a tablet computer, an e-book reader, a personal portable computer, or the like. In a possible implementation manner, the arrival reminder method provided by the embodiment of the present application may be implemented as an application program or a part of the application program, and installed in the terminal. When the user takes a vehicle, the application can be manually opened (or the application can be automatically opened), so as to remind the user of the arrival of the station through the application.

In the related art, speech recognition technology is usually used to determine the station name of the current vehicle location according to the station announcement when the vehicle arrives at the station, and to remind the user of the arrival when the vehicle arrives at the target station. However, the noise generated by the vehicle during driving and the environmental sounds such as the voice of the passengers will affect the speech recognition, which may easily lead to errors in the speech recognition results, and the speech recognition model is difficult to run on the terminal, and usually needs to rely on the cloud to run.

In addition, in the related art, the accelerometer is used to detect whether the vehicle is accelerating or decelerating, thereby judging whether the vehicle has entered the station. However, the acceleration direction recorded by the accelerometer sensor in the terminal is related to the direction in which the user holds the terminal. Walking in the interior will also affect the recording results of the sensor, and the vehicle sometimes stops temporarily between two stations, and it is difficult to accurately determine the location of the vehicle by using the accelerometer.

In order to solve the above problem, an embodiment of the present application provides a method for reminding of arrival at the station, and the flow of the method for reminding of arrival at the station is shown in FIG. 1 . Before the terminal uses the arrival reminder function for the first time, step 101 is performed to store the route map of the vehicle; when the terminal starts the arrival reminder function, step 102 is first performed to determine the travel route; after entering the vehicle, step 103 is performed, Obtain the ambient sound in real time through the microphone; go to step 104, the terminal identifies whether the ambient sound contains the target alarm tone, and when it recognizes that the ambient sound does not contain the target alarm tone, it continues to identify the next segment of ambient sound, and when the terminal recognizes the ambient sound When the target alarm tone is included, go to step 105, add one to the number of stations that have been traveled; go to step 106, judge whether it is the destination station according to the number of stations that have been traveled, and if the station is the destination station, go to step 107, send Arrival reminder, if the site is not the destination site, go to step 108 to determine whether it is a transit site, and when it is determined to be a transit site, perform step 107 again to send an arrival reminder, otherwise continue to identify the next segment of ambient sound.

Compared with the arrival reminder method provided in the related art, the embodiment of the present application judges the site where the vehicle has traveled by identifying whether the current ambient sound contains the target alarm tone. Since the target alarm tone has obvious characteristics compared with other ambient sounds, There are fewer affected factors, so the recognition result has a high accuracy rate; and there is no need to use a complex speech recognition model for speech recognition, which helps to reduce the power consumption of the terminal.

Please refer to FIG. 2 , which shows a flowchart of a method for reminding arrival at a station according to an embodiment of the present application. This embodiment is described by taking the arrival reminder method for a terminal with audio collection and processing functions as an example, and the method includes:

Step 201, when in a vehicle, collect ambient sound through a microphone.

When in the vehicle, the terminal turns on the arrival reminder function, and collects the ambient sound in real time through the microphone.

In a possible implementation, when the arrival reminder method is applied to a map navigation application, the terminal acquires user location information in real time, and when it is determined that the user enters a vehicle according to the user location information, the terminal enables the arrival reminder function.

Optionally, when the user uses the payment application to swipe the card to take the vehicle, the terminal confirms entering the vehicle and enables the arrival reminder function.

Optionally, in order to reduce the power consumption of the terminal, the terminal may use a low-power microphone for real-time collection.

Step 202, identifying the ambient sound.

Optionally, the terminal converts the ambient sound collected in real time through the microphone into audio data, performs data processing on the audio data, and identifies whether the processed audio data contains audio data of the target alarm tone.

In a possible implementation manner, when the terminal acquires the vehicle route map of the city, it acquires the alarm bells of different vehicles, and saves its audio data locally. When the terminal cannot obtain the alarm tone of the current city or vehicle, the user needs to turn on the microphone to collect and save the alarm tone when riding the vehicle for the first time, so that the terminal can learn the alarm tone.

Step 203 , when it is recognized that the ambient sound contains the target alarm tone, the operation of adding one to the number of stations traveled is performed, and the target alarm tone is the door opening alarm tone or the door closing alarm tone.

When the terminal recognizes that the target alarm tone is included in the current ambient sound, it indicates that the current vehicle has arrived at a certain station, and then adds one to the number of stations it has traveled. Since the vehicle usually emits alarm bells when opening and closing the door, in order to avoid counting confusion, the terminal can be set in advance to recognize only the door-opening alarm bell or only the door-closing alarm bell. Usually, the time interval between the door opening alarm bell and the door closing alarm bell is small, so in the case that the door opening alarm bell and the door closing alarm bell are the same, when two alarm bells are recognized within a fixed time area, it is regarded as one door opening or one door closing.

Step 204 , when the number of traveled stations reaches the target number of stations, an arrival reminder is performed, the target station number is the number of stations between the starting station and the target station, and the target station is a transfer station or a destination station.

After the terminal performs an operation of adding one, if the current number of stations traveled reaches the target number, it means that the current station is the target station, and the user is reminded to arrive at the station. The number of target stations is the number of stations between the starting station and the target station, that is, the number of stations that the vehicle needs to travel from the starting station to the target station, and the target station includes the transfer station and the destination station.

Optionally, in order to prevent the time between the terminal issuing the arrival reminder and the vehicle closing the door and heading to the next station from being too short, and the user missing the time to get off the bus, you can set the message to be sent when reaching the previous station of the target station. Prompt to make the user prepare to get off the car in advance.

Optionally, the method of arrival reminder includes but is not limited to: voice reminder, vibration reminder, and interface reminder.

Regarding the method of obtaining the number of target stations, in a possible implementation, the terminal loads and stores the route map of the vehicle in the current city in advance, and the route map includes the station information, transfer information, first and last trains of each line. Time and site nearby maps, etc. Before the terminal turns on the microphone to collect ambient sound, it first obtains the user's ride information, which includes the starting station, the target station, the map near the station, the first and last shift times, etc., so as to determine the number of target stations according to the ride information.

Optionally, the way for the terminal to obtain the travel information may be manually input by the user, such as the names of the starting station and the target station, and the terminal selects the appropriate travel route according to the travel information input by the user and the route map of the vehicle. When arriving at the target site, the terminal sends a message of arrival reminder and a map near the target site to the user.

Optionally, the ride information manually input by the user may only be the number of stops between the starting stop and the target stop. Because the method of the embodiment of the present application is that the terminal judges the current station according to the alarm sound when the vehicle opens or closes the door, and when the target alarm sound is recognized, the operation of adding one to the number of traveled stations is performed until the number of traveled stations is equal to the starting point The number of stations that the station needs to travel to reach the target station, so when the user has a definite bus route, he can only input the station number of the bus route, and the terminal can prompt the user to enter the starting station and the The number of stops between the staging site and the number of sites between the staging site and the destination site.

Optionally, the terminal may predict the user's bus route according to the user's historical travel records, take the bus route whose number of trips reaches the threshold of the number of trips as the preferred route, and prompt the user to make a selection.

To sum up, in the embodiment of the present application, by collecting ambient sounds in real time and identifying whether the current ambient sound contains target alarm bells, when it is identified that the target alarm bells are included, the number of stations traveled is updated, and the When the number of driving stops reaches the target number of stations, a stop reminder will be given; since the alarm bells are used to warn passengers, the sound features are more obvious and easy to identify, so the arrival reminder based on the alarm bells in the ambient sound can improve the arrival of the station. The accuracy and effectiveness of reminders.

In a possible implementation, when identifying whether the ambient sound contains the target alarm tone, in order to improve the recognition accuracy, it is necessary to preprocess the audio data corresponding to the ambient sound, and then input the processed audio data into the sound recognition model. , so as to determine whether the current ambient sound contains the target alarm tone according to the target alarm tone recognition result output by the sound recognition model. Illustrative embodiments are used for description below.

Please refer to FIG. 3 , which shows a flowchart of a method for reminding arrival at a station according to another embodiment of the present application. This embodiment is described by taking the arrival reminder method for a terminal with audio collection and processing functions as an example, and the method includes:

Step 301, when in a vehicle, collect ambient sound through a microphone.

For the implementation of step 301, reference may be made to the foregoing step 201, and details are not described herein again in this embodiment.

Step 302: Perform frame-by-frame processing on the audio data corresponding to the ambient sound to obtain audio data corresponding to different audio frames.

Since the sound recognition model cannot directly recognize the audio data, it is necessary to pre-process the audio data to obtain digital features that can be recognized by the sound recognition model. Since the microphone of the terminal collects ambient sound in real time, the audio data is not stable as a whole, but its part can be regarded as stable data, and the sound recognition model can only identify the stable data, so the terminal first divides the corresponding audio data into frames process to obtain audio data corresponding to different audio frames.

In a possible implementation, the audio data preprocessing process is shown in FIG. 4 , the audio data is first subjected to pre-emphasis processing by the pre-emphasis module 401, and the pre-emphasis process adopts a high-pass filter, which only allows the frequency higher than a certain frequency. The signal component passes through, and the signal component lower than this frequency is suppressed, thereby removing unnecessary low-frequency interference such as human conversation, footsteps and mechanical noise in the audio data, and flattening the frequency spectrum of the audio signal. The mathematical expression for the high-pass filter is:

H(z)=1-az ^-1

Among them, a is the correction coefficient, generally in the range of 0.95 to 0.97, and z is the audio signal.

The audio data after the noise has been removed is subjected to frame-by-frame processing by the frame-by-frame windowing module 402 to obtain audio data corresponding to different audio frames.

Illustratively, in this embodiment, audio data including 1024 data points is divided into one frame, and when the sampling frequency of the audio data is selected as 16000 Hz, the duration of one frame of audio data is 64 ms. In order to avoid excessive changes between the two frames of data, and also to avoid data loss at both ends of the audio frame after windowing, the audio data is not directly divided into frames in a back-to-back manner, but after each frame of data is taken , then slide for 32ms and then take the next frame of data, that is, two adjacent frames of data overlap for 32ms.

Since the audio data after frame-by-frame processing needs to undergo discrete Fourier transform in subsequent feature extraction, a frame of audio data has no obvious periodicity, that is, the left end of the frame and the right end of the frame are discontinuous, and after Fourier transform, it is the same as the original data. Errors will occur, and the more framing, the greater the error. Therefore, in order to make the audio data after framing continuous, and each frame of audio data shows the characteristics of a periodic function, it is necessary to perform framing and windowing processing by the framing windowing module 402. .

In a possible implementation, a Hamming window is used to perform windowing processing on the audio frame. Multiply each frame of data by the Hamming window function, and the obtained audio data has obvious periodicity. The functional form of the Hamming window is:

Wherein n is an integer, the value range of n is 0 to M, and M is the number of Fourier transform points. Illustratively, in this embodiment, 1024 data points are taken as Fourier transform points.

Step 303: Perform feature extraction on the audio data corresponding to each audio frame to obtain a corresponding audio feature matrix.

After the audio data is framed and windowed, feature extraction needs to be performed to obtain a feature matrix that can be recognized by the voice recognition model.

In a possible implementation, the Mel-Frequency Cepstral Coefficients (MFCC) of the audio frame is extracted, and the process is shown in FIG. 4 , since it is difficult to obtain from the audio signal transformation in the time domain Its signal characteristics usually need to be processed by converting the time-domain signal into the energy distribution in the frequency domain. Therefore, the terminal first inputs the audio frame data into the Fourier transform module 403 for Fourier transform, and then converts the Fourier-transformed audio The frame data is input to the energy spectrum calculation module 404 to calculate the energy spectrum of the audio frame data. In order to convert its energy spectrum into a Mel spectrum conforming to human hearing, the energy spectrum needs to be input into the Mel filtering processing module for filtering processing; the mathematical expression of filtering processing is:

Among them, f is the frequency point after Fourier transform.

After obtaining the mel spectrum of the audio frame, the terminal takes the logarithm of it through the discrete cosine transform (Discrete Cosine Transform, DCT) module 406, and the obtained DCT coefficient is the MFCC feature.

Illustratively, the embodiment of the present application selects 40-dimensional MFCC features. When the terminal actually extracts features, the input window length of audio data is selected as 1056ms, and the time length of one frame of signal is 64ms. The overlapping part of 32ms, so each 1056ms input window data corresponds to a matrix of 32*40 features.

Step 304: Input the audio feature matrix into the sound recognition model to obtain the target alarm tone recognition result output by the voice recognition model, and the target alarm tone recognition result is used to indicate whether the audio frame contains the target alarm tone.

Optionally, the terminal inputs the audio feature matrix obtained by the feature extraction of the audio frame into the sound recognition model, and the model recognizes whether the current audio frame contains the target alarm tone, and outputs the recognition result.

In a possible implementation, if the terminal cannot autonomously acquire the alarm tone of the vehicle in the current city, the user needs to collect the target alarm tone in advance. After the target alarm tone is collected, the audio data containing the target alarm tone is also Carry out the frame division processing and feature extraction process of the above steps 302 to 303, and save the audio feature matrix of different target alarm tones to the local.

Step 305 , when the number of audio frames containing the target alarm tone within the predetermined duration reaches the number threshold, it is determined that the ambient sound contains the target alarm tone.

Since the terminal processes the audio data into frames before identifying the target alarm tone, and the time of one frame of audio is very short, when an audio frame contains the target alarm tone, it cannot be ruled out that there are other similar sounds or feature extraction. In the event of an error in the data processing process at the time, it cannot be immediately determined that the ambient sound contains the target alarm tone. Therefore, the terminal sets a predetermined duration, and when the output result of the voice recognition model indicates that the number of audio frames containing the target alarm tone within the predetermined duration reaches the threshold, it is determined that the ambient sound contains the target alarm tone.

Illustratively, the terminal sets the predetermined duration to 5 seconds and the number threshold to 2. When the terminal recognizes that the target alarm tone is contained in 2 or more than 2 audio frames within 5 seconds, it is determined that the current ambient sound contains the target alarm tone. Target alarm bell.

Step 306 , acquiring the last alarm bell recognition time, where the last alarm bell recognition time is the last time when the target alarm tone was recognized in the ambient sound.

When the output result of the sound recognition model indicates that the number of audio frames containing the target alarm tone within the predetermined duration reaches the threshold, the terminal records the current moment and obtains the last time that the ambient sound contains the target alarm tone, that is, Get the last alarm bell recognition time.

Step 307 , if the time interval between the last alarm bell recognition time and the current alarm bell recognition time is greater than the time interval threshold, the operation of adding one to the number of traveled stations is performed.

During the actual ride, the closing and opening alarms of the vehicle may be the same, which will cause the terminal to recognize two alarms at the same station, or other vehicles of the same vehicle may have the same alarm tones as the vehicle where the terminal is located. , when the vehicle where the terminal is parked at a certain station emits the same alarm tone, the terminal count will be wrong. Therefore, the terminal presets the time interval threshold. If the time between the last alarm bell recognition time and the current alarm bell recognition time is If the time interval is greater than the time interval threshold, add one to the number of traveled stations.

Illustratively, the preset time interval threshold is 1 minute, and each time the terminal recognizes that the ambient sound contains the target alarm tone, it records the current time and obtains the last alarm recognition time, if the time interval between the two is greater than one minutes, it is determined that the vehicle has traveled one stop, and the operation of adding one to the number of traveled stops is performed.

Step 308 , when the number of traveled stations reaches the target number of stations, an arrival reminder is performed, the target station number is the number of stations between the starting station and the target station, and the target station is a transfer station or a destination station.

For the implementation of step 308, reference may be made to the foregoing step 204, and details are not described herein again in this embodiment.

In the embodiment of the present application, the audio data of the ambient sound is framed and windowed, and the audio frame is feature extracted to obtain data that can be recognized by the voice recognition model; by post-processing the output results of the voice recognition model, Confirm whether the identified alarm tone is the target alarm tone, avoid misidentifying the alarm tone or similar sounds of other vehicles as the target alarm tone, and improve the accuracy of the arrival reminder.

Since the terminal needs to always turn on the microphone to obtain ambient sound while the vehicle is running, and input the audio data of the ambient sound into the sound recognition model for recognition, the sound recognition model is always in a working state. In order to reduce the power consumption of the terminal, in a possible implementation manner, the terminal uses a simple similarity model as a first-level voice recognition model for recognition, and at the same time, in order to improve the accuracy of the voice recognition model, a convolutional neural network (Convolutional Neural Networks, CNN) model as a secondary sound recognition model. The sound recognition process is shown in Figure 5. The terminal executes step 501 to input environmental sounds. Before identifying the environmental sounds, it extracts audio data features through step 502, and then inputs the extracted audio feature matrix into a first-level sound recognition model. Step 503 judges the similarity, when the similarity judgment result cannot determine whether the ambient sound contains the target alarm tone, the audio feature matrix is input into the secondary sound recognition model, and is classified by the CNN in step 504. If the recognition result of the CNN model is that the ambient sound contains If the target alarm tone is present, the post-processing of step 505 and step 506 are performed to determine whether to add an operation. If the identification result is that the ambient sound does not contain the target alarm tone, the terminal continues to identify the next frame of audio data.

In a possible implementation manner, on the basis of FIG. 3 , as shown in FIG. 6 , the above step 304 includes steps 304a to 304d.

Step 304a: Input the audio feature matrix into the first-level sound recognition model to obtain the first recognition result output by the first-level sound recognition model.

In a possible implementation, the first-level sound recognition model adopts a cosine similarity model, and the model establishment and recognition process is shown in FIG. 7 , and the upper branch is the model establishment process. Extract the audio features of the audio data obtained (for the specific process, refer to steps 302 to 303), and further, take the average value of the extracted audio features through step 702, and then obtain the 32*40 audio of each target alarm tone. The feature matrix is converted into a one-dimensional feature vector with 1280 elements, added to the database, and finally a feature database is generated through step 703 . After the model is established, the specific identification process is shown in the lower branch of Figure 7. After inputting the ambient sound audio data, the audio feature extraction is carried out through step 704, and then through step 705, a feature vector (corresponding to the audio frame) is generated according to the extracted audio features. The audio feature matrix is converted into a one-dimensional feature vector containing 1280 elements), and finally traverses the one-dimensional feature vector Y _i in the feature database through step 706, and calculates its cosine similarity with the one-dimensional feature vector X of the current audio frame, The calculation formula is:

Among them, n represents the number of types of target alarm tone, that is, the total number of all one-dimensional feature vectors in the feature database.

In other possible implementations, the first-level voice recognition model may use other similarity models, such as Manhattan distance model, Mahalanobis distance model, Euclidean distance model, and so on. The embodiments of the present application only take the cosine similarity model as an example for schematic illustration.

After the terminal obtains the first recognition result of the audio feature matrix by the first-level sound recognition model, it cannot directly determine whether the current ambient sound contains the target alarm tone, and needs to determine whether the first recognition result of the first-level sound recognition model meets the output conditions.

In a possible implementation, step 304a includes the following steps one to two.

1. If the first identification result indicates that the similarity is greater than the first similarity threshold or smaller than the second similarity threshold, it is determined that the first identification result meets the output condition, and the first similarity threshold is greater than the second similarity threshold.

In a possible implementation, when the first recognition result indicates that the similarity is greater than or equal to the first similarity threshold, the terminal determines that the current ambient sound contains the target alarm tone; when the first recognition result indicates that the similarity is less than the second similarity When the degree threshold is reached, the terminal determines that there is no target alarm tone in the current ambient sound, and the first similarity threshold is greater than the second similarity threshold. The above two results meet the output conditions.

Illustratively, the first similarity threshold is set to 0.9, and the second similarity threshold is 0.5. When the first recognition result indicates that the similarity is greater than or equal to 0.9, or less than 0.5, it is determined that the first recognition result meets the output condition.

2. If the first identification result indicates that the similarity is greater than the second similarity threshold and smaller than the first similarity threshold, it is determined that the first identification result does not meet the output condition.

When the first recognition result indicates that the similarity is greater than the second similarity threshold and less than the first similarity threshold, the first-level sound recognition model cannot determine whether the ambient sound contains the target alarm tone, and the first recognition result does not meet the output conditions. A secondary sound recognition model is required to further identify the audio feature matrix of the current audio frame.

Illustratively, the first similarity threshold is set to 0.9, and the second similarity threshold is set to 0.5. When the first recognition result is less than 0.9 and greater than 0.5, it is determined that the first recognition result does not meet the output condition.

Step 304b, if the first recognition result meets the output condition, the first recognition result is used as the target alarm tone recognition result and output.

When the first recognition result meets the output conditions, it indicates that the first-level sound recognition model can determine whether the current ambient sound contains the target alarm tone, so the terminal outputs the first recognition result as the target alarm tone recognition result.

Illustratively, when the cosine similarity of the first identification result is 0.95, it is determined that the current ambient sound contains the target alarm tone, and the first identification result of 0.95 is output.

Step 304c, if the first recognition result does not meet the output conditions, input the audio feature matrix into the secondary voice recognition model to obtain the second recognition result output by the secondary voice recognition model.

When the first recognition result does not meet the output conditions, it indicates that the first-level sound recognition model cannot determine whether the current ambient sound contains the target alarm tone. At this time, the second-level sound recognition model needs to be further identified, and the first recognition result does not meet the output conditions. The audio feature matrix corresponding to the audio frame is input to the secondary sound recognition model, and the second recognition result output by the secondary sound recognition model is obtained.

In a possible implementation, the secondary voice recognition model adopts a CNN classification model, and the model training process is as follows:

1. Convert the collected ambient sound containing the target alarm tone into a spectrogram.

As shown in Figure 8, the difference between the target alarm tone and other ambient sounds can be clearly seen. The short line in the black box in the figure is the spectrum of the target alarm tone. The target alarm tone is marked as a positive sample, and the rest of the ambient sounds are used as negative samples. .

2. Perform feature extraction on the collected ambient sound.

The feature extraction method of the environmental sound collected in advance is the same as the feature extraction method in the above-mentioned embodiment, the audio feature matrix corresponding to each audio frame is used as a training sample, the label corresponding to the audio feature matrix of the target alarm tone is 0, and the other environmental sounds The corresponding label of the audio feature matrix is 1.

Third, build a CNN model.

In a possible implementation manner, the structure of the CNN model is shown in FIG. 9 , the first convolutional layer 901 and the second convolutional layer 902 are used to extract the features of the input audio feature matrix, the first fully connected layer 903 and the first convolutional layer 902 The second fully-connected layer 904 integrates the category-distinguishing information in the convolutional layers 901 and 902, and finally connects to a normalized exponential function (Softmax) 905 to classify the information integrated by the fully-connected layer to obtain a second identification result.

Fourth, build the loss function of the model.

When the vehicle is driving, the target alarm tone is usually only about 5 seconds, while the rest of the ambient sound lasts for several minutes, and the positive and negative sample data are very unbalanced. Therefore, the focus loss function (Focal loss) is selected to solve the problem of unbalanced samples. Focal loss The formula is as follows:

Among them, y′ is the probability output by the CNN classification model, y is the label corresponding to the training sample, and α and γ are manual adjustment parameters used to adjust the ratio of positive and negative samples.

5. Import training samples for model training.

In a possible implementation, a CNN classification model can be trained by using the Tensorflow system, and Focal loss and gradient descent algorithms can be used until the model converges.

In a possible implementation manner, the secondary voice recognition model may also use other traditional machine learning classifiers or deep learning classification models, which is not limited in this embodiment.

Step 304d, taking the second recognition result as the target alarm tone recognition result and outputting it.

The secondary sound recognition model is a CNN classification model with high accuracy. Therefore, after obtaining the second recognition result of the CNN classification model, the second recognition result is used as the target alarm tone recognition result and output.

In the embodiment of the present application, two kinds of sound recognition models are used. The first-level sound recognition model has low power consumption and is easy to implement. It is enabled in real time and recognizes the audio data of ambient sounds. The second-level sound recognition model has high accuracy but high power consumption. When the first-level sound recognition model cannot determine whether the current environmental sound contains the target alarm tone, the second-level sound recognition model is turned on to recognize the current environmental sound, which improves the accuracy of the model recognition results and reduces the power consumption of the terminal.

Please refer to FIG. 10 , which shows a structural block diagram of an arrival reminder device provided by an exemplary embodiment of the present application. The apparatus can be implemented as all or a part of the terminal through software, hardware or a combination of the two. The device includes:

The collection module 1001 is used to collect ambient sound through a microphone when it is in a vehicle;

an identification module 1002, configured to identify the ambient sound;

The counting module 1003 is used for adding one to the number of stations that have been traveled when it is identified that the ambient sound contains a target alarm tone, and the target alarm tone is an open-door alarm tone or a door-close alarm tone;

Reminder module 1004, for when the number of stations that have been traveled reaches the target number of stations, a reminder of arrival is performed, and the number of target stations is the number of stations between the starting station and the target station, and the target station is a transfer station or destination site.

Optionally, the identification module 1002 includes:

a frame-by-frame processing unit, configured to perform frame-by-frame processing on the audio data corresponding to the ambient sound to obtain audio data corresponding to different audio frames;

a feature extraction unit, configured to perform feature extraction on the audio data corresponding to each audio frame to obtain a corresponding audio feature matrix;

The sound recognition unit is used for inputting the audio feature matrix into a sound recognition model to obtain a target alarm tone recognition result output by the voice recognition model, and the target alarm tone recognition result is used to indicate whether the audio frame contains the Target alarm bell.

Optionally, the voice recognition model includes a primary voice recognition model and a secondary voice recognition model, and the recognition accuracy of the secondary voice recognition model is higher than the recognition accuracy of the primary voice recognition model;

The voice recognition unit is also used for:

Inputting the audio feature matrix into the first-level sound recognition model to obtain the first recognition result output by the first-level sound recognition model;

If the first recognition result meets the output condition, the first recognition result is used as the target alarm tone recognition result and output;

If the first recognition result does not meet the output condition, then input the audio feature matrix into the secondary voice recognition model to obtain a second recognition result output by the secondary voice recognition model;

The second recognition result is used as the target alarm tone recognition result and output.

Optionally, the first-level sound recognition model is used to calculate the similarity between the audio feature matrix and the sample feature matrix in the feature database, and the sample feature matrix is the audio feature matrix of the target alarm tone;

The voice recognition unit is also used for:

If the first identification result indicates that the similarity is greater than a first similarity threshold or smaller than a second similarity threshold, it is determined that the first identification result meets the output condition, and the first similarity threshold is greater than the the second similarity threshold;

If the first identification result indicates that the similarity is greater than the second similarity threshold and smaller than the first similarity threshold, it is determined that the first identification result does not meet the output condition.

Optionally, the identification module 1002 further includes:

A determining unit, configured to determine that the ambient sound contains the target alarm tone when the number of audio frames containing the target alarm tone within a predetermined duration reaches a threshold number.

Optionally, the counting module 1003 includes:

an acquisition unit, configured to acquire the last alarm bell recognition time, where the last alarm bell recognition time is the time when the target alarm bell was identified in the ambient sound last time;

A comparison unit, configured to perform an operation of adding one to the number of traveled stations if the time interval between the last alarm bell recognition time and the current alarm bell recognition time is greater than a time interval threshold.

In this embodiment, the arrival reminder device provided by the embodiment is used to collect ambient sounds in real time and identify whether the current ambient sound contains target alarm bells, so that when it is identified that the target alarm bells are included, the number of traveled stops is updated. , when the number of traveled stations reaches the target number of stations, a reminder for entering the station is carried out; since the alarm bell is used to warn passengers, the sound characteristics are more obvious and easy to be recognized, so the arrival reminder is based on the alarm bell in the ambient sound. It can improve the accuracy and effectiveness of the arrival reminder.

Please refer to FIG. 11 , which shows a structural block diagram of a terminal 1100 provided by an exemplary embodiment of the present application. The terminal 1100 may be an electronic device, such as a smart phone, a tablet computer, an electronic book, a portable personal computer, etc., on which application programs are installed and run. The terminal 1100 in this application may include one or more of the following components: a processor 1110 , a memory 1120 and a screen 1130 .

The processor 1110 may include one or more processing cores. The processor 1110 uses various interfaces and lines to connect various parts of the entire terminal 1100, and executes the terminal by running or executing the instructions, programs, code sets or instruction sets stored in the memory 1120, and calling the data stored in the memory 1120. 1100's various functions and processing data. Optionally, the processor 1110 may employ at least one of a digital signal processing (Digital Signal Processing, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA). implemented in hardware. The processor 1110 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing the content that needs to be displayed on the screen 1130; the modem is used for processing wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1110, and is implemented by a communication chip alone.

The memory 1120 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory). Optionally, the memory 1120 includes a non-transitory computer-readable storage medium. Memory 1120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 1120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing the operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the above method embodiments, etc., the operating system can be an Android (Android) system (including a system based on the deep development of the Android system), an IOS system developed by Apple (including a system based on the deep development of the IOS system) or other systems. The storage data area may also store data created by the terminal 1100 during use (such as phone book, audio and video data, chat record data) and the like.

The screen 1130 may be a capacitive touch display screen for receiving a user's touch operation on or near it using any suitable object such as a finger, a stylus pen, etc., and displaying a user interface of each application program. The touch display screen is usually provided on the front panel of the terminal 1100 . The touch screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed to be a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.

In addition, those skilled in the art can understand that the structure of the terminal 1100 shown in the above drawings does not constitute a limitation on the terminal 1100, and the terminal may include more or less components than those shown in the drawings, or combine some components, or a different arrangement of components. For example, the terminal 1100 further includes components such as a radio frequency circuit, a photographing component, a sensor, an audio circuit, a wireless fidelity (Wireless Fidelity, WiFi) component, a power supply, and a Bluetooth component, which will not be repeated here.

Embodiments of the present application further provide a computer-readable medium, where at least one instruction is stored in the computer-readable medium, and the at least one instruction is loaded and executed by the processor to implement the arrival reminder as described in the above embodiments method.

Embodiments of the present application further provide a computer program product, where the computer program product stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the arrival reminder method described in each of the above embodiments.

Those skilled in the art should realize that, in one or more of the above examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (16)

1. an arrival reminder method, is characterized in that, described method comprises: When in a vehicle, collect ambient sound through a microphone; identifying the ambient sound; When it is recognized that the ambient sound contains a target alarm bell, the operation of adding one to the number of stations that have been traveled is performed, and the target alarm bell is an open-door alarm bell or a door-close alarm bell; When the number of traveled stations reaches the target number of stations, an arrival reminder is performed, and the target station number is the number of stations between the starting station and the target station, and the target station is a transfer station or a destination station. 2. The method according to claim 1, wherein the identifying the ambient sound comprises: Framing processing is performed on the audio data corresponding to the ambient sound to obtain audio data corresponding to different audio frames; Perform feature extraction on the audio data corresponding to each audio frame to obtain a corresponding audio feature matrix; Inputting the audio feature matrix into a sound recognition model to obtain a target alarm tone recognition result output by the voice recognition model, where the target alarm tone recognition result is used to indicate whether the audio frame contains the target alarm tone. 3. The method according to claim 2, wherein the sound recognition model comprises a first-level sound recognition model and a second-level sound recognition model, and the recognition accuracy of the second-level sound recognition model is higher than the first-level sound recognition model The recognition accuracy of the voice recognition model; The described audio feature matrix is input into the sound recognition model to obtain the target alarm tone recognition result output by the sound recognition model, including: Inputting the audio feature matrix into the first-level sound recognition model to obtain the first recognition result output by the first-level sound recognition model; If the first recognition result meets the output condition, the first recognition result is used as the target alarm tone recognition result and output; If the first recognition result does not meet the output condition, then input the audio feature matrix into the secondary voice recognition model to obtain a second recognition result output by the secondary voice recognition model; The second recognition result is used as the target alarm tone recognition result and output. 4. The method according to claim 3, wherein the first-level voice recognition model is used to calculate the similarity between the audio feature matrix and the sample feature matrix in the feature database, and the sample feature matrix is The audio feature matrix of the target alarm tone; After inputting the audio feature matrix into the first-level sound recognition model, and obtaining the first recognition result output by the first-level sound recognition model, it also includes: If the first identification result indicates that the similarity is greater than a first similarity threshold or smaller than a second similarity threshold, it is determined that the first identification result meets the output condition, and the first similarity threshold is greater than the the second similarity threshold; If the first identification result indicates that the similarity is greater than the second similarity threshold and smaller than the first similarity threshold, it is determined that the first identification result does not meet the output condition. 5. The method according to claim 3, wherein the secondary voice recognition model is a two-class model using a convolutional neural network (CNN), and the two-class model is trained according to positive and negative samples, and uses focal loss focal. loss is the loss function, which is obtained by training with the gradient descent algorithm. 6. The method according to any one of claims 2 to 5, wherein the audio feature matrix is input into a sound recognition model, after obtaining the target alarm tone recognition result output by the sound recognition model, further comprising: When the number of audio frames containing the target alarm tone within a predetermined duration reaches a threshold, it is determined that the ambient sound contains the target alarm tone. 7. The method according to any one of claims 1 to 6, wherein the operation of adding one to the number of traveled stations comprises: Obtain the last alarm bell recognition time, the last alarm bell recognition time is the last time when the target alarm bell was recognized in the ambient sound; If the time interval between the last alarm bell recognition time and the current alarm bell recognition time is greater than the time interval threshold, the operation of adding one to the number of traveled stations is performed. 8. An arrival reminder device, wherein the device comprises: The acquisition module is used to collect ambient sound through a microphone when it is in a vehicle; an identification module for identifying the ambient sound; a counting module, used for adding one to the number of stations that have been traveled when it is identified that the ambient sound contains a target alarm tone, and the target alarm tone is an open-door alarm tone or a door-closed alarm tone; Reminder module, used to remind the arrival of the station when the number of the traveled stations reaches the target station number, the target station number is the station number between the starting station and the target station, and the target station is a transfer station or a destination local site. 9. The device according to claim 8, wherein the identification module comprises: a frame-by-frame processing unit, configured to perform frame-by-frame processing on the audio data corresponding to the ambient sound to obtain audio data corresponding to different audio frames; a feature extraction unit, configured to perform feature extraction on the audio data corresponding to each audio frame to obtain a corresponding audio feature matrix; The sound recognition unit is used for inputting the audio feature matrix into a sound recognition model to obtain a target alarm tone recognition result output by the voice recognition model, and the target alarm tone recognition result is used to indicate whether the audio frame contains the Target alarm bell. 10 . The device according to claim 9 , wherein the voice recognition model comprises a first-level voice recognition model and a second-level voice recognition model, and the recognition accuracy of the second-level voice recognition model is higher than that of the first-level voice recognition model. 11 . The recognition accuracy of the voice recognition model; The voice recognition unit is also used for: Inputting the audio feature matrix into the first-level sound recognition model to obtain the first recognition result output by the first-level sound recognition model; If the first recognition result meets the output condition, the first recognition result is used as the target alarm tone recognition result and output; If the first recognition result does not meet the output condition, then input the audio feature matrix into the secondary voice recognition model to obtain a second recognition result output by the secondary voice recognition model; The second recognition result is used as the target alarm tone recognition result and output. 11. The device according to claim 10, wherein the first-level voice recognition model is used to calculate the similarity between the audio feature matrix and the sample feature matrix in the feature database, and the sample feature matrix is The audio feature matrix of the target alarm tone; The voice recognition unit is also used for: If the first identification result indicates that the similarity is greater than a first similarity threshold or smaller than a second similarity threshold, it is determined that the first identification result meets the output condition, and the first similarity threshold is greater than the the second similarity threshold; If the first identification result indicates that the similarity is greater than the second similarity threshold and smaller than the first similarity threshold, it is determined that the first identification result does not meet the output condition. 12. The device according to claim 10, wherein the secondary voice recognition model is a binary classification model using CNN, and the binary classification model is trained according to positive and negative samples, and takes focalloss as a loss function, and uses gradients as a loss function. The descent algorithm is trained. 13. The device according to any one of claims 9 to 12, wherein the identification module further comprises: A determining unit, configured to determine that the ambient sound contains the target alarm tone when the number of audio frames containing the target alarm tone within a predetermined duration reaches a threshold number. 14. The device according to any one of claims 8 to 13, wherein the counting module comprises: an acquisition unit, configured to acquire the last alarm bell recognition time, where the last alarm bell recognition time is the time when the target alarm bell was identified in the ambient sound last time; A comparison unit, configured to perform an operation of adding one to the number of traveled stations if the time interval between the last alarm bell recognition time and the current alarm bell recognition time is greater than a time interval threshold. 15. A terminal, characterized in that the terminal comprises a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement any one of claims 1 to 7. 1. The described arrival reminder method. 16. A computer-readable storage medium, characterized in that the storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the arrival station according to any one of claims 1 to 7 reminder method.

Images