WO2020125397A1 - 音频数据推送方法、装置、计算机设备和存储介质 - Google Patents

音频数据推送方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2020125397A1
WO2020125397A1 PCT/CN2019/122546 CN2019122546W WO2020125397A1 WO 2020125397 A1 WO2020125397 A1 WO 2020125397A1 CN 2019122546 W CN2019122546 W CN 2019122546W WO 2020125397 A1 WO2020125397 A1 WO 2020125397A1
Authority
WO
WIPO (PCT)
Prior art keywords
expression
data
audio data
facial
classifier
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/122546
Other languages
English (en)
French (fr)
Inventor
江波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OneConnect Smart Technology Co Ltd
Original Assignee
OneConnect Smart Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OneConnect Smart Technology Co Ltd filed Critical OneConnect Smart Technology Co Ltd
Publication of WO2020125397A1 publication Critical patent/WO2020125397A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63—Querying

Definitions

  • the present application relates to an audio data pushing method, device, computer equipment and storage medium.
  • the driver's behavior is restricted during driving. Updating music in a traditional car player is cumbersome, and usually plays downloaded music or other audiobooks for users randomly. When users do not like current music, they need to manually replace the current music, which brings inconvenience to users. With the rapid development of artificial intelligence technology, there have been some operations that perform corresponding operations by recognizing the user's voice commands, but there is greater noise pollution during driving, the accuracy of voice command recognition is low, and user replacement operations are more troublesome.
  • the inventor realizes that in some existing ways of automatically pushing audio data, usually the audio data related to the ranking or the user's history is pushed, and the automatically pushed audio playback data does not necessarily conform to the user's preferences and emotions , Resulting in a low push accuracy for audio playback data such as music or audiobooks.
  • an audio data pushing method device, computer device, and storage medium are provided.
  • a method for pushing audio data includes:
  • An audio data pushing device the device includes:
  • the data acquisition module is used to acquire the multi-frame face pictures uploaded by the user terminal, the multi-frame face pictures carrying the user identification;
  • a face recognition module used to recognize a face image according to the multi-frame face pictures, and extract face features corresponding to the face image
  • An expression classification module used to obtain a trained expression classifier, and classify the facial features through the expression classifier to obtain an expression category corresponding to the user identifier;
  • a data matching module used to obtain an expression mapping data table, and corresponding audio data according to the expression type
  • a data pushing module is used to push the audio data to the user terminal.
  • a computer device includes a memory and one or more processors.
  • the memory stores computer-readable instructions.
  • the one or more processors are executed The following steps:
  • One or more non-volatile computer-readable storage media storing computer-readable instructions.
  • the computer-readable instructions When executed by one or more processors, the one or more processors perform the following steps:
  • FIG. 1 is an application scenario diagram of an audio data pushing method according to one or more embodiments.
  • FIG. 2 is a schematic flowchart of a method for pushing audio data according to one or more embodiments.
  • FIG. 3 is a schematic flowchart of the steps of training an expression classifier according to one or more embodiments.
  • FIG. 4 is a schematic flowchart of a step of classifying facial features by an expression classifier according to one or more embodiments.
  • FIG. 5 is a block diagram of an audio data pushing device according to one or more embodiments.
  • Figure 6 is a block diagram of a computer device in accordance with one or more embodiments.
  • the audio data pushing method provided by this application can be applied in the application environment shown in FIG. 1.
  • the user terminal 102 communicates with the server 104 through the network through the network.
  • the user terminal 102 may be an in-vehicle system terminal installed in a vehicle with a camera function, or a smartphone, a tablet computer, and a portable device with a camera function separated from the vehicle.
  • the server 104 may use an independent server or multiple servers To form a server cluster.
  • the server 104 obtains the multi-frame face pictures uploaded by the user terminal 102, and the multi-frame face pictures carry the user identification.
  • the server 104 further recognizes the facial image based on the multi-frame facial pictures, and extracts facial features corresponding to the facial image.
  • the server 104 further obtains a preset expression classifier, and classifies the facial features through the expression classifier to obtain the expression category corresponding to the user identification.
  • a preset expression classifier based on a neural network model to recognize and classify the user's multi-frame facial features
  • the server 104 After the server 104 recognizes the expression type corresponding to the current facial feature of the user, it obtains a preset expression mapping data table, obtains corresponding audio data according to the expression type, and pushes the audio data to the user terminal 102.
  • the neural network-based facial expression classifier By recognizing the facial expression characteristics of the user during driving, and classifying the user's facial expression by the neural network-based facial expression classifier, the current facial expression and emotion of the user can be accurately and effectively identified, and then effectively based on the user Expressions and emotions push corresponding audio data, which can effectively improve the accuracy of pushing audio data.
  • a method for pushing audio data is provided. Taking the method applied to the server in FIG. 1 as an example for illustration, it includes the following steps:
  • Step 202 Acquire multi-frame face pictures uploaded by the user terminal, where the multi-frame face pictures carry the user identification.
  • the user terminal may be an in-vehicle system terminal installed in a vehicle with a camera function, or a smartphone, a tablet computer, and a portable device with a camera function that are separate from the vehicle.
  • the user terminal can be triggered to play the recommendation instruction.
  • the camera device captures the user's face picture according to the preset frequency, and uploads the captured multi-frame face pictures to the server in time series.
  • the multi-frame face pictures captured by the vehicle-mounted terminal may be continuous multi-frame still pictures or dynamic videos.
  • the user terminal can also automatically capture the user's face picture or face video through the camera device according to the preset frequency, and follow the captured face picture or face video according to The time series is uploaded to the server.
  • the server further acquires the multi-frame face pictures uploaded by the user terminal and captured by the camera device, and the multi-frame face pictures carry the user identification.
  • Step 204 Recognize the face image according to the multi-frame face pictures, and extract the face features corresponding to the face image.
  • the server After the server obtains the multi-frame face pictures captured by the user terminal through the camera device, it performs face recognition on the multi-frame face pictures, recognizes the face images in the multi-frame face pictures, and recognizes the multi-frame face pictures The image is subjected to feature extraction to extract facial features corresponding to each frame of facial image. Multi-frame face pictures also include corresponding image sequences. Specifically, the server locates the key points of the multi-frame face pictures, recognizes the multi-frame face images in the multi-frame face pictures after positioning according to a preset face recognition algorithm, and performs feature extraction on the multi-frame face images To obtain the facial features corresponding to the multi-frame facial images, thereby effectively identifying and extracting the facial features corresponding to the facial images.
  • Step 206 Acquire a trained expression classifier, and classify the facial features by the expression classifier to obtain the expression category corresponding to the user identification.
  • the server After extracting the facial features corresponding to each frame of the facial image, the server obtains a preset expression classifier.
  • the facial expression classifier may be a pre-trained classifier based on a neural network model. Specifically, the server inputs the extracted facial features corresponding to each frame of facial images into the expression classifier, recognizes the facial features of each frame through the convolutional neural network in the expression classifier, and according to the image sequence and multiple frames of faces The feature calculates the corresponding dynamic face feature, and then calculates the probability value that the dynamic face feature belongs to each expression category. The server obtains the expression category with the highest probability value, thereby obtaining the expression category corresponding to the user identification.
  • Step 208 Acquire a preset expression mapping data table, and obtain corresponding audio data according to the expression type.
  • Step 210 Push the audio data to the user terminal.
  • the server classifies the multi-frame facial features through the expression classifier to obtain the corresponding expression category, and then further obtains a preset expression mapping data table.
  • the expression mapping data table includes expression categories and corresponding mood tags, and audio data identifiers corresponding to the mood tags.
  • the server may obtain multiple video data in advance, and each video data may include a corresponding video data identifier.
  • the server further obtains a preset video data classification model, classifies multiple video data through the video data classification model, and adds corresponding mood tags to the video data.
  • Mood tags can correspond to one or more expression categories.
  • the server obtains the corresponding expression category according to the mood tag, and establishes an expression mapping data table according to the mood tag and the expression class and the corresponding audio data identification.
  • the server matches the mood tag corresponding to the expression category according to the expression mapping data table, and obtains the corresponding audio data according to the mood tag.
  • the audio data may include various music, broadcast audio, audiobooks and other data.
  • the mood tag may correspond to a music playlist
  • the server recognizes the current expression category of the user
  • the audio data in the playlist corresponding to the expression category is pushed to the user terminal, so that the user terminal plays the pushed audio data.
  • the server obtains the multi-frame face pictures uploaded by the user terminal, and the multi-frame face pictures carry the user identification.
  • the server further recognizes the facial image based on the multi-frame facial pictures, and extracts facial features corresponding to the facial image.
  • the server further obtains a preset expression classifier, and classifies the facial features through the expression classifier to obtain the expression category corresponding to the user identifier.
  • the server After the server recognizes the expression category corresponding to the current facial feature of the user, it obtains a preset expression mapping data table, obtains corresponding audio data according to the expression category, and pushes the audio data to the user terminal.
  • the server recognizes the facial expression characteristics of the user during driving, and classifying the user's facial expression by the neural network-based facial expression classifier, the current facial expression and emotion of the user can be accurately and effectively identified, and then effectively based on the user Expressions and emotions push corresponding audio data, which can effectively improve the accuracy of pushing audio data.
  • the step of recognizing the face image based on the multi-frame face pictures and extracting the face features corresponding to the face image includes: positioning key points on the multi-frame face pictures to obtain the positioned multi-frame people Face picture; recognize the multi-frame face images in the multi-frame face pictures after positioning according to the preset face recognition algorithm, and obtain the facial features corresponding to the multi-frame face images.
  • the user terminal can capture the user's multi-frame face pictures through the camera device according to a preset frequency, and upload the captured multi-frame face pictures to the server.
  • the multi-frame face pictures carry the user identification.
  • the server After acquiring the multi-frame face pictures uploaded by the user terminal, the server further recognizes the facial image according to the multi-frame face pictures, and extracts facial features corresponding to the facial image.
  • the server detects key points in multi-frame face pictures according to a preset algorithm, and locates key points in the multi-frame face pictures.
  • the server further recognizes the multi-frame face images in the multi-frame face pictures after the keypoint positioning according to the preset face recognition algorithm.
  • the server can also segment the face pictures to extract the face image parts, and further extract the face images
  • the human face image is gray-scaled and normalized.
  • the server performs feature extraction on the multi-frame facial images to obtain the facial features corresponding to the multi-frame facial images, thereby effectively identifying and extracting the facial features corresponding to the facial images.
  • the server can also determine the start frame and end frame of the user's facial expression and the duration by identifying whether the facial features in each frame of the face image are consistent, and extract the start frame and end frame of the facial expression and the duration
  • the user's current multi-frame facial features are displayed. In this way, multi-frame facial features corresponding to the current expression of the user can be accurately and effectively extracted.
  • the server After the server recognizes and extracts the multi-frame facial features of the user, it further obtains a preset facial expression classifier, classifies the facial features through the facial expression classifier, and obtains the facial expression category corresponding to the user identifier.
  • a preset facial expression classifier By using an expression classifier based on a neural network model to recognize and classify the user's multi-frame facial features, the user's current expression category can be accurately and effectively identified.
  • the server After the server recognizes the expression category corresponding to the current facial feature of the user, it obtains a preset expression mapping data table, obtains corresponding audio data according to the expression category, and pushes the audio data to the user terminal.
  • the multi-frame facial features corresponding to the current facial expression of the user can be accurately and effectively extracted.
  • the expression classifier based on neural network to classify the user's expression can accurately and effectively identify the user's current expression and emotion, and then can effectively push the corresponding audio data according to the user's expression and emotion, which can effectively Improve the accuracy of pushing audio data.
  • the method before obtaining the preset expression classifier, the method further includes a step of training the expression classifier, which specifically includes the following content:
  • Step 302 Acquire multiple expression data from a preset database.
  • Step 304 Use the obtained multiple expression data to generate a training set and a verification set; the training set includes labeled expression data, and the verification set includes unlabeled expression data.
  • Step 306 Use the marked expression data in the training set to train through a preset algorithm to obtain a preliminary expression classifier.
  • Step 308 Input the unlabeled expression data in the verification set to the preliminary expression classifier for verification training.
  • Step 310 until the amount of verification set data that reaches the preset probability value reaches the preset ratio, the training is stopped, and the trained expression classifier is obtained.
  • the server Before acquiring the preset expression classifier, the server also needs to use a large amount of expression data to train to obtain an expression classifier.
  • the server may obtain a large amount of expression data from a local or third-party database, and the expression data may include expression pictures, expression videos, and dynamic expression images.
  • the server further generates a training set and a verification set from the acquired large amount of facial expression data, where the facial expression data in the training set may be facial expression data manually annotated, and the facial expression data in the verification set may be unlabeled facial expression data.
  • the server performs feature extraction on the expression data in the training set and performs training through a preset algorithm to obtain a preliminary expression classifier.
  • the server may perform a convolution operation on the facial image in the expression data through CNN (Convolutional Neural Network) to obtain facial features corresponding to the facial image.
  • CNN Convolutional Neural Network
  • the multi-frame facial features corresponding to the multi-frame face pictures in the expression data are input to BLSTM (Bidirectional Long-Short-term Memory, bidirectional long-term memory neural network), and the dynamic expression features corresponding to the expression data are calculated by a preset function.
  • the server further calculates the probability value of each dynamic expression feature corresponding to each label category according to a preset algorithm, so as to train the expression classifier, thereby effectively obtaining a preliminary expression classifier.
  • the server further inputs the data in the verification set into the initial expression classifier for continuous training to obtain the probability value of each expression data corresponding to each category, and obtains the expression data that reaches the preset probability threshold when the preset probability threshold is reached.
  • the training is stopped, and the trained expression classifier is obtained.
  • the multi-frame face pictures include corresponding image sequences, and the steps of classifying the face features by the expression classifier to obtain the expression category corresponding to the user identification include the following contents:
  • step 402 the multi-frame facial features corresponding to the multi-frame face pictures are input to the facial expression classifier, and the facial feature vector of each frame is identified by the convolutional neural network in the facial expression classifier.
  • Step 404 Calculate the corresponding dynamic expression features according to the image sequence and the multi-frame face feature vectors.
  • Step 406 Calculate the probability value that the dynamic expression feature belongs to each expression category.
  • Step 408 Obtain the expression type with the highest probability value to obtain the expression type corresponding to the user ID.
  • the user terminal can capture the user's multi-frame face pictures through the camera device according to a preset frequency, and upload the captured multi-frame face pictures to the server.
  • the multi-frame face pictures carry the user identification.
  • the server After acquiring the multi-frame face pictures uploaded by the user terminal, the server further recognizes the facial image according to the multi-frame face pictures, and extracts facial features corresponding to the facial image.
  • the server After extracting the facial features corresponding to each frame of the facial image, the server obtains a preset expression classifier.
  • the expression classifier may be a classifier based on a neural network model that is pre-trained. Specifically, the server inputs the extracted facial features corresponding to each frame of facial images into the expression classifier, recognizes the facial feature vectors of each frame through the convolutional neural network in the expression classifier, and according to the image sequence and the multi-frame human The face feature vector calculates the corresponding dynamic face feature, and then calculates the probability value that the dynamic face feature belongs to each expression category. The server obtains the expression category with the highest probability value, thereby obtaining the expression category corresponding to the user identification.
  • the server may perform a convolution operation on the multi-frame face features corresponding to the multi-frame face images through CNN (Convolutional Neural Network) to identify and obtain the face feature vector corresponding to the face image.
  • Multi-frame face feature vectors are input to BLSTM (Bidirectional Long-Short-term Memory, bidirectional long-term memory neural network), and dynamic expression features corresponding to multi-frame face feature vectors are calculated by a preset function.
  • the server further calculates the probability value of each dynamic expression feature corresponding to each label category according to a preset algorithm, and the server obtains the expression category with the highest probability value.
  • the server After the server recognizes the expression category corresponding to the current facial feature of the user, it obtains a preset expression mapping data table, obtains corresponding audio data according to the expression category, and pushes the audio data to the user terminal. Recognition and classification of the user's expression through the neural network-based expression classifier can accurately and effectively identify the user's current expression and emotion, and then can effectively push the corresponding audio data according to the user's expression and emotion, which can effectively improve Push accuracy of audio data.
  • the method before acquiring the expression mapping data table, further includes: acquiring multiple video data; acquiring a trained video data classification model, classifying the multiple video data through the video data classification model, and according to the classification result
  • the corresponding mood tags are added to each video data; the corresponding expression categories are obtained according to the mood tags, and the mood tags correspond to one or more expression categories; the expression mapping data table is established according to the mood tags and the expression categories and the corresponding audio data.
  • the server may also establish the expression data mapping table in advance. Specifically, the server may obtain multiple video data in advance, and each video data may include a corresponding video data identifier. The server further obtains a preset video data classification model, classifies multiple video data through the video data classification model, and adds corresponding mood tags to the multiple video data according to the classification result. Mood tags can correspond to one or more expression categories. The server obtains the corresponding expression category according to the mood tag, and establishes an expression mapping data table according to the mood tag and the expression category and the corresponding audio data identification, thereby effectively establishing the association relationship between the user's expression and the audio data.
  • the server After the server creates the expression mapping data table, the server obtains the multi-frame face pictures uploaded by the user terminal, and then recognizes the face image according to the multi-frame face pictures, and extracts the face features corresponding to the face images.
  • the server obtains a preset expression classifier, classifies the multi-frame facial features through the expression classifier, and obtains the corresponding expression category, and then further obtains a preset expression mapping data table.
  • the expression mapping data table includes expression categories and corresponding mood tags, and audio data identifiers corresponding to the mood tags.
  • the server matches the mood tag corresponding to the expression category according to the expression mapping data table, and obtains the corresponding audio data according to the mood tag.
  • the audio data may include various data such as music, broadcast audio, and audiobooks.
  • the server pushes the acquired audio data to the user terminal.
  • the mood tag may correspond to a music playlist
  • the server recognizes the current emoticon category of the user
  • the audio data in the playlist corresponding to the emoticon category is pushed to the user terminal, so that the user terminal plays the pushed audio data.
  • the method further includes: acquiring historical record data corresponding to the user ID according to a preset frequency; acquiring a preset analysis model, and analyzing the historical data through the analysis model to obtain an analysis result; and according to an expression category label Match the corresponding audio data with the analysis result; obtain the audio data with the highest matching degree and push the audio data to the user terminal.
  • the server can also obtain the user's historical record data according to the preset frequency, such as the user's song listening record and on-demand record.
  • the server performs big data analysis on the user's historical record data.
  • the server can obtain a preset analysis model.
  • the analysis model may be a model based on a neural network or a model based on a decision tree.
  • the server inputs the user's historical record data into the analysis model, analyzes the historical data through the analysis model, and obtains corresponding analysis results.
  • the user terminal can capture the user's multi-frame face pictures through the camera device according to a preset frequency, and upload the captured multi-frame face pictures to the server.
  • the multi-frame face pictures carry the user identification.
  • the server After acquiring the multi-frame face pictures uploaded by the user terminal, the server further recognizes the facial image according to the multi-frame face pictures, and extracts facial features corresponding to the facial image.
  • the server After the server recognizes and extracts the multi-frame facial features of the user, it further obtains a preset facial expression classifier, classifies the facial features through the facial expression classifier, and obtains the facial expression category corresponding to the user identifier.
  • a preset facial expression classifier By using an expression classifier based on a neural network model to recognize and classify the user's multi-frame facial features, the user's current expression category can be accurately and effectively identified.
  • the server After the server recognizes the expression type corresponding to the current facial feature of the user, it obtains the analysis result of the historical data corresponding to the user identification.
  • the server matches the corresponding audio data according to the expression category label and the analysis result. Specifically, the server obtains a preset expression mapping data table, matches corresponding audio data according to the expression type, and obtains audio data matching the analysis result according to the analysis result.
  • the server then pushes the audio data to the user terminal. For example, the server may analyze the user's preference for audio data based on the user's frequent listening records and favorite records, and push the user's preference audio data corresponding to the tag category according to the user's current expression like the user terminal.
  • the multi-frame facial features corresponding to the current facial expression of the user can be accurately and effectively extracted. Recognizing and classifying the user's expression through the expression classifier based on the neural network can accurately and effectively identify the user's current expression type. And by analyzing the user's historical data, the user can push audio data according to the user's expression and preferences, which can effectively push the user's favorite audio data according to the user's expression, thereby effectively improving the accuracy of the audio data push rate.
  • the method further includes: when the expression category label is fatigue, acquiring corresponding prompt information and audio data; sending the prompt information and audio data to the user terminal, so that the user terminal performs in a preset manner Prompt and play audio data.
  • the user terminal can capture the user's multi-frame face pictures through the camera device according to a preset frequency, and upload the captured multi-frame face pictures to the server.
  • the multi-frame face pictures carry the user identification.
  • the server After acquiring the multi-frame face pictures uploaded by the user terminal, the server further recognizes the facial image according to the multi-frame face pictures, and extracts facial features corresponding to the facial image.
  • the server After the server recognizes and extracts the multi-frame facial features of the user, it further obtains a preset facial expression classifier, classifies the facial features through the facial expression classifier, and obtains the facial expression category corresponding to the user identifier.
  • a preset facial expression classifier By using an expression classifier based on a neural network model to recognize and classify the user's multi-frame facial features, the user's current expression category can be accurately and effectively identified.
  • the server After the server recognizes the expression type corresponding to the current facial feature of the user, when the server recognizes that the user's current expression type is fatigue, the corresponding prompt information and audio data are obtained, and the prompt information and audio data are sent to the user terminal To enable the user terminal to prompt and play audio data in a preset manner.
  • the server When the server recognizes the user's current fatigue through the expression, the server will strengthen the reminder according to the preset method, such as high-decibel volume audio and Prompt voice to remind, which can effectively remind the user to remind the user to drive safely.
  • steps in the flowcharts of FIGS. 2-4 are displayed in order according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated in this article, the execution of these steps is not strictly limited in order, and these steps may be executed in other orders. Moreover, at least some of the steps in FIGS. 2-4 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. These sub-steps or stages The execution order of is not necessarily sequential, but may be executed in turn or alternately with at least a part of other steps or sub-steps or stages of other steps.
  • an audio data pushing device including: a data acquiring module 502, a face recognition module 504, an expression classification module 506, a data matching module 508, and a data pushing module 510, among them:
  • the data obtaining module 502 is used to obtain multi-frame face pictures uploaded by the user terminal, and the multi-frame face pictures carry the user identification;
  • the face recognition module 504 is used to recognize face images based on multi-frame face pictures, and extract face features corresponding to the face images;
  • the facial expression classification module 506 is used to obtain a trained facial expression classifier, and classify facial features through the facial expression classifier to obtain the facial expression category corresponding to the user identification;
  • the data matching module 508 is used to obtain an expression mapping data table, and obtain corresponding audio data according to expression categories;
  • the data pushing module 510 is used to push audio data to the user terminal.
  • the face recognition module 504 is also used to locate key points of multi-frame face pictures to obtain multi-frame face pictures after positioning; identify the multi-frame positions after positioning according to a preset face recognition algorithm
  • the multi-frame face image in the face picture obtains the facial features corresponding to the multi-frame face image.
  • the device further includes an expression classifier training module for acquiring multiple expression data from a preset database; using the acquired multiple expression data to generate a training set and a verification set; the training set includes the labeled Expression data, the verification set includes unmarked expression data; use the marked expression in the training set to train through a preset algorithm to obtain a preliminary expression classifier; input the unmarked expression data in the verification set into the preliminary expression classifier for verification Training; until the amount of verification set data that reaches the preset probability value reaches the preset ratio, the training is stopped, and the trained expression classifier is obtained.
  • an expression classifier training module for acquiring multiple expression data from a preset database; using the acquired multiple expression data to generate a training set and a verification set; the training set includes the labeled Expression data, the verification set includes unmarked expression data; use the marked expression in the training set to train through a preset algorithm to obtain a preliminary expression classifier; input the unmarked expression data in the verification set into the preliminary expression classifier for verification Training; until the amount of verification set data that
  • the multi-frame face picture includes a corresponding image sequence
  • the expression classification module 506 is further used to input the multi-frame face features corresponding to the multi-frame face picture into the expression classifier.
  • the convolutional neural network recognizes each face feature vector; calculates the corresponding dynamic expression features based on the image sequence and multi-frame face feature vectors; calculates the probability value of the dynamic expression feature belonging to each expression category; obtains the expression category with the highest probability value, Get the emoticon category corresponding to the user logo.
  • the device further includes an expression mapping data table creation module for acquiring multiple video data; acquiring a trained video data classification model, and classifying the multiple video data through the video data classification model, according to the classification As a result, corresponding mood tags are added to multiple video data; corresponding expression categories are obtained according to the mood tags, and the mood tags correspond to one or more expression categories; and an expression mapping data table is established based on the mood tags and expression categories and corresponding audio data.
  • the device further includes a data analysis module for acquiring historical record data corresponding to the user ID according to a preset frequency; acquiring a preset analysis model, and analyzing historical data through the analysis model to obtain an analysis result;
  • the data pushing module 510 is also used to match the corresponding audio data according to the expression category label and the analysis result; obtain the matching audio data, and push the audio data to the user terminal.
  • the device further includes a prompting module for acquiring corresponding prompting information and audio data when the expression category label is fatigue; sending the prompting information and audio data to the user terminal, so that the user terminal follows the Set the way to prompt and play audio data.
  • Each module in the above audio data pushing device may be implemented in whole or in part by software, hardware, or a combination thereof.
  • the above modules may be embedded in the hardware form or independent of the processor in the computer device, or may be stored in the memory in the computer device in the form of software so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided.
  • the computer device may be a server, and its internal structure may be as shown in FIG. 6.
  • the computer device includes a processor, memory, network interface, and database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, computer-readable instructions, and a database.
  • the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium.
  • the database of the computer device is used to store data such as face pictures, expression categories and audio data.
  • the network interface of the computer device is used to communicate with external terminals through a network connection.
  • the computer-readable instructions are executed by the processor to implement an audio data pushing method.
  • FIG. 6 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied.
  • the specific computer equipment may It includes more or fewer components than shown in the figure, or some components are combined, or have a different component arrangement.
  • a computer device includes a memory and one or more processors.
  • the memory stores computer-readable instructions.
  • the one or more processors perform the following steps:
  • the processor also implements the following steps when executing computer-readable instructions: positioning key points on multiple frames of face pictures to obtain positioned multiple frames of face pictures; and identifying positioning according to a preset face recognition algorithm
  • the multi-frame face images in the subsequent multi-frame face pictures obtain the facial features corresponding to the multi-frame face images.
  • the processor also implements the following steps when executing computer-readable instructions: acquiring multiple expression data from a preset database; using the acquired multiple expression data to generate a training set and a verification set; the training set includes the marked Expression data, the verification set includes unmarked expression data; use the marked expression data in the training set to train through a preset algorithm to obtain a preliminary expression classifier; input the unmarked expression data in the verification set into the preliminary expression classifier for Verification training; and until the amount of verification set data that reaches a preset probability value reaches a preset ratio, the training is stopped, and an expression classifier that has been trained is obtained.
  • the multi-frame face pictures include corresponding image sequences.
  • the processor executes the computer-readable instructions, the following steps are also implemented: the multi-frame face features corresponding to the multi-frame face pictures are input to the expression classifier by The convolutional neural network in the expression classifier identifies each frame of face feature vectors; calculates the corresponding dynamic expression features according to the image sequence and multi-frame face feature vectors; calculates the probability value of the dynamic expression features belonging to each expression category; and obtains the probability The emoticon category with the highest value obtains the emoticon category corresponding to the user ID.
  • the processor also implements the following steps when executing computer-readable instructions: acquiring multiple video data; acquiring a trained video data classification model, classifying multiple video data through the video data classification model, and classifying the multiple video data according to the classification result Add corresponding mood tags to multiple video data; obtain corresponding expression categories according to the mood tags, and the mood tags correspond to one or more expression categories; and establish an expression mapping data table based on the mood tags and expression categories and corresponding audio data.
  • the processor also implements the following steps when executing computer-readable instructions: acquiring historical record data corresponding to the user ID according to a preset frequency; acquiring a preset analysis model, and analyzing historical data through the analysis model to obtain an analysis Results; matching the corresponding audio data according to the expression category label and the analysis result; and obtaining the matching audio data, and pushing the audio data to the user terminal.
  • the processor also implements the following steps when executing the computer-readable instructions: when the expression category label is fatigue, acquiring corresponding prompt information and audio data; and sending the prompt information and audio data to the user terminal, so that The user terminal prompts and plays audio data according to a preset method.
  • One or more non-volatile computer-readable storage media storing computer-readable instructions.
  • the one or more processors When the computer-readable instructions are executed by one or more processors, the one or more processors perform the following steps: obtain user terminal uploads Multi-frame face pictures, the multi-frame face pictures carry the user identification;
  • multi-frame face pictures are located at key points to obtain multi-frame face pictures after positioning; and recognition is performed according to a preset face recognition algorithm Multi-frame face images in the multi-frame face pictures after positioning, to obtain the facial features corresponding to the multi-frame face images.
  • the following steps are further achieved: acquiring multiple expression data from a preset database; using the acquired multiple expression data to generate a training set and a verification set; the training set includes marked Expression data, the verification set includes unmarked expression data; use the marked expression data in the training set to train through a preset algorithm to obtain a preliminary expression classifier; input the unmarked expression data in the verification set into the preliminary expression classifier Perform verification training; and until the amount of verification set data that reaches a preset probability value reaches a preset ratio, then stop training and obtain an expression classifier that has been trained.
  • the multi-frame face pictures include corresponding image sequences.
  • the following steps are further implemented: multi-frame face features corresponding to the multi-frame face pictures are input to the expression classifier, Identify each frame of face feature vectors through the convolutional neural network in the expression classifier; calculate the corresponding dynamic expression features according to the image sequence and multi-frame face feature vectors; calculate the probability value of the dynamic expression features belonging to each expression category; and obtain The expression category with the highest probability value is obtained as the expression category corresponding to the user ID.
  • the following steps are also achieved: acquiring multiple video data; acquiring a trained video data classification model, and classifying the multiple video data through the video data classification model, according to the classification As a result, corresponding mood tags are added to multiple video data; corresponding expression categories are obtained according to the mood tags, and the mood tags correspond to one or more expression categories; and an expression mapping data table is established according to the mood tags and expression categories and corresponding audio data.
  • the following steps are further implemented: acquiring historical record data corresponding to the user ID according to a preset frequency; acquiring a preset analysis model, and analyzing the historical data through the analysis model to obtain Analysis results; matching corresponding audio data according to expression category tags and analysis results; and obtaining matching audio data, and pushing the audio data to the user terminal.
  • the following steps are also implemented: when the expression category tag is fatigue, acquiring corresponding prompt information and audio data; and sending the prompt information and audio data to the user terminal to The user terminal prompts and plays audio data in a preset manner.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM random access memory
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous chain (Synchlink) DRAM
  • RDRAM direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Image Analysis (AREA)

Abstract

一种音频数据推送方法, 包括: 获取用户终端上传的多帧人脸图片, 多帧人脸图片携带了用户标识; 根据多帧人脸图片识别人脸图像, 提取人脸图像对应的人脸特征; 获取已训练的表情分类器, 通过表情分类器对人脸特征进行分类, 得到用户标识对应的表情类别; 获取表情映射数据表, 根据表情类别获取对应的音频数据; 及将音频数据推送至用户终端.

Description

音频数据推送方法、装置、计算机设备和存储介质
相关申请的交叉引用：
本申请要求于2018年12月18日提交至中国专利局，申请号为2018115466286，申请名称为“音频数据推送方法、装置、计算机设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及一种音频数据推送方法、装置、计算机设备和存储介质。
背景技术
驾驶员在驾驶的过程中，行为受到一定限制。传统的车载播放器中的音乐更新较为繁琐，通常是随机为用户进行播放已下载的音乐或其他有声读物，当用户不喜欢当前的音乐时，需要手动更换当前的音乐，给用户带了不便。随着人工智能技术的迅速发展，出现了一些通过识别用户的语音指令执行相应的操作，但在驾驶过程中存在较大的噪音污染，语音指令识别的准确率较低，用户更换操作较为麻烦。
然而，发明人意识到，现有的一些音频数据自动推送的方式中，通常是根据排行或用户的历史记录相关的音频数据进行推送，而自动推送的音频播放数据不一定符合用户的喜好和情绪，导致了音乐或有声读物等音频播放数据的推送准确率较低。
发明内容
根据本申请公开的各种实施例，提供一种音频数据推送方法、装置、计算机设备和存储介质。
一种音频数据推送方法，所述方法包括：
获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
将所述音频数据推送至所述用户终端。
一种音频数据推送装置，所述装置包括：
数据获取模块，用于获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
人脸识别模块，用于根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
表情分类模块，用于获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
数据匹配模块，用于获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
数据推送模块，用于将所述音频数据推送至所述用户终端。
一种计算机设备，包括存储器和一个或多个处理器，所述存储器中储存有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述一个或多个处理器执行以下步骤：
获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
将所述音频数据推送至所述用户终端。
一个或多个存储有计算机可读指令的非易失性计算机可读存储介质，计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行以下步骤：
获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
将所述音频数据推送至所述用户终端。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1为根据一个或多个实施例中音频数据推送方法的应用场景图。
图2为根据一个或多个实施例中音频数据推送方法的流程示意图。
图3为根据一个或多个实施例中训练表情分类器步骤的流程示意图。
图4为根据一个或多个实施例中通过表情分类器对人脸特征分类步骤的流程示意图。
图5为根据一个或多个实施例中音频数据推送装置的框图；
图6为根据一个或多个实施例中计算机设备的框图。
具体实施方式
为了使本申请的技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供的音频数据推送方法，可以应用于如图1所示的应用环境中。用户终端102通过网络与服务器104通过网络进行通信。用户终端102可以是安装在车辆中具有摄像功能的车载系统终端，也可以是与车辆分离的具有摄像功能的智能手机、平板电脑和便携式设备等，服务器104可以用独立的服务器或者是多个服务器组成的服务器集群来实现。服务器104获取用户终端102上传的多帧人脸图片，多帧人脸图片携带了用户标识。服务器104进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。服务器104则进一步获取预设的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。通过利用基于神经网络模型的表情分类器对用户的多帧人脸特征进行识别并分类，由此能够准确有效地识别出该用户当前的表情类别。服务器104识别出该用户当前的人脸特征对应的表情类别后，则获取预设的表情映射数据表，根据表情类别获取对应的音频数据，并将音频数据推送至用户终端102。通过对用户在驾驶过程中的人脸表情特征进行识别，并通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，如图2所示，提供了一种音频数据推送方法，以该方法应用于图1中的服务器为例进行说明，包括以下步骤：
步骤202，获取用户终端上传的多帧人脸图片，多帧人脸图片携带用户标识。
用户终端可以是安装在车辆中具有摄像功能的车载系统终端，也可以是与车辆分离的具有摄像功能的智能手机、平板电脑和便携式设备等。用户在驾驶的过程中，用户开启对应的用户终端后，可以向用户终端触发播放推荐指令。用户终端响应播放推荐指令后，通过摄像装置按照预设频率捕捉用户的人脸图片，并将捕捉的多帧人脸图片按照时间序列上传至服务器。其中，车载终端捕捉的多帧人脸图片可以是连续的多帧静态图片，也可以是动态视频。
进一步地，用户开启用户终端中相应的应用程序后，用户终端还可以根据预设的频率通过摄像装置自动拍摄用户的人脸图片或人脸视频，并将拍摄的人脸图片或人脸视频按照时间序列上传至服务器。
服务器进而获取用户终端上传的通过摄像装置拍摄的多帧人脸图片，多帧人脸图片携带了用户标识。
步骤204，根据多帧人脸图片识别人脸图像，提取人脸图像对应的人脸特征。
服务器获取用户终端通过摄像装置拍摄的多帧人脸图片后，则对多帧人脸图片进行人脸识别，识别出多帧人脸图片中的人脸图像，并对识别出的多帧人脸图像进行特征提取，提取出每帧人脸图像对应的人脸特征。多帧人脸图片还包括对应的图像序列。具体地，服务器对多帧人脸图片进行关键点定位，根据预设的人脸识别算法识别定位后的多帧人脸图片中的多帧人脸图像，并对多帧人脸图像进行特征提取，得到多帧人脸图像对应的人脸特征，由此能够有效地识别并提取出人脸图像对应的人脸特征。
步骤206，获取已训练的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。
服务器提取每帧人脸图像对应的人脸特征后，获取预设的表情分类器。表情分类器可以是预先训练得到的基于神经网络模型的分类器。具体地，服务器将提取的每帧人脸图像对应的人脸特征输入至表情分类器中，通过表情分类器中的卷积神经网络识别每帧人脸特征，并根据图像序列和多帧人脸特征计算对应的动态人脸特征，进而计算动态人脸特征属于每个表情类别的概率值。服务器则获取概率值最高的表情类别，由此得到对用户标识对应的表情类别。
步骤208，获取预设的表情映射数据表，根据表情类别获取对应的音频数据。
步骤210，将音频数据推送至用户终端。
服务器通过表情分类器对多帧人脸特征进行分类，得到对应的表情类别后，进一步获取预设的表情映射数据表。其中，表情映射数据表中包括表情类别和对应的心情标签，以及心 情标签对应的音频数据标识。具体地，服务器可以预先获取多个视频数据，每个视频数据可以包括对应的视频数据标识。服务器进一步获取预设的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，并对视频数据添加对应的心情标签。心情标签可以对应一个或多个表情类别。服务器则根据心情标签获取对应的表情类别，并根据心情标签和表情类别以及对应的音频数据标识建立表情映射数据表。
服务器则根据表情映射数据表匹配与表情类别对应的心情标签，并根据心情标签获取对应的音频数据。其中，音频数据可以包括各种音乐、广播音频以及有声读物等数据。
例如，心情标签可以对应音乐播放列表，服务器识别出该用户当前的表情类别后，则将表情类别对应的播放列表中的音频数据推送给用户终端，使得用户终端对推送的音频数据进行播放。通过对用户在驾驶过程中的人脸表情特征进行识别，并通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
上述音频数据推送方法中，服务器获取用户终端上传的多帧人脸图片，多帧人脸图片携带了用户标识。服务器进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。服务器则进一步获取预设的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。通过利用基于神经网络模型的表情分类器对用户的多帧人脸特征进行识别并分类，由此能够准确有效地识别出该用户当前的表情类别。服务器识别出该用户当前的人脸特征对应的表情类别后，则获取预设的表情映射数据表，根据表情类别获取对应的音频数据，并将音频数据推送至用户终端。通过对用户在驾驶过程中的人脸表情特征进行识别，并通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，根据多帧人脸图片识别人脸图像，提取人脸图像对应的人脸特征的步骤，包括：对多帧人脸图片进行关键点定位，得到定位后的多帧人脸图片；根据预设人脸识别算法识别定位后的多帧人脸图片中的多帧人脸图像，得到多帧人脸图像对应的人脸特征。
用户终端可以根据预设的频率通过摄像装置捕捉用户的多帧人脸图片，并将捕捉的多帧人脸图片上传至服务器，多帧人脸图片携带了用户标识。服务器获取用户终端上传的多帧人脸图片后，进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。
具体地，服务器根据预设算法检测多帧人脸图片中的关键点，并对多帧人脸图片进行关键点定位。服务器进而根据预设的人脸识别算法识别关键点定位后的多帧人脸图片中的多帧人脸图像，服务器还可以对人脸图片进行分割，提取出人脸图像部分，进一步对提取出的人脸图像进行灰度化处理和归一化处理。服务器则对多帧人脸图像进行特征提取，得到多帧人脸图像对应的人脸特征，由此能够有效地识别并提取出人脸图像对应的人脸特征。
服务器还可以通过识别每帧人脸图片中的人脸特征是否一致，来判断用户的的人脸表情开始帧和结束帧以及持续的时间，通过人脸表情开始帧和结束帧以及持续的时间提取出用户的当前的多帧人脸特征。由此能够准确有效地提取出用户当前表情对应的多帧人脸特征。
服务器识别并提取出该用户的多帧人脸特征后，进一步获取预设的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。通过利用基于神经网络模型的表情分类器对用户的多帧人脸特征进行识别并分类，由此能够准确有效地识别出该用户当前的表情类别。服务器识别出该用户当前的人脸特征对应的表情类别后，则获取预设的表情映射数据表，根据表情类别获取对应的音频数据，并将音频数据推送至用户终端。通过对用户在驾驶过程中的人脸表情特征进行识别，能够准确有效地提取出用户当前表情对应的多帧人脸特征。并通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，如图3所示，在获取预设的表情分类器之前，该方法还包括训练表情分类器的步骤，该步骤具体包括以下内容：
步骤302，从预设数据库中获取多个表情数据。
步骤304，利用获取的多个表情数据生成训练集和验证集；训练集中包括已标注的表情数据，验证集中包括未标注的表情数据。
步骤306，利用训练集中已标注的表情数据通过预设算法进行训练得到初步的表情分类器。
步骤308，将验证集中未标注的表情数据输入至初步的表情分类器中进行验证训练。
步骤310，直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
服务器在获取预设的表情分类器之前，还需要利用大量的表情数据训练得到表情分类器。具体地，服务器可以从本地或第三方数据库中获取大量的表情数据，表情数据可以包括表情图片和表情视频以及动态表情图像等。服务器进而将获取的大量的表情数据生成训练集 和验证集，其中，训练集中的表情数据可以是通过人工标注后的表情数据，验证集中的表情数据可以是未进行标注的表情数据。
服务器则将训练集中的表情数据进行特征提取，通过预设的算法进行训练，训练得到初步的表情分类器。例如，服务器可以通过CNN(Convolutional Neural Network，卷积神经网络)对表情数据中的人脸图像进行卷积操作，得到人脸图像对应的人脸特征。并将表情数据中多帧人脸图片对应的多帧人脸特征输入至BLSTM(Bidirectional Long Short-term Memory，双向长短期记忆神经网络)，通过预设函数计算出表情数据对应的动态表情特征。服务器进而根据预设算法计算出每个动态表情特征对应每个标签类别的概率值，从而对表情分类器进行训练，由此能够有效地得到初步的表情分类器。
服务器进一步将验证集中的数据输入至初始表情分类器中进行持续训练，得到每个表情数据对应每个类别的概率值，获取达到预设的概率阈值的表情数据，当达到预设的概率阈值的表情数据的数量达到预设比值时，则停止训练，则得到训练完成的表情分类器。通过利用大量的表情数据对表情分类器进行训练和验证，从而可以有效地训练出分类准确率较高的表情分类器。
在其中一个实施例中，如图4所示，多帧人脸图片包括对应的图像序列，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别的步骤，具体包括一下内容：
步骤402，将多帧人脸图片对应的多帧人脸特征输入至表情分类器，通过表情分类器中的卷积神经网络识别每帧人脸特征向量。
步骤404，根据图像序列和多帧人脸特征向量计算对应的动态表情特征。
步骤406，计算动态表情特征属于每个表情类别的概率值。
步骤408，获取概率值最高的表情类别，得到用户标识对应的表情类别。
用户终端可以根据预设的频率通过摄像装置捕捉用户的多帧人脸图片，并将捕捉的多帧人脸图片上传至服务器，多帧人脸图片携带了用户标识。服务器获取用户终端上传的多帧人脸图片后，进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。
服务器提取每帧人脸图像对应的人脸特征后，获取预设的表情分类器。其中，表情分类器可以是预先训练得到的基于神经网络模型的分类器。具体地，服务器将提取的每帧人脸图像对应的人脸特征输入至表情分类器中，通过表情分类器中的卷积神经网络识别每帧人脸特征向量，并根据图像序列和多帧人脸特征向量计算对应的动态人脸特征，进而计算动态人脸特征属于每个表情类别的概率值。服务器则获取概率值最高的表情类别，由此得到对用户标识对应的表情类别。
例如，服务器可以通过CNN(Convolutional Neural Network，卷积神经网络)对多帧人脸图像对应的多帧人脸特征进行卷积操作，识别得到人脸图像对应的人脸特征向量。并将多帧人脸特征向量输入至BLSTM(Bidirectional Long Short-term Memory，双向长短期记忆神经网络)，通过预设函数计算出多帧人脸特征向量对应的动态表情特征。服务器进而根据预设算法计算出每个动态表情特征对应每个标签类别的概率值，服务器则获取概率值最高的表情类别。
服务器识别出该用户当前的人脸特征对应的表情类别后，则获取预设的表情映射数据表，根据表情类别获取对应的音频数据，并将音频数据推送至用户终端。通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，获取表情映射数据表之前，还包括：获取多个视频数据；获取已训练的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据心情标签获取对应的表情类别，心情标签对应一个或多个表情类别；根据心情标签和表情类别以及对应的音频数据建立表情映射数据表。
服务器在获取预设的表情数据映射表之前，还可以预先建立表情数据映射表。具体地，服务器可以预先获取多个视频数据，每个视频数据可以包括对应的视频数据标识。服务器进一步获取预设的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签。心情标签可以对应一个或多个表情类别。服务器则根据心情标签获取对应的表情类别，并根据心情标签和表情类别以及对应的音频数据标识建立表情映射数据表，由此能够有效地建立用户表情与音频数据之间的关联关系。
服务器建立表情映射数据表后，服务器获取用户终端上传的多帧人脸图片，进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。服务器获取预设的表情分类器，通过表情分类器对多帧人脸特征进行分类，得到对应的表情类别后，进一步获取预设的表情映射数据表。其中，表情映射数据表中包括表情类别和对应的心情标签，以及心情标签对应的音频数据标识。
服务器则根据表情映射数据表匹配与表情类别对应的心情标签，并根据心情标签获取对应的音频数据。音频数据可以包括各种音乐、广播音频以及有声读物等数据。服务器则将获取的音频数据推送至用户终端。
例如，心情标签可以对应音乐播放列表，服务器识别出该用户当前的表情类别后，则将 表情类别对应的播放列表中的音频数据推送给用户终端，使得用户终端对推送的音频数据进行播放。通过对用户在驾驶过程中的人脸表情特征进行识别，并通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情情绪，进而能够有效地根据用户的表情情绪推送相对应的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，该方法还包括：根据预设的频率获取用户标识对应的历史记录数据；获取预设的分析模型，通过分析模型对历史数据进行分析，得到分析结果；根据表情类别标签和分析结果匹配对应的音频数据；获取匹配度最高的音频数据，将音频数据推送至用户终端。
服务器还可以根据预设的频率获取用户的历史记录数据，例如用户的听歌记录和点播记录等。服务器通过对用户的历史记录数据进行大数据分析，具体地，服务器可以获取预设的分析模型，分析模型可以是基于神经网络的模型，也可以是基于决策树的模型。服务器将用户的历史记录数据输入至分析模型中，通过分析模型对历史数据进行分析，并得到对应的得到分析结果。
用户终端可以根据预设的频率通过摄像装置捕捉用户的多帧人脸图片，并将捕捉的多帧人脸图片上传至服务器，多帧人脸图片携带了用户标识。服务器获取用户终端上传的多帧人脸图片后，进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。
服务器识别并提取出该用户的多帧人脸特征后，进一步获取预设的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。通过利用基于神经网络模型的表情分类器对用户的多帧人脸特征进行识别并分类，由此能够准确有效地识别出该用户当前的表情类别。
服务器识别出该用户当前的人脸特征对应的表情类别后，则获取该用户标识对应的历史数据的分析结果。服务器则根据表情类别标签和分析结果匹配对应的音频数据。具体地，服务器获取预设的表情映射数据表，根据表情类别匹配对应的音频数据，并根据分析结果获取与分析结果相匹配的音频数据。服务器进而将音频数据推送至用户终端。例如，服务器可以根据用户的常听记录和收藏记录等数据分析出用户对音频数据的偏好度，并根据用户当前的表情像用户终端推送该标签类别对应的用户偏好度较高的音频数据。通过对用户在驾驶过程中的人脸表情特征进行识别，能够准确有效地提取出用户当前表情对应的多帧人脸特征。通过基于神经网络的表情分类器对用户的表情进行识别分类，能够准确有效地识别出用户当前的表情类别。并通过对用户的历史数据进行分析，根据用户的表情和偏好对用户进行推送音 频数据，进而能够有效地根据用户的表情推送相用户喜好的音频数据，由此能够有效地提高音频数据的推送准确率。
在其中一个实施例中，该方法还包括：当表情类别标签为疲劳时，获取对应的提示信息和音频数据；将提示信息和音频数据发送至用户终端，以使用户终端按照预设的方式进行提示和播放音频数据。
用户终端可以根据预设的频率通过摄像装置捕捉用户的多帧人脸图片，并将捕捉的多帧人脸图片上传至服务器，多帧人脸图片携带了用户标识。服务器获取用户终端上传的多帧人脸图片后，进而根据多帧人脸图片识别人脸图像，并提取人脸图像对应的人脸特征。
服务器识别并提取出该用户的多帧人脸特征后，进一步获取预设的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别。通过利用基于神经网络模型的表情分类器对用户的多帧人脸特征进行识别并分类，由此能够准确有效地识别出该用户当前的表情类别。
服务器识别出该用户当前的人脸特征对应的表情类别后，当服务器识别出用户当前的表情类别为疲劳时，获取相对应的提示信息和音频数据，并将提示信息和音频数据发送至用户终端，以使用户终端按照预设的方式进行提示和播放音频数据。用户在驾驶过程中，若用户出现疲劳驾驶时，此时存在较高的风险隐患，服务器通过表情识别出用户的当前存在疲劳现象时，按照预设方式进行加强提醒，如高分贝音量的音频和提示语音进行提醒，由此能够有效地对用户进行提醒，以提示用户安全驾驶。
应该理解的是，虽然图2-4的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2-4中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
在其中一个实施例中，如图5所示，提供了一种音频数据推送装置，包括：数据获取模块502、人脸识别模块504、表情分类模块506、数据匹配模块508和数据推送模块510，其中：
数据获取模块502，用于获取用户终端上传的多帧人脸图片，多帧人脸图片携带了用户标识；
人脸识别模块504，用于根据多帧人脸图片识别人脸图像，提取人脸图像对应的人脸特 征；
表情分类模块506，用于获取已训练的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别；
数据匹配模块508，用于获取表情映射数据表，根据表情类别获取对应的音频数据；
数据推送模块510，用于将音频数据推送至用户终端。
在其中一个实施例中，人脸识别模块504还用于对多帧人脸图片进行关键点定位，得到定位后的多帧人脸图片；根据预设的人脸识别算法识别定位后的多帧人脸图片中的多帧人脸图像，得到多帧人脸图像对应的人脸特征。
在其中一个实施例中，该装置还包括表情分类器训练模块，用于从预设数据库中获取多个表情数据；利用获取的多个表情数据生成训练集和验证集；训练集中包括已标注的表情数据，验证集中包括未标注的表情数据；利用训练集中已标注的表情通过预设算法进行训练得到初步的表情分类器；将验证集中未标注的表情数据输入至初步的表情分类器中进行验证训练；直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
在其中一个实施例中，多帧人脸图片包括对应的图像序列，表情分类模块506还用于将多帧人脸图片对应的多帧人脸特征输入至表情分类器，通过表情分类器中的卷积神经网络识别每帧人脸特征向量；根据图像序列和多帧人脸特征向量计算对应的动态表情特征；计算动态表情特征属于每个表情类别的概率值；获取概率值最高的表情类别，得到用户标识对应的表情类别。
在其中一个实施例中，该装置还包括表情映射数据表建立模块，用于获取多个视频数据；获取已训练的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据心情标签获取对应的表情类别，心情标签对应一个或多个表情类别；根据心情标签和表情类别以及对应的音频数据建立表情映射数据表。
在其中一个实施例中，该装置还包括数据分析模块，用于根据预设频率获取用户标识对应的历史记录数据；获取预设的分析模型，通过分析模型对历史数据进行分析，得到分析结果；数据推送模块510还用于根据表情类别标签和分析结果匹配对应的音频数据；获取相匹配的音频数据，将音频数据推送至用户终端。
在其中一个实施例中，该装置还包括提示模块，用于当表情类别标签为疲劳时，获取对应的提示信息和音频数据；将提示信息和音频数据发送至用户终端，以使用户终端按照预设的方 式进行提示和播放音频数据。
关于音频数据推送装置的具体限定可以参见上文中对于音频数据推送方法的限定，在此不再赘述。上述音频数据推送装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图6所示。该计算机设备包括通过系统总线连接的处理器、存储器、网络接口和数据库。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机可读指令和数据库。该内存储器为非易失性存储介质中的操作系统和计算机可读指令的运行提供环境。该计算机设备的数据库用于存储人脸图片、表情类别和音频数据等数据。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机可读指令被处理器执行时以实现一种音频数据推送方法。
本领域技术人员可以理解，图6中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
一种计算机设备，包括存储器和一个或多个处理器，存储器中储存有计算机可读指令，计算机可读指令被处理器执行时，使得一个或多个处理器执行以下步骤：
获取用户终端上传的多帧人脸图片，多帧人脸图片携带了用户标识；
根据多帧人脸图片识别人脸图像，提取人脸图像对应的人脸特征；
获取已训练的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别；
获取表情映射数据表，根据表情类别获取对应的音频数据；及
将音频数据推送至用户终端。
在一个实施例中，处理器执行计算机可读指令时还实现以下步骤：对多帧人脸图片进行关键点定位，得到定位后的多帧人脸图片；及根据预设人脸识别算法识别定位后的多帧人脸图片中的多帧人脸图像，得到多帧人脸图像对应的人脸特征。
在一个实施例中，处理器执行计算机可读指令时还实现以下步骤：从预设数据库中获取多个表情数据；利用获取的多个表情数据生成训练集和验证集；训练集中包括已标注的表情数据，验证集中包括未标注的表情数据；利用训练集中已标注的表情数据通过预设算法进行 训练得到初步的表情分类器；将验证集中未标注的表情数据输入至初步的表情分类器中进行验证训练；及直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
在一个实施例中，多帧人脸图片包括对应的图像序列，处理器执行计算机可读指令时还实现以下步骤：将多帧人脸图片对应的多帧人脸特征输入至表情分类器，通过表情分类器中的卷积神经网络识别每帧人脸特征向量；根据图像序列和多帧人脸特征向量计算对应的动态表情特征；计算动态表情特征属于每个表情类别的概率值；及获取概率值最高的表情类别，得到用户标识对应的表情类别。
在一个实施例中，处理器执行计算机可读指令时还实现以下步骤：获取多个视频数据；获取已训练的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据心情标签获取对应的表情类别，心情标签对应一个或多个表情类别；及根据心情标签和表情类别以及对应的音频数据建立表情映射数据表。
在一个实施例中，处理器执行计算机可读指令时还实现以下步骤：根据预设频率获取用户标识对应的历史记录数据；获取预设的分析模型，通过分析模型对历史数据进行分析，得到分析结果；根据表情类别标签和分析结果匹配对应的音频数据；及获取相匹配的音频数据，将音频数据推送至用户终端。
在一个实施例中，处理器执行计算机可读指令时还实现以下步骤：当表情类别标签为疲劳时，获取对应的提示信息和音频数据；及将提示信息和音频数据发送至用户终端，以使用户终端按照预设的方式进行提示和播放音频数据。
一个或多个存储有计算机可读指令的非易失性计算机可读存储介质，计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行以下步骤：获取用户终端上传的多帧人脸图片，多帧人脸图片携带了用户标识；
根据多帧人脸图片识别人脸图像，提取人脸图像对应的人脸特征；
获取已训练的表情分类器，通过表情分类器对人脸特征进行分类，得到用户标识对应的表情类别；
获取表情映射数据表，根据表情类别获取对应的音频数据；
将音频数据推送至用户终端。
在一个实施例中，计算机可读指令被处理器执行时还实现以下步骤：对多帧人脸图片进行关键点定位，得到定位后的多帧人脸图片；及根据预设人脸识别算法识别定位后的多帧人 脸图片中的多帧人脸图像，得到多帧人脸图像对应的人脸特征。
在一个实施例中，计算机可读指令被处理器执行时还实现以下步骤：从预设数据库中获取多个表情数据；利用获取的多个表情数据生成训练集和验证集；训练集中包括已标注的表情数据，验证集中包括未标注的表情数据；利用训练集中已标注的表情数据通过预设算法进行训练得到初步的表情分类器；将验证集中未标注的表情数据输入至初步的表情分类器中进行验证训练；及直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
在一个实施例中，多帧人脸图片包括对应的图像序列，计算机可读指令被处理器执行时还实现以下步骤：将多帧人脸图片对应的多帧人脸特征输入至表情分类器，通过表情分类器中的卷积神经网络识别每帧人脸特征向量；根据图像序列和多帧人脸特征向量计算对应的动态表情特征；计算动态表情特征属于每个表情类别的概率值；及获取概率值最高的表情类别，得到用户标识对应的表情类别。
在一个实施例中，计算机可读指令被处理器执行时还实现以下步骤：获取多个视频数据；获取已训练的视频数据分类模型，通过视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据心情标签获取对应的表情类别，心情标签对应一个或多个表情类别；及根据心情标签和表情类别以及对应的音频数据建立表情映射数据表。
在一个实施例中，计算机可读指令被处理器执行时还实现以下步骤：根据预设频率获取用户标识对应的历史记录数据；获取预设的分析模型，通过分析模型对历史数据进行分析，得到分析结果；根据表情类别标签和分析结果匹配对应的音频数据；及获取相匹配的音频数据，将音频数据推送至用户终端。
在一个实施例中，计算机可读指令被处理器执行时还实现以下步骤：当表情类别标签为疲劳时，获取对应的提示信息和音频数据；及将提示信息和音频数据发送至用户终端，以使用户终端按照预设的方式进行提示和播放音频数据。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机可读指令来指令相关的硬件来完成，所述的计算机可读指令可存储于一非易失性计算机可读取存储介质中，该计算机可读指令在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。 易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (20)

  1. 一种音频数据推送方法，所述方法包括：
    获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
    根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
    获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
    获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
    将所述音频数据推送至所述用户终端。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征，包括：
    对所述多帧人脸图片进行关键点定位，得到定位后的多帧人脸图片；及
    根据预设人脸识别算法识别定位后的多帧人脸图片中的多帧人脸图像，得到多帧人脸图像对应的人脸特征。
  3. 根据权利要求1所述的方法，其特征在于，所述获取预设的表情分类器之前，所述方法还包括：
    从预设数据库中获取多个表情数据；
    利用获取的多个表情数据生成训练集和验证集；所述训练集中包括已标注的表情数据，所述验证集中包括未标注的表情数据；
    利用所述训练集中已标注的表情数据通过预设算法进行训练得到初步的表情分类器；
    将所述验证集中未标注的表情数据输入至所述初步的表情分类器中进行验证训练；及
    直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
  4. 根据权利要求1至3任意一项所述的方法，其特征在于，所述多帧人脸图片包括对应的图像序列，所述通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别，包括：
    将多帧人脸图片对应的多帧人脸特征输入至所述表情分类器，通过所述表情分类器中的卷积神经网络识别每帧人脸特征向量；
    根据所述图像序列和多帧人脸特征向量计算对应的动态表情特征；
    计算所述动态表情特征属于每个表情类别的概率值；及
    获取所述概率值最高的表情类别，得到所述用户标识对应的表情类别。
  5. 根据权利要求1所述的方法，其特征在于，所述获取表情映射数据表之前，还包括：
    获取多个视频数据；
    获取已训练的视频数据分类模型，通过所述视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；
    根据所述心情标签获取对应的表情类别，所述心情标签对应一个或多个表情类别；及
    根据所述心情标签和所述表情类别以及对应的音频数据建立表情映射数据表。
  6. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    根据预设频率获取所述用户标识对应的历史记录数据；
    获取预设的分析模型，通过所述分析模型对所述历史数据进行分析，得到分析结果；
    根据所述表情类别标签和所述分析结果匹配对应的音频数据；及
    获取相匹配的音频数据，将所述音频数据推送至所述用户终端。
  7. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    当所述表情类别标签为疲劳时，获取对应的提示信息和音频数据；及
    将所述提示信息和音频数据发送至所述用户终端，以使所述用户终端按照预设的方式进行提示和播放所述音频数据。
  8. 一种音频数据推送装置，所述装置包括：
    数据获取模块，用于获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
    人脸识别模块，用于根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
    表情分类模块，用于获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
    数据匹配模块，用于获取预设的表情映射数据表，根据所述表情类别获取对应的音频数据；及
    数据推送模块，用于将所述音频数据推送至所述用户终端。
  9. 根据权利要求8所述的装置，其特征在于，所述表情分类模块还用于将多帧人脸图片对应的多帧人脸特征输入至所述表情分类器，通过所述表情分类器中的卷积神经网络识别每帧人脸特征向量；根据所述图像序列和多帧人脸特征向量计算对应的动态表情特征；计算所述动态表情特征属于每个表情类别的概率值；及获取所述概率值最高的表情类别，得到所述用户标识对应的表情类别。
  10. 根据权利要求8所述的装置，其特征在于，所述数据匹配模块还用于根据预设频率获取所述用户标识对应的历史记录数据；获取预设的分析模型，通过所述分析模型对所述历史数据进行分析，得到分析结果；根据所述表情类别标签和所述分析结果匹配对应的音频数据；及获取相匹配的音频数据，将所述音频数据推送至所述用户终端。
  11. 一种计算机设备，包括存储器及一个或多个处理器，所述存储器中储存有计算机可读指令，所述计算机可读指令被所述一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：
    获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
    根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
    获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
    获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
    将所述音频数据推送至所述用户终端。
  12. 根据权利要求11所述的计算机设备，其特征在于，所述处理器执行所述计算机可读指令时还执行以下步骤：从预设数据库中获取多个表情数据；利用获取的多个表情数据生成训练集和验证集；所述训练集中包括已标注的表情数据，所述验证集中包括未标注的表情数据；利用所述训练集中已标注的表情数据通过预设算法进行训练得到初步的表情分类器；将所述验证集中未标注的表情数据输入至所述初步的表情分类器中进行验证训练；及直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
  13. 根据权利要求11所述的计算机设备，其特征在于，所述处理器执行所述计算机可读指令时还执行以下步骤：将多帧人脸图片对应的多帧人脸特征输入至所述表情分类器，通过所述表情分类器中的卷积神经网络识别每帧人脸特征向量；根据所述图像序列和多帧人脸特征向量计算对应的动态表情特征；计算所述动态表情特征属于每个表情类别的概率值；及获取所述概率值最高的表情类别，得到所述用户标识对应的表情类别。
  14. 根据权利要求11所述的计算机设备，其特征在于，所述处理器执行所述计算机可读指令时还执行以下步骤：获取多个视频数据；获取已训练的视频数据分类模型，通过所述视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据所述心情标签获取对应的表情类别，所述心情标签对应一个或多个表情类别；及根据所述心情标签和所述表情类别以及对应的音频数据建立表情映射数据表。
  15. 根据权利要求11所述的计算机设备，其特征在于，所述处理器执行所述计算机可读 指令时还执行以下步骤：根据预设频率获取所述用户标识对应的历史记录数据；获取预设的分析模型，通过所述分析模型对所述历史数据进行分析，得到分析结果；根据所述表情类别标签和所述分析结果匹配对应的音频数据；及获取相匹配的音频数据，将所述音频数据推送至所述用户终端。
  16. 一个或多个存储有计算机可读指令的非易失性计算机可读存储介质，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：
    获取用户终端上传的多帧人脸图片，所述多帧人脸图片携带了用户标识；
    根据所述多帧人脸图片识别人脸图像，提取所述人脸图像对应的人脸特征；
    获取已训练的表情分类器，通过所述表情分类器对所述人脸特征进行分类，得到所述用户标识对应的表情类别；
    获取表情映射数据表，根据所述表情类别获取对应的音频数据；及
    将所述音频数据推送至所述用户终端。
  17. 根据权利要求16所述的存储介质，其特征在于，所述计算机可读指令被所述处理器执行时还执行以下步骤：从预设数据库中获取多个表情数据；利用获取的多个表情数据生成训练集和验证集；所述训练集中包括已标注的表情数据，所述验证集中包括未标注的表情数据；利用所述训练集中已标注的表情数据通过预设算法进行训练得到初步的表情分类器；将所述验证集中未标注的表情数据输入至所述初步的表情分类器中进行验证训练；及直到达到预设概率值的验证集数据的数量达到预设比值时，则停止训练，得到训练完成的表情分类器。
  18. 根据权利要求16所述的存储介质，其特征在于，所述计算机可读指令被所述处理器执行时还执行以下步骤：将多帧人脸图片对应的多帧人脸特征输入至所述表情分类器，通过所述表情分类器中的卷积神经网络识别每帧人脸特征向量；根据所述图像序列和多帧人脸特征向量计算对应的动态表情特征；计算所述动态表情特征属于每个表情类别的概率值；及获取所述概率值最高的表情类别，得到所述用户标识对应的表情类别。
  19. 根据权利要求16所述的存储介质，其特征在于，所述计算机可读指令被所述处理器执行时还执行以下步骤：获取多个视频数据；获取已训练的视频数据分类模型，通过所述视频数据分类模型对多个视频数据进行分类，根据分类结果对多个视频数据添加对应的心情标签；根据所述心情标签获取对应的表情类别，所述心情标签对应一个或多个表情类别；及根据所述心情标签和所述表情类别以及对应的音频数据建立表情映射数据表。
  20. 根据权利要求16所述的存储介质，其特征在于，所述计算机可读指令被所述处理器执行时还执行以下步骤：根据预设频率获取所述用户标识对应的历史记录数据；获取预设的 分析模型，通过所述分析模型对所述历史数据进行分析，得到分析结果；根据所述表情类别标签和所述分析结果匹配对应的音频数据；及获取相匹配的音频数据，将所述音频数据推送至所述用户终端。
PCT/CN2019/122546 2018-12-18 2019-12-03 音频数据推送方法、装置、计算机设备和存储介质 Ceased WO2020125397A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811546628.6 2018-12-18
CN201811546628.6A CN109766765A (zh) 2018-12-18 2018-12-18 音频数据推送方法、装置、计算机设备和存储介质

Publications (1)

Publication Number Publication Date
WO2020125397A1 true WO2020125397A1 (zh) 2020-06-25

Family

ID=66451481

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/122546 Ceased WO2020125397A1 (zh) 2018-12-18 2019-12-03 音频数据推送方法、装置、计算机设备和存储介质

Country Status (2)

Country Link
CN (1) CN109766765A (zh)
WO (1) WO2020125397A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111881776A (zh) * 2020-07-07 2020-11-03 腾讯科技（深圳）有限公司 动态表情获取方法、装置、存储介质和电子设备
CN112330579A (zh) * 2020-10-30 2021-02-05 中国平安人寿保险股份有限公司 视频背景更换方法、装置、计算机设备及计算机可读介质
CN112784700A (zh) * 2021-01-04 2021-05-11 北京小米松果电子有限公司 人脸图像显示的方法、装置及存储介质
CN112966568A (zh) * 2021-02-09 2021-06-15 中国工商银行股份有限公司 一种视频客服服务质量分析方法及装置
CN115510266A (zh) * 2022-08-10 2022-12-23 博泰车联网(南京)有限公司 一种车内儿童安抚方法及装置

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109766765A (zh) * 2018-12-18 2019-05-17 深圳壹账通智能科技有限公司 音频数据推送方法、装置、计算机设备和存储介质
CN110363187B (zh) * 2019-08-29 2020-12-25 上海云从汇临人工智能科技有限公司 一种人脸识别方法、装置、机器可读介质及设备
CN111259729A (zh) * 2019-12-30 2020-06-09 视联动力信息技术股份有限公司 一种表情识别方法和装置
CN111290800A (zh) * 2020-02-11 2020-06-16 深圳壹账通智能科技有限公司 一种人脸图像的识别方法及设备
CN111553191A (zh) * 2020-03-30 2020-08-18 深圳壹账通智能科技有限公司 基于人脸识别的视频分类方法、装置及存储介质
CN111666444B (zh) * 2020-06-02 2021-04-27 中国科学院计算技术研究所 基于人工智能的音频推送方法及系统、相关方法及设备
CN113780062A (zh) * 2021-07-26 2021-12-10 岚图汽车科技有限公司 一种基于情感识别的车载智能交互方法、存储介质及芯片

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426404A (zh) * 2015-10-28 2016-03-23 广东欧珀移动通信有限公司 一种音乐信息推荐方法、装置和终端
CN107862292A (zh) * 2017-11-15 2018-03-30 平安科技（深圳）有限公司 人物情绪分析方法、装置及存储介质
CN108197185A (zh) * 2017-12-26 2018-06-22 努比亚技术有限公司 一种音乐推荐方法、终端及计算机可读存储介质
CN109766765A (zh) * 2018-12-18 2019-05-17 深圳壹账通智能科技有限公司 音频数据推送方法、装置、计算机设备和存储介质

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202718A (zh) * 2014-08-05 2014-12-10 百度在线网络技术（北京）有限公司 一种向用户提供信息的方法与装置
CN105956059A (zh) * 2016-04-27 2016-09-21 乐视控股（北京）有限公司 基于情绪识别的信息推荐方法和装置
CN106384083A (zh) * 2016-08-31 2017-02-08 上海交通大学 一种自动面部表情识别并进行信息推荐的方法
CN106980811A (zh) * 2016-10-21 2017-07-25 商汤集团有限公司 人脸表情识别方法和人脸表情识别装置
CN107392109A (zh) * 2017-06-27 2017-11-24 南京邮电大学 一种基于深度神经网络的新生儿疼痛表情识别方法
CN107832746A (zh) * 2017-12-01 2018-03-23 北京小米移动软件有限公司 表情识别方法及装置
CN108304823B (zh) * 2018-02-24 2022-03-22 重庆邮电大学 一种基于双卷积cnn和长短时记忆网络的表情识别方法
CN108596039B (zh) * 2018-03-29 2020-05-05 南京邮电大学 一种基于3d卷积神经网络的双模态情感识别方法及系统
CN108960122A (zh) * 2018-06-28 2018-12-07 南京信息工程大学 一种基于时空卷积特征的表情分类方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426404A (zh) * 2015-10-28 2016-03-23 广东欧珀移动通信有限公司 一种音乐信息推荐方法、装置和终端
CN107862292A (zh) * 2017-11-15 2018-03-30 平安科技（深圳）有限公司 人物情绪分析方法、装置及存储介质
CN108197185A (zh) * 2017-12-26 2018-06-22 努比亚技术有限公司 一种音乐推荐方法、终端及计算机可读存储介质
CN109766765A (zh) * 2018-12-18 2019-05-17 深圳壹账通智能科技有限公司 音频数据推送方法、装置、计算机设备和存储介质

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111881776A (zh) * 2020-07-07 2020-11-03 腾讯科技（深圳）有限公司 动态表情获取方法、装置、存储介质和电子设备
CN111881776B (zh) * 2020-07-07 2023-07-07 腾讯科技（深圳）有限公司 动态表情获取方法、装置、存储介质和电子设备
CN112330579A (zh) * 2020-10-30 2021-02-05 中国平安人寿保险股份有限公司 视频背景更换方法、装置、计算机设备及计算机可读介质
CN112784700A (zh) * 2021-01-04 2021-05-11 北京小米松果电子有限公司 人脸图像显示的方法、装置及存储介质
CN112784700B (zh) * 2021-01-04 2024-05-03 北京小米松果电子有限公司 人脸图像显示的方法、装置及存储介质
CN112966568A (zh) * 2021-02-09 2021-06-15 中国工商银行股份有限公司 一种视频客服服务质量分析方法及装置
CN115510266A (zh) * 2022-08-10 2022-12-23 博泰车联网(南京)有限公司 一种车内儿童安抚方法及装置

Also Published As

Publication number Publication date
CN109766765A (zh) 2019-05-17

Similar Documents

Publication Publication Date Title
WO2020125397A1 (zh) 音频数据推送方法、装置、计算机设备和存储介质
KR102416558B1 (ko) 영상 데이터 처리 방법, 장치 및 판독 가능 저장 매체
WO2020244153A1 (zh) 会议语音数据处理方法、装置、计算机设备和存储介质
US10635893B2 (en) Identity authentication method, terminal device, and computer-readable storage medium
US10108709B1 (en) Systems and methods for queryable graph representations of videos
WO2020140665A1 (zh) 双录视频质量检测方法、装置、计算机设备和存储介质
CN105518708B (zh) 用于验证活体人脸的方法、设备和计算机程序产品
WO2020125386A1 (zh) 表情识别方法、装置、计算机设备和存储介质
CN108986826A (zh) 自动生成会议记录的方法、电子装置及可读存储介质
US10535371B2 (en) Speaker segmentation and clustering for video summarization
CN110516083B (zh) 相册管理方法、存储介质及电子设备
CN107633203A (zh) 面部情绪识别方法、装置及存储介质
CN109766767A (zh) 行为数据推送方法、装置、计算机设备和存储介质
WO2017113680A1 (zh) 声纹认证处理方法及装置
US9406295B2 (en) Apparatus and method for voice based user enrollment with video assistance
CN109871807B (zh) 人脸图像处理方法和装置
CN110309254A (zh) 智能机器人与人机交互方法
CN112148922A (zh) 会议记录方法、装置、数据处理设备及可读存储介质
CN110134830A (zh) 视频信息数据处理方法、装置、计算机设备和存储介质
WO2020019591A1 (zh) 用于生成信息的方法和装置
CN111276140B (zh) 语音命令识别方法、装置、系统及存储介质
CN107992937B (zh) 基于深度学习的非结构化数据判决方法和装置
CN111933187A (zh) 情感识别模型的训练方法、装置、计算机设备和存储介质
CN109766474A (zh) 审讯信息审核方法、装置、计算机设备和存储介质
CN109934091A (zh) 基于图像识别的辅助发音方法、装置、计算机设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19899733

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 30/09/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19899733

Country of ref document: EP

Kind code of ref document: A1