WO2021000833A1 - 基于数据处理的图片显示方法、装置和计算机设备 - Google Patents

基于数据处理的图片显示方法、装置和计算机设备 Download PDF

Info

Publication number
WO2021000833A1
WO2021000833A1 PCT/CN2020/098808 CN2020098808W WO2021000833A1 WO 2021000833 A1 WO2021000833 A1 WO 2021000833A1 CN 2020098808 W CN2020098808 W CN 2020098808W WO 2021000833 A1 WO2021000833 A1 WO 2021000833A1
Authority
WO
WIPO (PCT)
Prior art keywords
continuous frame
atlas
frame atlas
picture
preset
Prior art date
Application number
PCT/CN2020/098808
Other languages
English (en)
French (fr)
Inventor
张磊
宋晨
李雪冰
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021000833A1 publication Critical patent/WO2021000833A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G06V40/176—Dynamic expression

Definitions

  • This application relates to the field of artificial intelligence technology, and in particular to a picture display method, device, computer equipment and storage medium based on data processing.
  • the common one is to obtain the corresponding mood by recognizing micro-expressions. For example, first perform facial expression recognition to get the corresponding mood, and then display the mood in the interface to let the user know. People find that in the prior art, the mood is mainly expressed through text. The user’s mood is first judged, and then the mood is displayed on the user interface through text description. For example, when the user’s facial expression is recognized and judged as "happy”, The word “happy” is displayed on the screen; when the user's expression is recognized and judged as "calm”, the word “calm” is displayed on the interface, but this does not accurately and vividly express the user's real-time mood.
  • the main purpose of this application is to provide a picture display method, device, computer equipment and storage medium based on data processing, aiming to solve the problem that the existing methods of displaying mood cannot vividly express the real-time mood of the user.
  • this application proposes a method for displaying pictures based on data processing, including:
  • the facial facial expression recognition model is trained through a training set of continuous frame images marked with facial expression categories. The recognition results correspond to different expression categories;
  • a first continuous frame atlas corresponding to the facial expression category is obtained from a preset database.
  • the first continuous frame atlas includes a specified number of first pictures representing the mood of the same category, each of the first pictures Sorted in a preset order;
  • the first picture in the first continuous frame atlas is played in a loop according to the preset order starting from the first picture in the preset order.
  • This application also provides a picture display device based on data processing, including:
  • the image acquisition unit is used to acquire the face image to be recognized in the video
  • the recognition image unit is used to input the face image into a preset facial expression recognition model based on convolutional neural network for recognition to obtain an expression recognition result, and the expression recognition model is obtained by training a continuous frame image training set with annotated expression categories , Different expression recognition results correspond to different expression categories;
  • the first acquiring unit is configured to acquire a first continuous frame atlas corresponding to an expression category from a preset database according to the expression recognition result, the first continuous frame atlas including a specified number of first pictures representing the mood of the same category , Each of the first pictures has been sorted in a preset order;
  • the first playing unit is configured to play the first picture in the first continuous frame atlas according to the preset order, starting from the first picture in the preset order.
  • the present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, a method for displaying pictures based on data processing is implemented, wherein the data-based
  • the processed image display methods include:
  • the facial facial expression recognition model is trained through a training set of continuous frame images marked with facial expression categories. The recognition results correspond to different expression categories;
  • a first continuous frame atlas corresponding to the facial expression category is obtained from a preset database.
  • the first continuous frame atlas includes a specified number of first pictures representing the mood of the same category, each of the first pictures Sorted in a preset order;
  • the first picture in the first continuous frame atlas is played in a loop according to the preset order starting from the first picture in the preset order.
  • the present application also provides a computer-readable storage medium on which a computer program is stored.
  • a computer program is stored on which a computer program is stored.
  • the computer program is executed by a processor, a method for displaying a picture based on data processing is implemented, wherein the picture display based on data processing Methods include:
  • the facial facial expression recognition model is trained through a training set of continuous frame images marked with facial expression categories. The recognition results correspond to different expression categories;
  • a first continuous frame atlas corresponding to the facial expression category is obtained from a preset database.
  • the first continuous frame atlas includes a specified number of first pictures representing the mood of the same category, each of the first pictures Sorted in a preset order;
  • the first picture in the first continuous frame atlas is played in a loop according to the preset order starting from the first picture in the preset order.
  • FIG. 1 is a schematic diagram of the steps of a picture display method based on data processing in an embodiment of this application;
  • FIG. 2 is a schematic block diagram of the structure of a picture display device based on data processing in an embodiment of the application;
  • FIG. 3 is a schematic block diagram of the structure of a computer device according to an embodiment of the application.
  • the image display method based on data processing in this embodiment includes:
  • Step S1 Obtain a face image to be recognized in the video
  • Step S2 Input the face image to a preset facial expression recognition model based on convolutional neural network for recognition to obtain facial expression recognition results.
  • the facial expression recognition model is obtained by training a continuous frame image training set with annotated facial expression categories.
  • the expression recognition results correspond to different expression categories;
  • Step S3 Acquire a first continuous frame atlas corresponding to the facial expression category from a preset database according to the expression recognition result, the first continuous frame atlas including a specified number of first pictures representing the mood of the same category, each The first picture has been sorted in a preset order;
  • Step S4 Play the first picture in the first continuous frame atlas according to the preset order, starting from the first picture in the preset order.
  • the face image to be recognized in the video includes a single frame or more than two frames of face images.
  • the face image is acquired in real time through the camera, or by receiving the face image sent by an external device, or based on the user's selection from an existing image database or video database, which is not limited here.
  • the facial image can be preprocessed.
  • the above step S1 includes:
  • Step S11 Obtain multiple frames of initial face images in the video, where the initial face images are unprocessed images;
  • Step S12 Perform face detection on the initial face image to determine the face area
  • Step S13 Obtain feature points of the face area, and perform alignment calibration on the initial face image according to the feature points to obtain the face image to be recognized.
  • multiple frames of initial face images in the video are acquired, where the initial face images are directly acquired from the video without any processing, such as the above-mentioned acquisition through a camera or external device, and then these initial Face detection is performed on the face image to determine the face area, which can be specifically detected by algorithms such as neural network algorithm, genetic algorithm, Adaboost face detection algorithm based on Haar-Like feature, and these algorithms are all existing technologies. Without going into details, after the face is detected, the face area can be determined by adjusting the size of the window.
  • the position of the feature points of the corresponding face (such as eyes, eyebrows, nose, mouth, outer contour of the face, etc.) from the face area.
  • the corresponding person is transformed by rigid body transformation.
  • the face image is aligned and calibrated, so that the positions of the feature points of the face in the image are basically the same, so that the face image to be recognized is obtained.
  • the above-mentioned facial expression recognition model based on convolutional neural network in this embodiment is followed by a convolutional neural network (CNN, Convolutional Neural Network).
  • CNN convolutional Neural Network
  • LSTM-RNN model long and short-term memory cyclic neural network model
  • the first pooling layer and the logistic regression model are constructed, which are obtained by training the continuous frame image training set with annotated expression categories.
  • the specific training process is the current There are technologies, which will not be repeated here.
  • the input is multiple frames of face images, and the output is a value.
  • the input is a laughing face image, and the model outputs the value "1".
  • This value is the result of the aforementioned expression recognition, and the expression category corresponding to "1" is excited, different values correspond to different preset expression categories, and each expression category corresponds to a mood category.
  • the corresponding continuous frame atlas can be obtained from the preset database according to the facial expression recognition result.
  • the pictures included are recorded as the first picture.
  • Each continuous frame atlas includes images representing the same category of mood.
  • the atlases all have a specified number of pictures with the same format.
  • each frame of pictures corresponds to a number, such as Respectively from the start number to the end number in a preset order, for example, each continuous frame atlas is set with 100 frames of pictures, the numbers are from 1 to 100, each frame of pictures is a different electronic line chart, in which the red The electronic line chart indicates happiness, and the yellow electronic line chart indicates sadness, etc.
  • the line chart is played in a loop, one or more continuous up and down moving pictures are presented, so that the user can intuitively feel the current mood.
  • the above animation is displayed on the side of the face image in the video, and can also be displayed at the designated position of the current video interface, which is not limited here.
  • step S3 includes:
  • Step S31 Compare the expression recognition result with the mapping value corresponding to each continuous frame atlas in the list of continuous frame atlases to find a target mapping value consistent with the expression recognition result, and the expression recognition result is all
  • the numerical value output by the expression recognition model, the continuous frame atlas list includes a plurality of preset continuous frame atlases corresponding to different expression categories, each continuous frame atlas includes a specified number of pictures representing the same type of mood, and All correspond to different mapping values;
  • Step S32 Find the associated name of the first continuous frame atlas from the list of continuous frame atlases according to the target mapping value, and obtain the first continuous frame from the database according to the associated name Atlas.
  • the foregoing continuous frame atlas list is a preset atlas list, and the list includes multiple associated names corresponding to different types of continuous frame atlases, such as "happy atlas”, “sad atlas”, “Angry Atlas”, “Afraid Atlas”, etc., and each successive frame atlas corresponds to a different mapping value.
  • the mapping value of "Happy Atlas” corresponds to "1”
  • “Sad Atlas” corresponds to The mapping value of is "2", etc.
  • the above mapping value is used for comparison with the value output by the expression recognition model.
  • the target mapping value corresponds to The continuous frame atlas is the atlas required for the current display, that is, the first continuous frame atlas.
  • the associated name corresponding to the first continuous frame atlas is found from the list of continuous frame atlases, because the database A plurality of consecutive frame atlases and corresponding associated names are stored therein, and the first consecutive frame atlases are obtained from the database according to the associated names.
  • step S4 since each picture in the first continuous frame atlas has a corresponding number, and the start number to the end number are sequentially set in a preset order, in this embodiment, after the first continuous frame atlas is acquired , From the first picture in the preset order, that is, from the start number, the pictures in the first consecutive frame atlas are displayed and played in the preset order. When the picture with the corresponding end number is played, the corresponding start number The picture starts to play again, and so on.
  • the above picture format adopts the png format, which has the following advantages: when used to store grayscale images, the depth of the grayscale images can be as much as 16 bits, and when storing color images, the depth of the color images can be as much as 48 bits, and It can also store up to 16 bits of alpha channel data.
  • the compression ratio is high and the generated file size is small. In real-time display of mood and emotions, it is more fluent and can more accurately display the user's current mood state and improve user experience.
  • step S4 after the above step S4, it includes:
  • Step S5 Receive an instruction to replace the first continuous frame atlas, where the instruction includes identification information of the second continuous frame atlas;
  • Step S6 Acquire the second continuous frame atlas according to the instruction, the second continuous frame atlas includes a specified number of second pictures representing the same category of mood, each of the second pictures has been sorted in a preset order , And the second picture and the first picture respectively represent different types of moods;
  • Step S7 Play the first second picture in the preset order in the second continuous frame atlas in the next frame of the current frame in the first continuous frame atlas, and compare the second pictures in the preset order The second picture in the continuous frame atlas is played in a loop.
  • the system will issue an instruction to change the playback atlas, that is, replace the above-mentioned first An instruction for a continuous frame atlas, which contains identification information for identifying a second continuous frame atlas, where the expression category corresponding to the second continuous frame atlas is the expression category recognized after the aforementioned face change expression, namely The first continuous frame atlas and the second continuous frame atlas represent different types of moods.
  • the second picture in the second continuous frame atlas is played in a preset order. It should be noted that the above-mentioned current frame is the first picture with any number in the first continuous frame atlas.
  • the preset mood text corresponding to the continuous frame atlas is called up and displayed at the same time, so that the meaning expressed on the interface is more accurate and vivid.
  • step S4 after the above step S4, it includes:
  • Step S41 Detect the loop playback duration of the first continuous frame atlas, where the loop playback duration is the duration from the initial playback moment of the first continuous frame atlas to the current moment;
  • Step S42 Determine whether the loop playback duration is greater than a preset duration
  • Step S43 If yes, obtain the prompt words corresponding to the first continuous frame atlas and display them.
  • the first continuous frame image is detected first.
  • the loop playback duration of the episode is the duration from the start time of playing the first picture of the continuous frame atlas to the current moment, and then determine whether the loop playback duration is greater than the preset duration, if so, It means that the emoticon lasts too long.
  • the prompt words corresponding to the first continuous frame atlas will be obtained and displayed.
  • the first continuous frame atlas start playing time is 10:20, and the current time is 11: 00, that is, the loop playback duration is 40 minutes, but the preset duration is 30 minutes, then the loop playback duration is greater than the preset duration, the prompts can be displayed, and each continuous frame atlas can correspond to different prompts ,
  • the prompt can be a reminder or a language that eases the atmosphere.
  • the prompt can be a language to remind the user "You have already I've been sad for xx minutes", or a joke with a funny reminder.
  • the aforementioned loop playback duration can be continuously monitored and detected. Once the loop playback duration is detected to be greater than the preset duration, the aforementioned prompt will be displayed.
  • step S4 after the above step S4, it includes:
  • Step S44 Determine whether the first continuous frame atlas is a preset designated atlas
  • Step S45 If yes, obtain a third continuous frame atlas having an associated relationship with the first continuous frame atlas, where the associated relationship is the mood category represented by the first continuous frame atlas and the first continuous frame atlas.
  • the associated relationship is the mood category represented by the first continuous frame atlas and the first continuous frame atlas.
  • Step S46 Replace the first continuous frame atlas currently played in a loop with the third continuous frame atlas, and display the alert for changing the atlas.
  • the above displayed prompt can be a text that encourages a change of mood, and in addition, a continuous frame atlas corresponding to the mood opposite to the current mood (such as happy, calm, etc.) can be called out , To encourage users to change their mood with a more vivid presentation.
  • the above-mentioned preset designated atlas can be an atlas set by the user according to the actual situation, generally set as an atlas of negative emotions, such as an atlas of anger, an atlas of sadness, etc.
  • the first consecutive frame is judged first Whether the atlas is the specified atlas, if not, there is no need to change the first continuous frame atlas currently played in a loop; if so, the third continuous frame atlas associated with the first continuous frame atlas will be obtained.
  • the association relationship can be a preset corresponding relationship, and the corresponding relationship is specifically the opposite relationship between the mood category represented by the first continuous frame atlas and the mood category represented by the third continuous frame atlas. For example, the happy atlas corresponds to the sad atlas, The calm atlas corresponds to the angry atlas, etc.
  • the third consecutive frame atlas can be replaced in the manner of step S7, and the warning of the changed atlas is displayed at the same time to remind the user.
  • the image display device based on data processing in this embodiment includes:
  • the image obtaining unit 100 is used to obtain a face image to be recognized in the video;
  • the recognition image unit 200 is configured to input the face image into a preset expression recognition model based on convolutional neural network for recognition to obtain an expression recognition result, and the expression recognition model is trained by a continuous frame image training set that annotates expression categories Obtained, different expression recognition results correspond to different expression categories;
  • the first acquiring unit 300 is configured to acquire a first continuous frame atlas corresponding to an expression category from a preset database according to the expression recognition result, the first continuous frame atlas including a specified number of first continuous frame atlases representing the same category of mood Pictures, each of the first pictures has been sorted in a preset order;
  • the first playing unit 400 is configured to play the first picture in the first continuous frame atlas according to the preset order, starting from the first picture in the preset order.
  • the face image to be recognized in the aforementioned video includes a single frame or two or more frames of face images.
  • the face image is acquired in real time through the camera, or by receiving the face image sent by an external device, or based on the user's selection from an existing image database or video database, which is not limited here.
  • the facial image may be preprocessed.
  • the above-mentioned image acquisition unit 100 includes:
  • the image acquisition subunit is used to acquire multiple frames of initial face images in the video, where the initial face images are unprocessed images;
  • the determining area subunit is used to perform face detection on the initial face image to determine the face area
  • the calibration image subunit is used to obtain the feature points of the face area, and perform alignment calibration on the initial face image according to the feature points to obtain the face image to be recognized.
  • multiple frames of initial face images in the video are acquired, where the initial face images are directly acquired from the video without any processing, such as the above-mentioned acquisition through a camera or external device, and then these initial Face detection is performed on the face image to determine the face area, which can be specifically detected by algorithms such as neural network algorithm, genetic algorithm, Adaboost face detection algorithm based on Haar-Like feature, and these algorithms are all existing technologies. Without going into details, after the face is detected, the face area can be determined by adjusting the size of the window.
  • the position of the feature points of the corresponding face (such as eyes, eyebrows, nose, mouth, outer contour of the face, etc.) from the face area.
  • the corresponding person is transformed by rigid body transformation.
  • the face image is aligned and calibrated, so that the positions of the feature points of the face in the image are basically the same, so that the face image to be recognized is obtained.
  • the above-mentioned facial expression recognition model based on convolutional neural network in this embodiment is composed of a convolutional neural network (CNN, Convolutional Neural Network) from input to output.
  • CNN convolutional neural network
  • Neural Network long and short-term memory cyclic neural network model
  • the first pooling layer and the logistic regression model are constructed, which are obtained by training the continuous frame image training set with annotated expression categories.
  • the specific training process is the current There are technologies, which will not be repeated here.
  • the input is multiple frames of face images, and the output is a value.
  • the input is a laughing face image, and the model outputs the value "1". This value is the result of the aforementioned expression recognition, and the expression category corresponding to "1" is excited, different values correspond to different preset expression categories, and each expression category corresponds to a mood category.
  • the corresponding continuous frame atlas can be acquired from the preset database according to the facial expression recognition result, in order to distinguish the continuous frame atlas. It is the first continuous frame atlas, and the included pictures are recorded as the first picture.
  • the database stores multiple continuous frame atlases, corresponding to different expression categories, and each continuous frame atlas includes images representing the same category of mood Each atlas has a specified number of pictures with the same format.
  • all the pictures in the atlas are sorted in a preset order, and the content between the previous and next frames is related, and each frame of picture corresponds to one Numbers, such as from the start number to the end number in a preset order, for example, each continuous frame atlas is set with 100 frames of pictures, the numbers are from 1 to 100, and each frame of pictures is a different electronic line chart , Where the red electronic line chart means happy, the yellow electronic line chart means sad, etc.
  • the line chart is played in a loop, one or more continuous up and down moving pictures are presented, making the user intuitively feel the current mood.
  • the aforementioned animation is displayed on the side of the face image in the video, and may also be displayed at a designated position on the current video interface, which is not limited here.
  • the aforementioned first acquiring unit 300 includes:
  • the comparison value subunit is used to compare the expression recognition result with the mapping value corresponding to each continuous frame atlas in the continuous frame atlas list to find a target mapping value consistent with the expression recognition result.
  • the recognition result is the value output by the facial expression recognition model.
  • the continuous frame atlas list includes a plurality of preset continuous frame atlases corresponding to different facial expression categories, and each continuous frame atlas includes a specified number of moods representing the same category. , And they all correspond to different mapping values;
  • the acquiring atlas subunit is configured to find the associated name of the first continuous frame atlas from the list of continuous frame atlases according to the target mapping value, and obtain all the associated names from the database according to the associated name.
  • the first consecutive frame atlas is configured to find the associated name of the first continuous frame atlas from the list of continuous frame atlases according to the target mapping value, and obtain all the associated names from the database according to the associated name. The first consecutive frame atlas.
  • the foregoing continuous frame atlas list is a preset atlas list, and the list includes multiple associated names corresponding to different types of continuous frame atlases, such as "happy atlas”, “sad atlas”, “Angry Atlas”, “Afraid Atlas”, etc., and each successive frame atlas corresponds to a different mapping value.
  • the mapping value of "Happy Atlas” corresponds to "1”
  • “Sad Atlas” corresponds to The mapping value of is "2", etc.
  • the above mapping value is used for comparison with the value output by the expression recognition model.
  • the target mapping value corresponds to The continuous frame atlas is the atlas required for the current display, that is, the first continuous frame atlas.
  • the associated name corresponding to the first continuous frame atlas is found from the list of continuous frame atlases, because the database A plurality of consecutive frame atlases and corresponding associated names are stored therein, and the first consecutive frame atlases are obtained from the database according to the associated names.
  • the first continuous frame is acquired in this embodiment After the atlas, from the first picture in the preset order, that is, from the start number, the pictures in the first continuous frame atlas are displayed and played in the preset order. When the picture with the corresponding end number is played, the corresponding The picture with the starting number starts to play again, and repeats like this.
  • the above picture format adopts the png format, which has the following advantages: when used to store grayscale images, the depth of the grayscale images can be as much as 16 bits, and when storing color images, the depth of the color images can be as much as 48 bits, and It can also store up to 16 bits of alpha channel data.
  • the compression ratio is high and the generated file size is small. In real-time display of mood and emotions, it is more fluent and can more accurately display the user's current mood state and improve user experience.
  • the above-mentioned picture display device based on data processing further includes:
  • An instruction receiving unit configured to receive an instruction to replace the first continuous frame atlas, where the instruction includes identification information of the second continuous frame atlas;
  • the second playback unit is configured to acquire the second continuous frame atlas according to the instruction, the second continuous frame atlas includes a specified number of second pictures representing the same category of mood, each of the second pictures has been clicked Sorting in a preset order, and the second picture and the first picture respectively represent different types of moods;
  • the second playback unit is configured to play the first and second pictures in the preset order in the second continuous frame atlas in the next frame of the current frame in the first continuous frame atlas, and to pair them in the preset order
  • the second picture in the second continuous frame atlas is played in a loop.
  • the system will issue an instruction to change the playback atlas, that is, replace the above-mentioned first An instruction for a continuous frame atlas, which contains identification information for identifying a second continuous frame atlas, where the expression category corresponding to the second continuous frame atlas is the expression category recognized after the aforementioned face change expression, namely The first continuous frame atlas and the second continuous frame atlas represent different types of moods.
  • the second picture in the second continuous frame atlas is played in a preset order. It should be noted that the above-mentioned current frame is the first picture with any number in the first continuous frame atlas.
  • the preset mood text corresponding to the continuous frame atlas is called up and displayed at the same time, so that the meaning expressed on the interface is more accurate and vivid.
  • the above-mentioned picture display device based on data processing further includes:
  • the detection duration unit is configured to detect the loop playback duration of the first continuous frame atlas, where the loop playback duration is the duration from the initial playback moment of the first continuous frame atlas to the current moment;
  • the duration determining unit is used to determine whether the duration of the loop playback is greater than a preset duration
  • the display sentence unit is used to determine that when the loop playback duration is greater than the preset duration, obtain the prompt language corresponding to the first continuous frame atlas, and display it.
  • the first continuous frame image is detected first.
  • the loop playback duration of the episode is the duration from the start time of playing the first picture of the continuous frame atlas to the current moment, and then determine whether the loop playback duration is greater than the preset duration, if so, It means that the emoticon lasts too long.
  • the prompt words corresponding to the first continuous frame atlas will be obtained and displayed.
  • the first continuous frame atlas start playing time is 10:20, and the current time is 11: 00, that is, the loop playback duration is 40 minutes, but the preset duration is 30 minutes, then the loop playback duration is greater than the preset duration, the prompts can be displayed, and each continuous frame atlas can correspond to different prompts ,
  • the prompt language can be a warning language or a language that eases the atmosphere, which is not limited here. If the loop playback duration is not greater than the preset duration, no response will be made until the album is changed or the loop playback duration is greater than the preset duration. In this embodiment, the aforementioned loop playback duration can be continuously monitored and detected. Once the loop playback duration is detected to be greater than the preset duration, the aforementioned prompt will be displayed.
  • the above-mentioned picture display device based on data processing further includes:
  • the third acquiring unit is configured to acquire a third continuous frame atlas having an association relationship with the first continuous frame atlas when judging that the first continuous frame atlas is a preset designated atlas, wherein the The association relationship is an opposite correspondence between the mood category represented by the first continuous frame atlas and the mood category represented by the third continuous frame atlas;
  • the replacement atlas unit is used to replace the first continuous frame atlas currently played in a loop with the third continuous frame atlas, and display a warning to change the atlas.
  • the above displayed prompt can be a text that encourages a change of mood, and in addition, a continuous frame atlas corresponding to the mood opposite to the current mood (such as happy, calm, etc.) can be called out , To encourage users to change their mood with a more vivid presentation.
  • the above-mentioned preset designated atlas can be an atlas set by the user according to the actual situation, generally set as an atlas of negative emotions, such as an atlas of anger, an atlas of sadness, etc.
  • the first consecutive frame is judged first Whether the atlas is the specified atlas, if not, there is no need to change the first continuous frame atlas currently played in a loop; if so, the third continuous frame atlas associated with the first continuous frame atlas will be obtained.
  • the association relationship can be a preset corresponding relationship, and the corresponding relationship is specifically the opposite relationship between the mood category represented by the first continuous frame atlas and the mood category represented by the third continuous frame atlas. For example, the happy atlas corresponds to the sad atlas, The calm atlas corresponds to the angry atlas, etc.
  • the third consecutive frame atlas can be replaced by the replacement method of the second playback unit, and the warning of the changed atlas is displayed at the same time to remind the user.
  • an embodiment of the present application also provides a computer device.
  • the computer device may be a server, and its internal structure may be as shown in FIG. 3.
  • the computer equipment includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the computer designed processor is used to provide calculation and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, a computer program, and a database.
  • the memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the database of the computer equipment is used to store all the data needed to display the picture.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection.
  • the computer program is executed by the processor to realize a picture display method based on data processing.
  • the processor executes the steps of the image display method based on data processing: acquiring the face image to be recognized in the video; inputting the face image to a preset facial expression recognition model based on convolutional neural network for recognition to obtain facial expression recognition
  • the expression recognition model is obtained by training a continuous frame image training set with annotated expression categories, and different expression recognition results correspond to different expression categories; according to the expression recognition results, the first expression category corresponding to the expression category is obtained from a preset database.
  • a continuous frame atlas the first continuous frame atlas includes a specified number of first pictures representing the same type of mood, each of the first pictures has been sorted in a preset order; the first continuous frame atlas The first picture is played in a loop according to the preset order starting from the first picture in the preset order.
  • the method includes: receiving and replacing the first picture An instruction for a set of continuous frame atlas, the instruction including the identification information of a second set of continuous frame atlas; the second set of continuous frame atlas is obtained according to the instruction, and the second set of continuous frame atlas includes a specified number representing the same category
  • the second picture of mood each of the second pictures has been sorted in a preset order, and the second picture and the first picture respectively represent different types of moods; in the first continuous frame atlas, the current frame is played in the next frame
  • the first second picture in the preset order in the second continuous frame atlas, and the second picture in the second continuous frame atlas is played cyclically in the preset order.
  • the method includes: detecting the first picture The loop playback duration of a continuous frame atlas, where the loop playback duration is the duration from the start playback moment of the first continuous frame atlas to the current moment; determine whether the loop playback duration is greater than a preset duration If yes, obtain the prompt words corresponding to the first continuous frame atlas and display them.
  • the method includes: determining whether the first continuous frame atlas is a preset designated atlas; if so , Acquiring a third continuous frame atlas having an association relationship with the first continuous frame atlas, where the association relationship is the mood category represented by the first continuous frame atlas and the third continuous frame atlas The corresponding relationship between the mood categories represented by the set is opposite; the first continuous frame atlas currently played in a loop is replaced with the third continuous frame atlas, and a warning to change the atlas is displayed.
  • the step of acquiring the face image to be recognized in the video includes: acquiring multiple frames of initial face images in the video, where the initial face images are unprocessed images; Face detection is performed on the face image to determine the face area; the feature points of the face area are acquired, and the initial face image is aligned and calibrated according to the feature points to obtain the face image to be recognized.
  • the step of obtaining the first continuous frame atlas corresponding to the facial expression category from a preset database according to the facial expression recognition result includes: combining the facial facial recognition result with each continuous frame atlas list in the continuous frame atlas list The corresponding mapping values of the frame atlases are compared to find a target mapping value that is consistent with the expression recognition result.
  • the expression recognition result is the value output by the expression recognition model, and the continuous frame atlas list includes a plurality of corresponding The preset continuous frame atlases of different expression categories, each continuous frame atlas includes a specified number of pictures representing the same type of mood, and each corresponds to a different mapping value; according to the target mapping value from the continuous frame Find the associated name of the first continuous frame atlas in the atlas list, and obtain the first continuous frame atlas from the database according to the associated name.
  • FIG. 3 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the storage medium is a volatile storage medium or a non-volatile storage medium, and a computer program is stored thereon.
  • a The image display method based on data processing specifically includes: acquiring a face image to be recognized in a video; inputting the face image into a preset expression recognition model based on convolutional neural network for recognition to obtain an expression recognition result,
  • the facial expression recognition model is obtained by training a training set of continuous frame images marked with facial expression categories, and different facial expression recognition results correspond to different facial expression categories; according to the facial expression recognition results, the first continuous frame image corresponding to the facial expression category is obtained from a preset database
  • the first continuous frame atlas includes a specified number of first pictures representing the mood of the same category, and each of the first pictures has been sorted in a preset order; and the first picture in the first continuous frame atlas is sorted from The first frame of the preset sequence starts to be played in a loop according to the prese
  • the method includes: receiving a replacement place The instruction of the first continuous frame atlas, the instruction includes the identification information of the second continuous frame atlas; the second continuous frame atlas is acquired according to the instruction, and the second continuous frame atlas includes a specified number of The second pictures representing the mood of the same category, each of the second pictures has been sorted in a preset order, and the second picture and the first picture respectively represent different categories of mood; in the first continuous frame atlas next to the current frame Frame playback of the first second picture in the preset order in the second continuous frame atlas, and loop playback of the second picture in the second continuous frame atlas in the preset order.
  • the method includes: detecting the first picture The loop playback duration of a continuous frame atlas, where the loop playback duration is the duration from the start playback moment of the first continuous frame atlas to the current moment; determine whether the loop playback duration is greater than a preset duration If yes, obtain the prompt words corresponding to the first continuous frame atlas and display them.
  • the method includes: determining whether the first continuous frame atlas is a preset designated atlas; if so , Acquiring a third continuous frame atlas having an association relationship with the first continuous frame atlas, where the association relationship is the mood category represented by the first continuous frame atlas and the third continuous frame atlas The corresponding relationship between the mood categories represented by the set is opposite; the first continuous frame atlas currently played in a loop is replaced with the third continuous frame atlas, and a warning to change the atlas is displayed.
  • the step of acquiring the face image to be recognized in the video includes: acquiring multiple frames of initial face images in the video, where the initial face images are unprocessed images; Face detection is performed on the face image to determine the face area; the feature points of the face area are acquired, and the initial face image is aligned and calibrated according to the feature points to obtain the face image to be recognized.
  • the step of obtaining the first continuous frame atlas corresponding to the facial expression category from a preset database according to the facial expression recognition result includes: combining the facial facial recognition result with each continuous frame atlas list in the continuous frame atlas list
  • the mapping values corresponding to the frame atlases are compared to find a target mapping value that is consistent with the expression recognition result, the expression recognition result is the value output by the expression recognition model, and the continuous frame atlas list includes multiple corresponding
  • the preset continuous frame atlases of different expression categories, each continuous frame atlas includes a specified number of pictures representing the same type of mood, and each corresponds to a different mapping value; according to the target mapping value from the continuous frame Find the associated name of the first continuous frame atlas in the atlas list, and obtain the first continuous frame atlas from the database according to the associated name.
  • Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • SDRAM dual-rate data rate SDRAM
  • SSRSDRAM dual-rate data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous Link (Synchlink) DRAM
  • SLDRAM synchronous Link (Synchlink) DRAM
  • Rambus direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请提出基于数据处理的图片显示方法、装置、计算机设备和存储介质，其中方法包括：获取视频中待识别的人脸图像；将人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的表情识别结果对应不同的表情类别；根据表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各第一图片已按预设顺序排序；将第一连续帧图集中的第一图片从预设顺序的第一张开始按照预设顺序进行循环播放，通过循环播放图片形成表情动图，从而生动地表现用户的实时心情，提高用户体验。

Description

基于数据处理的图片显示方法、装置和计算机设备
本申请要求于2019年7月3日提交中国专利局、申请号为201910596485.8，发明名称为“基于数据处理的图片显示方法、装置和计算机设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及到人工智能技术领域，特别是涉及到一种基于数据处理的图片显示方法、装置、计算机设备和存储介质。
 背景技术
目前检测心情的方法有多种，其中常用的是通过识别微表情来得到其对应的心情，如，先进行表情识别，得到对应的心情，然后将该心情在界面中显示以让用户知晓，发明人发现在现有技术中，主要通过文字来表达心情，先判断用户的心情，然后将该心情通过文字描述显示在用户界面上，例如，识别用户表情，判断其为“开心”时，在界面上显示“开心”二字；识别用户表情，判断其为“平静”时，在界面上显示“平静”二字，但这样并不能准确生动地表现用户的实时心情。
 技术问题
本申请的主要目的为提供一种基于数据处理的图片显示方法、装置、计算机设备和存储介质，旨在解决现有显示心情的方法不能生动地表现用户的实时心情。
 技术解决方案
基于上述发明目的，本申请提出一种基于数据处理的图片显示方法，包括：
获取视频中待识别的人脸图像；
将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
本申请还提供一种基于数据处理的图片显示装置，包括：
获取图像单元，用于获取视频中待识别的人脸图像；
识别图像单元，用于将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
第一获取单元，用于根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
第一播放单元，用于将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
本申请还提供一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现一种基于数据处理的图片显示方法，其中，所述基于数据处理的图片显示方法包括：
获取视频中待识别的人脸图像；
将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
本申请还提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现一种基于数据处理的图片显示方法，其中，所述基于数据处理的图片显示方法包括：
获取视频中待识别的人脸图像；
将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
 有益效果
通过实时识别出的表情类别获取对应的连续帧图集，然后将连续图集中的图片按预设顺序进行循环播放，通过循环播放图片形成表情动图，从而生动地表现用户的实时心情，准确地显示用户当下心情状态，提高用户体验。
 附图说明
图1 为本申请一实施例中基于数据处理的图片显示方法的步骤示意图；
图2为本申请一实施例中基于数据处理的图片显示装置的结构示意框图；
图3为本申请一实施例的计算机设备的结构示意框图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
 本发明的最佳实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
参照图1，本实施例中的基于数据处理的图片显示方法，包括：
步骤S1：获取视频中待识别的人脸图像；
步骤S2：将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
步骤S3：根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
步骤S4：将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
如上述步骤S1所述，上述视频中待识别的人脸图像包括单帧或两帧以上人脸图像。首先通过摄像头实时获取人脸图像，或者通过接收外部设备发送的人脸图像，或者基于用户在已有图像数据库或视频数据库中选取，此处不作限定。为便于识别人脸图像的人脸表情，可对人脸图像进行预处理，在一个实施例中，上述步骤S1包括：
步骤S11：获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；
步骤S12：对所述初始人脸图像进行人脸检测，以确定人脸区域；
步骤S13：获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
本实施例中，获取视频中多帧初始人脸图像，此处的初始人脸图像为从视频中直接获取，且未经任何处理的图像，如上述通过摄像头或外部设备获取，然后对这些初始人脸图像进行人脸检测，以便确定人脸区域，具体可通过神经网络算法、遗传算法、基于Haar-Like特征的Adaboost人脸检测算法等算法进行检测，这些算法均为现有技术，此处不再赘述，检测到人脸后，可通过调整窗口的大小来确定人脸区域。
然后从人脸区域获取对应人脸的特征点(例如眼睛、眉毛、鼻子、嘴巴、脸部外轮廓等)的位置，根据在人脸区域中检测到的特征点，通过刚体变换对相应的人脸图像进行对齐校准，使得人脸在图像中各特征点的位置基本一致，从而得到上述待识别的人脸图像。
如上述步骤S2所述，本实施例中上述基于卷积神经网络的表情识别模型，从输入端到输出端依次由卷积神经网络(CNN,Convolutional Neural Network)模型、长短时记忆循环神经网络模型(即LSTM-RNN模型)、第一池化层和逻辑回归模型构建，通过标注表情类别的连续帧图像训练集训练得到，具体的训练过程为现有技术，此处不再赘述，对于该表情识别模型，输入的为多帧人脸图像，输出的为一个数值，如输入的为大笑的人脸图像，模型输出的为数值“1”，该数值即上述表情识别结果，而“1”对应的表情类别为兴奋，不同的数值对应不同的预设的表情类别，每一表情类别对应一种心情类别。
如上述步骤S3所述，在得表情识别模型识别后的结果之后，可根据表情识别结果从预设的数据库中获取对应的连续帧图集，为了便于区别将该连续帧图集记为第一连续帧图集，包含的图片记为第一图片，该数据库中存储有多个连续帧图集，分别对应不同的表情类别，每个连续帧图集均包括表示同一类别心情的图像，每个图集均具有指定数量格式一致的图片，本实施例中事先已将图集中的所有图片均按预设的顺序进行排序，且前后帧之间内容相关，每一帧图片均对应一个编号，如按预设顺序分别从起始编号到结束编号，举例地，每个连续帧图集均设置有100帧图片，编号分别是从1到100，每帧图片均为不同的电子折线图，其中红色电子折线图表示开心，黄色电子折线图表示难过等，当对折线图进行循环播放时，呈现出一条或多条连续的上下起伏的动图，使得用户直观地感受到当前的心情，本实施例中，上述动图显示在视频中人脸图像的旁侧，也可显示在当前视频界面的指定位置，此处不作限定。
具体而言，上述步骤S3包括：
步骤S31：将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；
步骤S32：依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
本实施例中，上述连续帧图集列表为预设的图集列表，该列表包括多个分别对应不同类别的连续帧图集的关联名称，如“开心图集”、“难过图集”、“生气图集”、“害怕图集”等等，且每个连续帧图集均分别对应不同的映射值，如“开心图集”对应的映射值为“1”、“难过图集”对应的映射值为“2”等，上述映射值用于与表情识别模型输出的值作对比。首先将表情识别结果，即表情识别模型输出的数值与上述连续帧图集中的各个连续帧图集对应的映射值进行对比，从中找到与模型输出值一致的目标映射值，该目标映射值对应的连续帧图集即为当前显示所需的图集，也即第一连续帧图集，根据目标映射值从上述连续帧图集列表中找到上述第一连续帧图集对应的关联名称，由于数据库中存储有多个连续帧图集以及对应的关联名称，这时根据关联名称从上述数据库中获取上述第一连续帧图集。
如上述步骤S4所述，由于第一连续帧图集中每帧图片均对应有编号，且起始编号到结束编号依次按预设顺序进行设置，本实施例中获取到第一连续帧图集之后，从预设顺序的第一张图片，即从起始编号开始按预设顺序对第一连续帧图集中的图片进行显示播放，当播放到对应结束编号的图片时，再从对应起始编号的图片开始重新播放，如此循环播放。
进一步地，上述图片格式采用png格式，其具有如下优点：用来存储灰度图像时，灰度图像的深度可多到16位，存储彩色图像时，彩色图像的深度可多到48位，并且还可存储多到16位的α通道数据。压缩比高，生成文件体积小。在实时显示心情情绪时，更加流畅，可更准确的显示用户当下心情状态，提高用户体验。
在一个实施例中，上述步骤S4之后，包括：
步骤S5：接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
步骤S6：依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
步骤S7：在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
本实施例中，当上述视频中的人脸变更表情，那么对应的，获取到并识别出的表情类别与当前循环播放的不一致，这时系统会发出更换播放图集的指令，即更换上述第一连续帧图集的指令，该指令包含识别出第二连续帧图集的识别信息，此处的第二连续帧图集对应的表情类别为上述人脸变更表情之后识别得到的表情类别，即第一连续帧图集与第二连续帧图集表示不同类别的心情，为了使得显示的动图无缝衔接，直接在播放当前帧的下一帧开始播放第二连续帧图集中对应起始编号的第二图片，并按预设顺序对第二连续帧图集中的第二图片进行循环播放，需知，上述当前帧为第一连续帧图集中的任意编号的第一图片。
进一步地，在循环播放连续帧图集中的图片时，同时将预设好的与该连续帧图集对应的心情文字调出并显示，让呈现到界面的意思表示更准确生动。
在一个实施例中，上述步骤S4之后，包括：
步骤S41：检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；
步骤S42：判断所述循环播放时长是否大于预设的时长；
步骤S43：若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
本实施例中，由于上述基于数据处理的图片显示方法应用于实时识别视频人脸表情，为了给用户更贴心体验，可对长时间同一表情进行提示警醒，具体而已，先检测第一连续帧图集的循环播放时长，该循环播放时长即是从播放该连续帧图集第一张图片的起始时刻到当前时刻之间的时长，然后判断该循环播放时长是否大于预设的时长，若是，则说明该表情持续时间过长，这时会获取对应该第一连续帧图集的提示语，并进行显示，例如第一连续帧图集起始播放时刻为10:20，当前时刻为11:00，即循环播放时长为40分钟，但是预设的时长为30分钟，那么此时循环播放时长大于预设的时长，可将提示语进行显示，每个连续帧图集可对应不同的提示语，当然，该提示语可为警醒语，也可为缓解气氛的语言，此处不作限定，例如，当播放的为对应心情难过的图集，则其提示语可为提醒用户的语言“你已经难过xx分钟了”，或者提示语为搞笑的段子等。若循环播放时长不大于预设时长，不作任何响应，直到更换图集或循环播放时长大于预设时长。本实施例中，可持续监控并检测上述循环播放时长，一旦检测到循环播放时长大于预设的时长，即显示上述提示语。
在一个实施例中，上述步骤S4之后，包括：
步骤S44：判断所述第一连续帧图集是否为预设的指定图集；
步骤S45：若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；
步骤S46：将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
本实施例中，当心情呈现为负面情绪时，上述显示的提示语可为鼓励改变心情的文字，此外还可调出与当前心情反面的心情对应的连续帧图集（如开心，平静等），以更为生动的呈现方式来促使用户改变心情。具体而言，上述预设的指定图集可为用户根据实际情况设置的图集，一般设置为负面情绪的图集，例如表示愤怒的图集、难过的图集等，先判断第一连续帧图集是否为该指定图集，若否，无需对当前循环播放的第一连续帧图集进行变更，若是，则获取与第一连续帧图集具有关联关系的第三连续帧图集，上述关联关系可为预设的对应关系，该对应关系具体为第一连续帧图集表示的心情类别与第三连续帧图集表示的心情类别相反的对应关系，例如高兴图集对应难过图集、平静图集对应愤怒图集等，这时可将第三连续帧图集按步骤S7的方式，进行图集更换，同时显示变更图集的警醒语，以便提醒用户。
参照图2，本实施例中基于数据处理的图片显示装置，包括：
获取图像单元100，用于获取视频中待识别的人脸图像；
识别图像单元200，用于将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
第一获取单元300，用于根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
第一播放单元400，用于将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
如上述获取图像单元100所述，上述视频中待识别的人脸图像包括单帧或两帧以上人脸图像。首先通过摄像头实时获取人脸图像，或者通过接收外部设备发送的人脸图像，或者基于用户在已有图像数据库或视频数据库中选取，此处不作限定。为便于识别人脸图像的人脸表情，可对人脸图像进行预处理，在一个实施例中，上述获取图像单元100包括：
获取图像子单元，用于获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；
确定区域子单元，用于对所述初始人脸图像进行人脸检测，以确定人脸区域；
校准图像子单元，用于获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
本实施例中，获取视频中多帧初始人脸图像，此处的初始人脸图像为从视频中直接获取，且未经任何处理的图像，如上述通过摄像头或外部设备获取，然后对这些初始人脸图像进行人脸检测，以便确定人脸区域，具体可通过神经网络算法、遗传算法、基于Haar-Like特征的Adaboost人脸检测算法等算法进行检测，这些算法均为现有技术，此处不再赘述，检测到人脸后，可通过调整窗口的大小来确定人脸区域。
然后从人脸区域获取对应人脸的特征点(例如眼睛、眉毛、鼻子、嘴巴、脸部外轮廓等)的位置，根据在人脸区域中检测到的特征点，通过刚体变换对相应的人脸图像进行对齐校准，使得人脸在图像中各特征点的位置基本一致，从而得到上述待识别的人脸图像。
如上述识别图像单元200所述，本实施例中上述基于卷积神经网络的表情识别模型，从输入端到输出端依次由卷积神经网络(CNN,Convolutional Neural Network)模型、长短时记忆循环神经网络模型(即LSTM-RNN模型)、第一池化层和逻辑回归模型构建，通过标注表情类别的连续帧图像训练集训练得到，具体的训练过程为现有技术，此处不再赘述，对于该表情识别模型，输入的为多帧人脸图像，输出的为一个数值，如输入的为大笑的人脸图像，模型输出的为数值“1”，该数值即上述表情识别结果，而“1”对应的表情类别为兴奋，不同的数值对应不同的预设的表情类别，每一表情类别对应一种心情类别。
如上述第一获取单元300所述，在得表情识别模型识别后的结果之后，可根据表情识别结果从预设的数据库中获取对应的连续帧图集，为了便于区别将该连续帧图集记为第一连续帧图集，包含的图片记为第一图片，该数据库中存储有多个连续帧图集，分别对应不同的表情类别，每个连续帧图集均包括表示同一类别心情的图像，每个图集均具有指定数量格式一致的图片，本实施例中事先已将图集中的所有图片均按预设的顺序进行排序，且前后帧之间内容相关，每一帧图片均对应一个编号，如按预设顺序分别从起始编号到结束编号，举例地，每个连续帧图集均设置有100帧图片，编号分别是从1到100，每帧图片均为不同的电子折线图，其中红色电子折线图表示开心，黄色电子折线图表示难过等，当对折线图进行循环播放时，呈现出一条或多条连续的上下起伏的动图，使得用户直观地感受到当前的心情，本实施例中，上述动图显示在视频中人脸图像的旁侧，也可显示在当前视频界面的指定位置，此处不作限定。
具体而言，上述第一获取单元300包括：
比较数值子单元，用于将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；
获取图集子单元，用于依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
本实施例中，上述连续帧图集列表为预设的图集列表，该列表包括多个分别对应不同类别的连续帧图集的关联名称，如“开心图集”、“难过图集”、“生气图集”、“害怕图集”等等，且每个连续帧图集均分别对应不同的映射值，如“开心图集”对应的映射值为“1”、“难过图集”对应的映射值为“2”等，上述映射值用于与表情识别模型输出的值作对比。首先将表情识别结果，即表情识别模型输出的数值与上述连续帧图集中的各个连续帧图集对应的映射值进行对比，从中找到与模型输出值一致的目标映射值，该目标映射值对应的连续帧图集即为当前显示所需的图集，也即第一连续帧图集，根据目标映射值从上述连续帧图集列表中找到上述第一连续帧图集对应的关联名称，由于数据库中存储有多个连续帧图集以及对应的关联名称，这时根据关联名称从上述数据库中获取上述第一连续帧图集。
如上述第一播放单元400所述，由于第一连续帧图集中每帧图片均对应有编号，且起始编号到结束编号依次按预设顺序进行设置，本实施例中获取到第一连续帧图集之后，从预设顺序的第一张图片，即从起始编号开始按预设顺序对第一连续帧图集中的图片进行显示播放，当播放到对应结束编号的图片时，再从对应起始编号的图片开始重新播放，如此循环播放。
进一步地，上述图片格式采用png格式，其具有如下优点：用来存储灰度图像时，灰度图像的深度可多到16位，存储彩色图像时，彩色图像的深度可多到48位，并且还可存储多到16位的α通道数据。压缩比高，生成文件体积小。在实时显示心情情绪时，更加流畅，可更准确的显示用户当下心情状态，提高用户体验。
在一个实施例中，上述基于数据处理的图片显示装置，还包括：
接收指令单元，用于接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
第二播放单元，用于依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
第二播放单元，用于在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
本实施例中，当上述视频中的人脸变更表情，那么对应的，获取到并识别出的表情类别与当前循环播放的不一致，这时系统会发出更换播放图集的指令，即更换上述第一连续帧图集的指令，该指令包含识别出第二连续帧图集的识别信息，此处的第二连续帧图集对应的表情类别为上述人脸变更表情之后识别得到的表情类别，即第一连续帧图集与第二连续帧图集表示不同类别的心情，为了使得显示的动图无缝衔接，直接在播放当前帧的下一帧开始播放第二连续帧图集中对应起始编号的第二图片，并按预设顺序对第二连续帧图集中的第二图片进行循环播放，需知，上述当前帧为第一连续帧图集中的任意编号的第一图片。
进一步地，在循环播放连续帧图集中的图片时，同时将预设好的与该连续帧图集对应的心情文字调出并显示，让呈现到界面的意思表示更准确生动。
在一个实施例中，上述基于数据处理的图片显示装置，还包括：
检测时长单元，用于检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；
判断时长单元，用于判断所述循环播放时长是否大于预设的时长；
显示语句单元，用于判断所述循环播放时长大于预设的时长时，获取与所述第一连续帧图集对应的提示语，并进行显示。
本实施例中，由于上述基于数据处理的图片显示方法应用于实时识别视频人脸表情，为了给用户更贴心体验，可对长时间同一表情进行提示警醒，具体而已，先检测第一连续帧图集的循环播放时长，该循环播放时长即是从播放该连续帧图集第一张图片的起始时刻到当前时刻之间的时长，然后判断该循环播放时长是否大于预设的时长，若是，则说明该表情持续时间过长，这时会获取对应该第一连续帧图集的提示语，并进行显示，例如第一连续帧图集起始播放时刻为10:20，当前时刻为11:00，即循环播放时长为40分钟，但是预设的时长为30分钟，那么此时循环播放时长大于预设的时长，可将提示语进行显示，每个连续帧图集可对应不同的提示语，当然，该提示语可为警醒语，也可为缓解气氛的语言，此处不作限定。若循环播放时长不大于预设时长，不作任何响应，直到更换图集或循环播放时长大于预设时长。本实施例中，可持续监控并检测上述循环播放时长，一旦检测到循环播放时长大于预设的时长，即显示上述提示语。
在一个实施例中，上述基于数据处理的图片显示装置，还包括：
判断图集单元，用于判断所述第一连续帧图集是否为预设的指定图集；
第三获取单元，用于判断所述第一连续帧图集为预设的指定图集时，获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；
替换图集单元，用于将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
本实施例中，当心情呈现为负面情绪时，上述显示的提示语可为鼓励改变心情的文字，此外还可调出与当前心情反面的心情对应的连续帧图集（如开心，平静等），以更为生动的呈现方式来促使用户改变心情。具体而言，上述预设的指定图集可为用户根据实际情况设置的图集，一般设置为负面情绪的图集，例如表示愤怒的图集、难过的图集等，先判断第一连续帧图集是否为该指定图集，若否，无需对当前循环播放的第一连续帧图集进行变更，若是，则获取与第一连续帧图集具有关联关系的第三连续帧图集，上述关联关系可为预设的对应关系，该对应关系具体为第一连续帧图集表示的心情类别与第三连续帧图集表示的心情类别相反的对应关系，例如高兴图集对应难过图集、平静图集对应愤怒图集等，这时可将第三连续帧图集按第二播放单元的替换方式，进行图集更换，同时显示变更图集的警醒语，以便提醒用户。
参照图3，本申请实施例中还提供一种计算机设备，该计算机设备可以是服务器，其内部结构可以如图3所示。该计算机设备包括通过系统总线连接的处理器、存储器、网络接口和数据库。其中，该计算机设计的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的数据库用于存储显示图片所需的所有数据。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种基于数据处理的图片显示方法。
上述处理器执行上述基于数据处理的图片显示方法的步骤：获取视频中待识别的人脸图像；将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
上述计算机设备，上述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
在一个实施例中，上述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；判断所述循环播放时长是否大于预设的时长；若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
在一个实施例中，上述获取与所述第一连续帧图集对应的提示语，并进行显示的步骤之后，包括：判断所述第一连续帧图集是否为预设的指定图集；若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
在一个实施例中，上述获取视频中待识别的人脸图像的步骤，包括：获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；对所述初始人脸图像进行人脸检测，以确定人脸区域；获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
在一个实施例中，上述根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集的步骤，包括：将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
本领域技术人员可以理解，图3中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定。
本申请一实施例还提供一种计算机可读存储介质，所述存储介质为易失性存储介质或非易失性存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现一种基于数据处理的图片显示方法，具体为：获取视频中待识别的人脸图像；将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
上述计算机可读存储介质，上述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
在一个实施例中，上述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；判断所述循环播放时长是否大于预设的时长；若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
在一个实施例中，上述获取与所述第一连续帧图集对应的提示语，并进行显示的步骤之后，包括：判断所述第一连续帧图集是否为预设的指定图集；若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
在一个实施例中，上述获取视频中待识别的人脸图像的步骤，包括：获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；对所述初始人脸图像进行人脸检测，以确定人脸区域；获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
在一个实施例中，上述根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集的步骤，包括：将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储与一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的和实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可以包括只读存储器（ROM）、可编程ROM（PROM）、电可编程ROM（EPROM）、电可擦除可编程ROM（EEPROM）或闪存。易失性存储器可包括随机存取存储器（RAM）或者外部高速缓冲存储器。作为说明而非局限，RAM一多种形式可得，诸如静态RAM（SRAM）、动态RAM（DRAM）、同步DRAM（SDRAM）、双速据率SDRAM（SSRSDRAM）、增强型SDRAM（ESDRAM）、同步链路（Synchlink）DRAM（SLDRAM）、存储器总线（Rambus）直接RAM（RDRAM）、直接存储器总线动态RAM（DRDRAM）、以及存储器总线动态RAM（RDRAM）等。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、装置、物品或者方法不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、装置、物品或者方法所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、装置、物品或者方法中还存在另外的相同要素。
以上所述仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (20)

  1. 一种基于数据处理的图片显示方法，其中，包括：
    获取视频中待识别的人脸图像；
    将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
    根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
    将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
  2. 根据权利要求1所述基于数据处理的图片显示方法，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
    依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
    在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
  3. 根据权利要求1所述基于数据处理的图片显示方法，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；
    判断所述循环播放时长是否大于预设的时长；
    若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
  4. 根据权利要求3所述基于数据处理的图片显示方法，其中，所述获取与所述第一连续帧图集对应的提示语，并进行显示的步骤之后，包括：
    判断所述第一连续帧图集是否为预设的指定图集；
    若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；
    将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
  5. 根据权利要求1所述基于数据处理的图片显示方法，其中，所述获取视频中待识别的人脸图像的步骤，包括：
    获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；
    对所述初始人脸图像进行人脸检测，以确定人脸区域；
    获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
  6. 根据权利要求1所述基于数据处理的图片显示方法，其中，所述根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集的步骤，包括：
    将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；
    依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
  7. 一种基于数据处理的图片显示装置，其中，包括：
    获取图像单元，用于获取视频中待识别的人脸图像；
    识别图像单元，用于将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
    第一获取单元，用于根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
    第一播放单元，用于将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
  8. 根据权利要求1所述基于数据处理的图片显示装置，其中，包括：
    接收指令单元，用于接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
    第二播放单元，用于依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
    第二播放单元，用于在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
  9. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其中，所述处理器执行所述计算机程序时实现一种基于数据处理的图片显示方法，其中，所述基于数据处理的图片显示方法包括：
    获取视频中待识别的人脸图像；
    将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
    根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
    将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
  10. 根据权利要求9所述计算机设备，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
    依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
    在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
  11. 根据权利要求9所述计算机设备，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；
    判断所述循环播放时长是否大于预设的时长；
    若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
  12. 根据权利要求11所述计算机设备，其中，所述获取与所述第一连续帧图集对应的提示语，并进行显示的步骤之后，包括：
    判断所述第一连续帧图集是否为预设的指定图集；
    若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；
    将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
  13. 根据权利要求9所述计算机设备，其中，所述获取视频中待识别的人脸图像的步骤，包括：
    获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；
    对所述初始人脸图像进行人脸检测，以确定人脸区域；
    获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
  14. 根据权利要求9所述计算机设备，其中，所述根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集的步骤，包括：
    将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；
    依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
  15. 一种计算机可读存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现一种基于数据处理的图片显示方法，其中，所述基于数据处理的图片显示方法包括：
    获取视频中待识别的人脸图像；
    将所述人脸图像输入到预设的基于卷积神经网络的表情识别模型进行识别得到表情识别结果，所述表情识别模型通过标注表情类别的连续帧图像训练集训练得到，不同的所述表情识别结果对应不同的表情类别；
    根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集，所述第一连续帧图集包括指定数量的表示同一类别心情的第一图片，各所述第一图片已按预设顺序排序；
    将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放。
  16. 根据权利要求15所述计算机可读存储介质，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    接收更换所述第一连续帧图集的指令，所述指令包括第二连续帧图集的识别信息；
    依据所述指令获取所述第二连续帧图集，所述第二连续帧图集包括指定数量的表示同一类别心情的第二图片，各所述第二图片已按预设顺序排序，且第二图片与第一图片分别表示不同类别心情；
    在所述第一连续帧图集当前帧的下一帧播放所述第二连续帧图集中预设顺序的第一张第二图片，并按所述预设顺序对所述第二连续帧图集中的第二图片进行循环播放。
  17. 根据权利要求15所述计算机可读存储介质，其中，所述将所述第一连续帧图集中的第一图片从所述预设顺序的第一张开始按照所述预设顺序进行循环播放的步骤之后，包括：
    检测所述第一连续帧图集的循环播放时长，所述循环播放时长为从所述第一连续帧图集的起始播放时刻到当前时刻之间的时长；
    判断所述循环播放时长是否大于预设的时长；
    若是，则获取与所述第一连续帧图集对应的提示语，并进行显示。
  18. 根据权利要求17所述计算机可读存储介质，其中，所述获取与所述第一连续帧图集对应的提示语，并进行显示的步骤之后，包括：
    判断所述第一连续帧图集是否为预设的指定图集；
    若是，则获取与所述第一连续帧图集具有关联关系的第三连续帧图集，其中，所述关联关系为所述第一连续帧图集表示的心情类别与所述第三连续帧图集表示的心情类别相反的对应关系；
    将当前循环播放的所述第一连续帧图集替换成所述第三连续帧图集，并显示变更图集的警醒语。
  19. 根据权利要求15所述计算机可读存储介质，其中，所述获取视频中待识别的人脸图像的步骤，包括：
    获取视频中的多帧初始人脸图像，所述初始人脸图像为未经处理的图像；
    对所述初始人脸图像进行人脸检测，以确定人脸区域；
    获取所述人脸区域的特征点，并依据所述特征点对所述初始人脸图像进行对齐校准，得到所述待识别的人脸图像。
  20. 根据权利要求15所述计算机可读存储介质，其中，所述根据所述表情识别结果从预设的数据库获取对应表情类别的第一连续帧图集的步骤，包括：
    将所述表情识别结果与连续帧图集列表中的各个连续帧图集对应的映射值进行比较，以找到与所述表情识别结果一致的目标映射值，所述表情识别结果为所述表情识别模型输出的数值，所述连续帧图集列表包括多个对应不同表情类别的预设的连续帧图集，每个连续帧图集均包括指定数量的表示同一类别心情的图片，且均分别对应不同的映射值；
    依据所述目标映射值从所述连续帧图集列表中找出所述第一连续帧图集的关联名称，并依据所述关联名称从所述数据库中获取所述第一连续帧图集。
PCT/CN2020/098808 2019-07-03 2020-06-29 基于数据处理的图片显示方法、装置和计算机设备 WO2021000833A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910596485.8A CN110472488B (zh) 2019-07-03 2019-07-03 基于数据处理的图片显示方法、装置和计算机设备
CN201910596485.8 2019-07-03

Publications (1)

Publication Number Publication Date
WO2021000833A1 true WO2021000833A1 (zh) 2021-01-07

Family

ID=68507519

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098808 WO2021000833A1 (zh) 2019-07-03 2020-06-29 基于数据处理的图片显示方法、装置和计算机设备

Country Status (2)

Country Link
CN (1) CN110472488B (zh)
WO (1) WO2021000833A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114708627A (zh) * 2022-02-28 2022-07-05 厦门大学 一种应用于社交机器人的微表情识别方法
CN114708627B (zh) * 2022-02-28 2024-05-31 厦门大学 一种应用于社交机器人的微表情识别方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110472488B (zh) * 2019-07-03 2024-05-10 平安科技（深圳）有限公司 基于数据处理的图片显示方法、装置和计算机设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108009280A (zh) * 2017-12-21 2018-05-08 广东欧珀移动通信有限公司 图片处理方法、装置、终端及存储介质
US20180136615A1 (en) * 2016-11-15 2018-05-17 Roborus Co., Ltd. Concierge robot system, concierge service method, and concierge robot
CN109635680A (zh) * 2018-11-26 2019-04-16 深圳云天励飞技术有限公司 多任务属性识别方法、装置、电子设备及存储介质
CN109886697A (zh) * 2018-12-26 2019-06-14 广州市巽腾信息科技有限公司 基于表情组别的操作确定方法、装置及电子设备
CN109934115A (zh) * 2019-02-18 2019-06-25 苏州市科远软件技术开发有限公司 人脸识别模型的构建方法、人脸识别方法及电子设备
CN110472488A (zh) * 2019-07-03 2019-11-19 平安科技（深圳）有限公司 基于数据处理的图片显示方法、装置和计算机设备

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3065079A4 (en) * 2013-10-30 2017-06-21 Yulong Computer Telecommunication Scientific (Shenzhen) Co. Ltd. Terminal and method for managing video file

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180136615A1 (en) * 2016-11-15 2018-05-17 Roborus Co., Ltd. Concierge robot system, concierge service method, and concierge robot
CN108009280A (zh) * 2017-12-21 2018-05-08 广东欧珀移动通信有限公司 图片处理方法、装置、终端及存储介质
CN109635680A (zh) * 2018-11-26 2019-04-16 深圳云天励飞技术有限公司 多任务属性识别方法、装置、电子设备及存储介质
CN109886697A (zh) * 2018-12-26 2019-06-14 广州市巽腾信息科技有限公司 基于表情组别的操作确定方法、装置及电子设备
CN109934115A (zh) * 2019-02-18 2019-06-25 苏州市科远软件技术开发有限公司 人脸识别模型的构建方法、人脸识别方法及电子设备
CN110472488A (zh) * 2019-07-03 2019-11-19 平安科技（深圳）有限公司 基于数据处理的图片显示方法、装置和计算机设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114708627A (zh) * 2022-02-28 2022-07-05 厦门大学 一种应用于社交机器人的微表情识别方法
CN114708627B (zh) * 2022-02-28 2024-05-31 厦门大学 一种应用于社交机器人的微表情识别方法

Also Published As

Publication number Publication date
CN110472488A (zh) 2019-11-19
CN110472488B (zh) 2024-05-10

Similar Documents

Publication Publication Date Title
US10706873B2 (en) Real-time speaker state analytics platform
WO2020125386A1 (zh) 表情识别方法、装置、计算机设备和存储介质
US20200075007A1 (en) Voice interaction system, voice interaction method, program, learning model generation apparatus, and learning model generation method
CN113643047B (zh) 虚拟现实控制策略的推荐方法、装置、设备及存储介质
US20230177835A1 (en) Relationship modeling and key feature detection based on video data
CN109446968A (zh) 一种基于情绪调整学习情况的方法及系统
WO2024000867A1 (zh) 情绪识别方法、装置、设备及存储介质
KR102351008B1 (ko) 감정 인식 장치 및 감정 인식 방법
WO2019142127A1 (en) Method and system of creating multiple expression emoticons
US20210271864A1 (en) Applying multi-channel communication metrics and semantic analysis to human interaction data extraction
JP2016200969A (ja) 画像処理装置、画像処理方法、及びプログラム
CN115205764B (zh) 基于机器视觉的在线学习专注度监测方法、系统及介质
US10425578B1 (en) Image capturing assistant
WO2021000833A1 (zh) 基于数据处理的图片显示方法、装置和计算机设备
CN115206492A (zh) 基于眼动反馈的情绪识别能力自适应训练方法及其装置
WO2022193635A1 (zh) 客服服务系统、方法、装置、电子设备及存储介质
JP2020173787A (ja) 情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム
CN112149610A (zh) 一种目标对象的识别的方法和系统
CN114399827B (zh) 基于面部微表情的高校毕业生职业性格测试方法及系统
CN117795551A (zh) 用于自动捕捉和处理用户图像的方法和系统
US20230290505A1 (en) Context Aware Assessment
JP6285377B2 (ja) コミュニケーションスキル評価フィードバック装置、コミュニケーションスキル評価フィードバック方法及びコミュニケーションスキル評価フィードバックプログラム
WO2022100093A1 (zh) 一种视频字幕制作方法
KR101734212B1 (ko) 표정 연습 시스템
EP3799407B1 (en) Initiating communication between first and second users

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20834236

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01.06.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20834236

Country of ref document: EP

Kind code of ref document: A1