WO2017219450A1

WO2017219450A1 - Information processing method and device, and mobile terminal

Info

Publication number: WO2017219450A1
Application number: PCT/CN2016/093112
Authority: WO
Inventors: 郭辉
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-06-21
Filing date: 2016-08-03
Publication date: 2017-12-28
Also published as: CN107526994A

Abstract

Provided are an information processing method and device, and a mobile terminal. The method comprises: upon detection of a mobile terminal being operated, activating a camera to dynamically collect, according to a preset cycle, image information of a user; acquiring, according to the image information of the user, feature information of facial features of the user; and performing continuous comparative analyses on the feature information, and performing matching on the basis of pre-stored template feature information corresponding to the facial features to determine an emotion state of the user.

Description

Information processing method, device and mobile terminal

Technical field

This application relates to, but is not limited to, the field of communication technology.

Background technique

The facial recognition technology in the related art mainly uses a camera to collect facial information of a person, and uses facial detection or recognition technology to identify facial features of the user.

However, the facial emotion recognition in the related art is only a static recognition process. After the face information is collected by the camera, the facial features are extracted and compared with the preset feature values to obtain a similarity, thereby judging Out of user sentiment. This approach often has problems with lower accuracy and does not allow for real-time capture of changes in the user's facial features.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

This paper provides an information processing method, device and mobile terminal. In the process of static recognition, by dynamically analyzing the changes of the facial features of the user, it is possible to more accurately determine the emotional state of the user.

An information processing method includes:

When detecting that the user performs an operation on the mobile terminal, the startup camera dynamically collects the user image information according to the preset period;

Obtaining feature information of the facial feature part of the user according to the user image information;

The feature information is subjected to continuous comparative analysis, and matched with template feature information of the pre-stored corresponding facial feature parts to determine the emotional state of the user.

Optionally, the acquiring the feature information of the facial feature part of the user according to the user image information includes:

Determining a face area in the user image information by face detection;

Collecting facial feature information in the face area;

And distinguishing, according to the facial feature information, a facial feature part of the user, and extracting feature information corresponding to the facial feature part.

Optionally, the distinguishing the facial feature parts of the user according to the facial feature information, and extracting the feature information corresponding to the facial feature parts, including:

Distinguishing the mouth area of the user according to the facial feature information;

Obtaining a feature point location of the mouth region;

According to the feature point position, feature information of the mouth of the user is obtained.

Optionally, performing the continuous comparison analysis on the feature information, and matching the pre-stored template feature information of the corresponding facial feature part to determine the emotional state of the user, including:

Matching the feature information with the template feature information of the pre-stored corresponding facial feature part to obtain a matching result;

Comparing the feature information with the feature information of the corresponding facial feature portion acquired in the previous cycle to obtain an analysis result;

And determining an emotional state of the user according to the matching result and the analysis result.

Optionally, after the detecting that the user performs an operation on the mobile terminal, the method further includes:

Collect user voice information;

Determining the user's voice emotion according to the user voice information and the preset emotional sound template.

Optionally, the method further includes:

The emotional state of the user is verified in conjunction with the voice emotion.

Optionally, the method further includes:

The feature information of the determined emotional state of the user is saved in the template feature information corresponding to the facial feature portion.

Optionally, the method further includes:

Obtain a correspondence between a preset emotional state and a session template;

Determining, according to the correspondence, a session template corresponding to the determined emotional state of the user;

Initiating a session with the user according to the session template.

An information processing apparatus comprising:

The first processing module is configured to: when detecting that the user performs an operation on the mobile terminal, start the camera to dynamically collect user image information according to a preset period;

The first obtaining module is configured to: acquire feature information of the facial feature part of the user according to the user image information collected by the first processing module;

The second processing module is configured to: perform continuous comparative analysis on the feature information acquired by the first acquiring module, and match the pre-stored template feature information of the corresponding facial feature part to determine the emotional state of the user .

Optionally, the first obtaining module includes:

a determining unit, configured to: determine, by the face detection, a face area in the user image information collected by the first processing module;

The collecting unit is configured to: collect the facial feature information in the facial region determined by the determining unit;

The extracting unit is configured to: according to the facial feature information collected by the collecting unit, distinguish the facial feature part of the user, and extract feature information corresponding to the facial feature part.

Optionally, the extracting unit comprises:

a region molecular unit, configured to: distinguish the mouth region of the user according to the facial feature information collected by the collecting unit;

Obtaining a subunit, configured to: acquire a feature point position of the mouth region obtained by the molecular unit of the region;

The processing subunit is configured to: obtain feature information of the mouth of the user according to the feature point position acquired by the acquiring subunit.

Optionally, the second processing module includes:

a matching unit, configured to: the feature information acquired by the first acquiring module and the pre- The stored template feature information of the corresponding facial feature part is matched to obtain a matching result;

The analyzing unit is configured to compare and analyze the feature information acquired by the first acquiring module with the feature information of the corresponding facial feature part acquired in the previous cycle to obtain an analysis result;

The processing unit is configured to: determine the emotional state of the user according to the matching result obtained by the matching unit and the analysis result obtained by the analyzing unit.

Optionally, the information processing apparatus further includes:

The acquisition module is configured to: collect user voice information;

The third processing module is configured to: determine the voice mood of the user according to the user voice information collected by the collection module and a preset emotional sound template.

Optionally, the information processing apparatus further includes:

The verification module is configured to: verify the emotional state of the user in conjunction with the voice emotion determined by the third processing module.

Optionally, the information processing apparatus further includes:

The fourth processing module is configured to: save the feature information of the determined emotional state of the user in the template feature information corresponding to the facial feature part.

Optionally, the information processing apparatus further includes:

The second obtaining module is configured to: obtain a correspondence between the preset emotional state and the session template;

a selection module, configured to: select, according to the correspondence relationship acquired by the second obtaining module, a session template corresponding to the determined emotional state of the user;

The session initiation module is configured to initiate a session with the user according to the session template selected by the selection module.

A mobile terminal comprising the information processing apparatus according to any of the above.

The information processing method and device and the mobile terminal provided by the embodiment of the present invention acquire user image information according to a preset period by automatically starting the camera when detecting that the user performs an operation on the mobile terminal; and then acquiring the user according to the user image information. The feature information of the facial feature part; since the camera is dynamically collected, the user image information at different time points can be collected. Therefore, the feature information corresponding to the acquired facial feature part of the user is also the feature information at different time points; After that, the feature information can be continuously compared and analyzed, and matched with the pre-stored template feature information of the corresponding facial feature part to determine the emotional state of the user. In the embodiment of the present invention, not only the static processing of the acquired feature information of the facial feature part and the pre-stored template feature information is performed, but also the feature information of the facial feature part is dynamically processed, and the feature information of the facial feature part is realized. The change more accurately analyzes the user's emotional state and gives the user a better experience.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

FIG. 1 is a flowchart of an information processing method according to an embodiment of the present invention;

2 is a flowchart of another information processing method according to an embodiment of the present invention;

FIG. 3 is a flowchart of a method for performing step 123 in the information processing method provided by the embodiment shown in FIG. 2;

4 is a flowchart of still another information processing method according to an embodiment of the present invention;

FIG. 5 is a flowchart of still another information processing method according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an information processing apparatus according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of another information processing apparatus according to an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of still another information processing apparatus according to an embodiment of the present invention.

Embodiments of the invention

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments herein may be arbitrarily combined with each other.

The steps illustrated in the flowchart of the figures may be executed in a computer system in accordance with a set of computer executable instructions. Also, although logical sequences are shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.

The embodiment of the present invention is directed to the process of recognizing the user's emotions in the face of the related art, which is only a process of static recognition, and cannot capture the change of the facial features of the user in real time, and the problem of low accuracy is mentioned. For an information processing method, in the process of static recognition, by dynamically analyzing changes in the facial features of the user, the emotional state of the user is more accurately determined.

As shown in FIG. 1 , which is a flowchart of an information processing method according to an embodiment of the present invention, the information processing method provided in this embodiment may include the following steps: Step 110 to Step 130:

Step 110: When detecting that the user performs an operation on the mobile terminal, the startup camera dynamically collects user image information according to a preset period;

Step 120: Acquire feature information of a facial feature part of the user according to the user image information;

Step 130: Perform continuous comparative analysis on the feature information, and match the pre-stored template feature information of the corresponding facial feature part to determine the emotional state of the user.

The information processing method provided by the embodiment of the present invention is applied to a mobile terminal, where the feature information of a facial feature part (such as an eye, a mouth, etc.) of different emotions (such as laughing, crying, angry, etc.) can be stored in advance as Template feature information. Taking the mouth as an example, the pre-stored feature information may include the curvature of the corner of the mouth (for example, rising or falling) and the height difference of the upper and lower lip feature points obtained by constructing the coordinate system with the left and right corner feature points and the upper and lower lip feature points, corresponding to different The mood, the angular range of the mouth, and the height difference between the upper and lower lip feature points may have a certain preset threshold. The feature information is stored in the database according to the template feature information of different emotions in advance for subsequent matching.

When the information processing method provided by the embodiment of the present invention detects that the user performs an operation on the mobile terminal, that is, when the user is using the mobile terminal, the mobile terminal automatically starts the camera (for example, adopts a front camera) to collect according to a preset period. User image information; then, according to the user image information, acquiring feature information of the user's facial feature part; since the camera is dynamically collected, the user image information at different time points can be collected, and therefore, the acquired facial features of the user The feature information of the part is also feature information at different time points; subsequently, the feature information can be continuously compared and analyzed, and matched with the template feature information of the pre-stored corresponding facial feature part to determine the emotional state of the user.

Since the facial expression of a person has a process of forming a change, if the state obtained by only a certain moment cannot accurately reflect the current emotional feature of the user, the information processing method provided by the embodiment of the present invention not only acquires the facial feature acquired. The feature information of the part and the pre-stored template feature information The static processing of line matching also dynamically processes the feature information (ie, continuous contrast analysis), which realizes more accurate analysis of the user's emotional state through the change of the feature information of the facial feature parts, and brings a better experience to the user.

Considering that the user image information should be captured as much as possible when comparing and analyzing the feature information, the user's expression change can be clearly understood, so it is necessary to reduce the time interval at which the camera collects the user image information. Optionally, the preset period in the embodiment of the present invention is, for example, less than or equal to 1 second (s); on the other hand, in order to prevent the collected user image information from being processed in time, each collected user may be collected. The image information is added to the collection queue, so that the collected user image information is subsequently extracted one by one from the collection queue for processing.

Optionally, in the embodiment of the present invention, the application scenario in which the user performs operations on the mobile terminal includes, but is not limited to, the following scenario: the user lights the screen through the power button, unlocks the mobile phone, and detects that the screen is touched by the user. In the state where the mobile terminal is not operated, in order to avoid power consumption caused by the camera being always turned on in the background, there is no need to start.

Optionally, as shown in FIG. 2, a flowchart of another information processing method according to an embodiment of the present invention. On the basis of the embodiment shown in FIG. 1, the step 120 in this embodiment may include the following steps. That is, steps 121 to 123:

In step 121, the face area is determined in the user image information by face detection.

In the embodiment of the present invention, the user image information collected after the camera is started may include not only a facial image but also other body part images and/or background images, and the user image information collected at a certain time point. The facial image may not be included. Therefore, in order to ensure the validity of the finally obtained feature information, in this step, the user image information is extracted one by one from the collection queue, and the face region is determined through face detection to avoid other uselessness. Interference caused by regional image information.

Step 122, collecting facial feature information in the face area.

In this step, the collection of facial feature information is performed in the face region determined in step 121, wherein the facial feature information may include information such as the position, region, and the like of each facial organ. In practical applications, facial feature information can be obtained by facial recognition, shape description of facial organs, and distance characteristics between each facial organ, and can also be based on algebraic features or statistical learning characterization. The method obtains facial feature information, which will not be described here.

Step 123: Differentiate the facial feature part of the user according to the facial feature information, and extract feature information corresponding to the facial feature part.

Since the facial feature information including the position, the area, and the like of each facial organ has been collected in step 122, in this step, the facial feature portion of the user, such as the eye and the mouth, can be distinguished according to the facial feature information. And correspondingly extracting feature information of the facial feature portion.

In a human facial expression, the characteristic change of the mouth is more apparent with respect to other parts when the emotion is different. Therefore, based on the above-described embodiment of the present invention, an information processing method based on the feature information corresponding to the mouth is provided.

Optionally, as shown in FIG. 3, a flowchart of a method for performing step 123 in the information processing method provided in the embodiment shown in FIG. In this embodiment, the foregoing step 123 may include the following steps, that is, steps 1231 to 1233:

Step 1231, distinguishing the mouth area of the user according to the facial feature information.

As can be seen from the above-described embodiments, the facial feature information includes information such as the position and region of each facial organ. Therefore, in this step, the mouth region of the user can be distinguished based on the facial feature information.

Step 1232, acquiring a feature point position of the mouth area.

In order to finally obtain the feature information of the mouth region, the feature points of the mouth region, such as the left and right mouth corners, and the feature points of the upper and lower lip vertices are set in advance. Therefore, in this step, corresponding features can be determined in the mouth region. Click to get the feature point location. In practical applications, in order to ensure the authenticity of the data, a preset number of sample positions can be selected between the feature point positions for auxiliary analysis.

Step 1233, according to the feature point position, obtaining feature information of the mouth of the user.

In this step, according to the left and right mouth corner positions, the upper and lower lip vertex positions, and the sample position assistance in the feature point position obtained in step 1232, the feature information of the user's mouth, such as the mouth angle curvature and the upper and lower lip feature points, is obtained. Geometric feature information such as height difference. In the implementation process, the coordinate system can be constructed according to the left and right mouth angular position, the upper and lower lip vertex positions, and the sample position, and the height difference between the mouth angle arc and the upper and lower lip feature points is calculated by the coordinates of each of the above positions.

The embodiment shown in FIG. 3 only takes the facial feature as the mouth as an example, and illustrates that the feature information corresponding to the facial feature is extracted. In addition, the human emotion is also revealed by the eye, and the facial feature may also be The eye area, the feature information corresponding to the eye area, can select the size of the pupil. Based on the extraction of the mouth feature information, the eye feature information can be combined for processing to obtain a more accurate analysis result. Of course, in addition to the mouth area and the eye area, it can be combined with other parts for processing, and will not be enumerated here.

Optionally, in the information processing method provided by the embodiment of the present invention, in order to improve the accuracy of the judgment of the user's emotional state, the formation process of the facial expression change of the person is considered, for example, for the step 130, the stored template feature information is When the matching is performed, the corresponding feature information at different time points can be continuously compared and analyzed. As shown in FIG. 4, it is a flowchart of another information processing method provided by the embodiment of the present invention, which is based on the foregoing embodiment. The step 130 in this embodiment may include the following steps, that is, steps 131 to 133; the embodiment shown in FIG. 4 is shown as an example on the basis of the embodiment shown in FIG. 1.

Step 131: Match the feature information with the pre-stored template feature information of the corresponding facial feature part to obtain a matching result.

In this step, the feature information of the facial feature part acquired in step 120 is extracted one by one, and the template feature information of the same facial feature part pre-stored in the database is matched, and the corresponding matching result can be obtained, thereby obtaining the matching result. The degree of matching of the feature information of the facial feature portion with each template feature information. Taking the characteristic information of the mouth as an example, the difference between the height of the stored mouth and the height difference of the upper and lower lip feature points in the database is a threshold range having a corresponding emotional state. For example, when the emotional state is laughing, the geometrical contour of the corner is Ascending, the height difference between the corner of the mouth and the feature points of the upper and lower lips are positive, and within a predetermined threshold range; the height difference between the current user's mouth angle and the upper and lower lip feature points is in the emotional state. When the template corresponds to the threshold range, the current user's mouth feature information and the emotional state are the highest match degree.

In step 132, the feature information is compared with the feature information of the corresponding facial feature part acquired in the previous cycle, and the analysis result is obtained.

In this step, the feature information of the facial feature part acquired in step 120 is compared with the feature information of the corresponding facial feature part acquired in the previous cycle according to the chronological order, and the corresponding information is obtained. The analysis results, to understand the changes in the characteristics of the information.

Step 133, determining the emotional state of the user according to the matching result and the analysis result.

In this step, the matching degree of the user's facial feature part and each template feature information and the change of the feature information can be known by combining the matching result and the analysis result, and the user's emotional state can be determined.

Taking the user image information at a certain time point as an example, after obtaining the current mouth feature information in step 120, through step 131, the matching result obtained by the matching is used to know the matching degree between the current mouth feature information and the emotional state of the user. highest. However, in the process of laughing, the curvature of the beginning of the mouth is constantly increasing, and then it will continue to become smaller. Therefore, by analyzing the trend of the curvature of the corner of the mouth, it is possible to determine the change of the user's emotional state and find that the change or Increasingly large, indicating that the user is laughing; the discovery is getting smaller, indicating that the emotional state of the user's laugh is coming to an end. In addition to the curvature of the corner of the mouth, the analysis can be combined with the change of the height difference of the upper and lower lip feature points to more accurately complete the contrast analysis. For example, during the opening and closing of the mouth, the height difference between the upper and lower lip feature points is constantly changing. With the present embodiment, not only the type of the user's emotional state can be determined, but also the changing phase of the emotional state can be determined, thereby improving the accuracy of the information processing to a greater extent.

Optionally, the information processing method of the embodiment of the present invention may further include: saving feature information of the determined emotional state of the user in template feature information corresponding to the facial feature part, in consideration of the applicability of the mobile terminal to the user.

Through this step, the mobile terminal performs self-learning, stores feature information of the facial feature part after determining the user's emotional state, and fills the template feature information of the facial feature part in the database, so that personalized data is established according to the user. Information, in the subsequent information processing process will also get a higher degree of matching, recognition is faster and more accurate.

Optionally, in the information processing method of the embodiment of the present invention, after detecting that the user performs an operation on the mobile terminal, the method may further include the following steps:

Collect user voice information;

The user's voice emotion is determined according to the user voice information and the preset emotional sound template.

Through the above steps, in addition to the analysis of the emotion of the face, the embodiment of the present invention can also be combined with the voice assisted analysis. Starting the user's voice after detecting that the user performs an operation on the mobile terminal The information can be collected by opening the call microphone to collect the user's voice information, and then identifying the emotional sound template (such as laughter, crying, snoring, etc. with emotional state) of the sound information of different emotional states pre-stored by the mobile terminal. To determine the user's voice emotions.

Optionally, in the embodiment of the present invention, the method further includes: verifying the emotional state of the user in combination with the voice emotion.

The emotional state determined by the feature information of the facial feature portion of the current user is verified by the recognized sound emotion, and the real emotional state of the user is comprehensively analyzed.

Mobile terminals have become an important part of people's daily lives, but they are just a physical tool, an "entry" to external information, but very boring. Optionally, in the embodiment of the present invention, after determining the emotional state of the user, the method may further include:

Initiate a session with the user based on the session template.

Through the above steps, after accurately determining the emotional state of the user, the mobile terminal may also select an appropriate session template according to the emotional state of the user, and automatically initiate the voice function to initiate a dialogue with the user, so that the user has a better user experience and moves. The terminal is not only a physical tool, but also a "friend" of the chat.

In an actual application, after determining the emotional state of the user, in addition to starting the adapted voice conversation, other functions, such as playing of music, etc., may be performed, and are not enumerated here.

Optionally, as shown in FIG. 5, which is a flowchart of still another information processing method according to an embodiment of the present invention, this embodiment provides an application example of a mobile terminal, and the method may include the following steps, that is, steps 501 to 511:

Step 501: In the normal standby sleep state of the mobile terminal, detecting whether the user performs an operation on the mobile terminal, if yes, executing step 502, and if not, continuing to detect;

Step 502: automatically start the front camera and the microphone in the background, respectively collect the user image information and the sound information, and add the collected information to the collection information queue; wherein the user image information is collected according to a preset period, for example, the preset period is 0.5s;

Step 503: Acquire current user image information from the collection information queue, and perform face detection. Determining a facial area to obtain facial feature information of the user;

Step 504: Extract feature information of the mouth region from the facial feature information by using face recognition;

Step 505, the feature information of the mouth area and the template feature information of the mouth area pre-stored by the mobile terminal are matched one by one;

Step 506: Find an emotional state with the highest matching degree according to the matching result.

Step 507, comparing the feature information of the current mouth region with the feature information of the mouth region of the user image information of the previous cycle; if the corresponding emotional states of the two are different, performing step 511, if the emotional states of the two are corresponding Similarly, step 508 is performed;

Step 508: Initially determine the emotional state of the user according to the current mouth angle radians and the difference between the height difference of the upper and lower lip feature points and the previous period;

Step 509: Perform auxiliary analysis according to the collected sound information to determine a current emotional state of the user;

Step 510: Start a voice function, and start an adaptive session according to the current emotional state of the user;

Step 511: The feature information of the determined emotional state is stored as new template feature information in the template feature information of the feature part of the corresponding part.

In summary, the information processing method provided by the embodiment of the present invention automatically detects the user when performing operations on the mobile terminal, that is, when the user is using the mobile terminal, the mobile terminal automatically starts the camera (for example, adopting a front camera) The user image information is collected according to the preset period; then, according to the user image information, the feature information of the user's facial feature part is acquired; since the camera is dynamically collected, the user image information at different time points can be collected, and accordingly, the corresponding information is obtained. The feature information of the facial feature portion of the user is also feature information at different time points; subsequently, the feature information can be continuously compared and analyzed, and matched with the template feature information of the pre-stored corresponding facial feature portion to determine the user. The emotional state. In the embodiment of the present invention, not only the static processing of the acquired feature information of the facial feature part and the pre-stored template feature information is performed, but also the feature information of the facial feature part is dynamically processed, and the feature information of the facial feature part is realized. The change more accurately analyzes the user's emotional state and gives the user a better experience.

FIG. 6 is a schematic structural diagram of an information processing apparatus according to an embodiment of the present invention. The information processing apparatus provided in this embodiment may include: a first processing module 10 and a first acquiring module 20; And a second processing module 30.

The first processing module 10 is configured to: when detecting that the user performs an operation on the mobile terminal, start the camera to dynamically collect user image information according to a preset period;

The first obtaining module 20 is configured to: acquire feature information of a facial feature part of the user according to the user image information collected by the first processing module 10;

The second processing module 30 is configured to perform continuous comparative analysis on the feature information acquired by the first acquiring module 20, and match the template feature information of the pre-stored corresponding facial feature parts to determine the emotional state of the user.

Optionally, FIG. 7 is a schematic structural diagram of another information processing apparatus according to an embodiment of the present invention. Based on the structure of the apparatus shown in FIG. 6, the first obtaining module 20 in this embodiment may include: a determining unit 21, an obtaining unit 22, and an extracting unit 23.

The determining unit 21 is configured to: determine, by the face detection, the face area in the user image information collected by the first processing module 10;

The collecting unit 22 is configured to: collect the facial feature information in the face region determined by the determining unit 21;

The extracting unit 23 is configured to: according to the facial feature information collected by the collecting unit 22, distinguish the facial feature part of the user, and extract feature information corresponding to the facial feature part.

Optionally, in the embodiment of the present invention, the foregoing extracting unit 23 may include:

The regional molecular unit is configured to: distinguish the mouth region of the user according to the facial feature information collected by the collecting unit 22;

Obtaining a subunit, which is set to: acquire a feature point position of a mouth region obtained by the molecular unit of the region;

The processing subunit is configured to obtain the feature information of the user's mouth according to the position of the feature point acquired by the acquiring subunit.

Optionally, FIG. 8 is a schematic structural diagram of still another information processing apparatus according to an embodiment of the present invention. On the basis of the foregoing embodiment, the second processing module 30 in this embodiment may include: a matching unit 31, an analyzing unit 32, and a processing unit 33; the embodiment shown in FIG. 8 is based on the device in the embodiment shown in FIG. The above is shown as an example.

The matching unit 31 is configured to: match the feature information acquired by the first acquiring module 20 with the template feature information of the pre-stored corresponding facial feature part to obtain a matching result;

The analyzing unit 32 is configured to compare and analyze the feature information acquired by the first acquiring module 20 and the feature information of the corresponding facial feature part acquired in the previous cycle to obtain an analysis result;

The processing unit 33 is configured to determine the emotional state of the user according to the matching result obtained by the matching unit 31 and the analysis result obtained by the analyzing unit 32.

Optionally, in the embodiment of the present invention, the information processing apparatus may further include:

The acquisition module is configured to: collect user voice information;

The third processing module is configured to: determine a user's voice emotion according to the user voice information collected by the collection module and the preset emotional sound template.

The selection module is configured to: select a session template corresponding to the determined emotional state of the user according to the correspondence acquired by the second obtaining module;

When the information processing apparatus provided by the embodiment of the present invention detects that the user performs an operation on the mobile terminal, that is, when the user is using the mobile terminal, the first processing module automatically starts the camera (for example, adopts a front camera) according to a preset period. Collecting user image information; then, the first obtaining module acquires feature information of the facial feature part of the user according to the user image information; since the camera is dynamically collected, the user image information at different time points can be collected, and accordingly, the corresponding image is acquired. of The feature information of the facial feature portion of the user is also feature information at different time points; subsequently, the feature information can be continuously compared and analyzed by the second processing module, and the template feature information of the pre-stored corresponding facial feature portion is performed. Match to determine the emotional state of the user. In the embodiment of the present invention, not only the static processing of the acquired feature information of the facial feature part and the pre-stored template feature information is performed, but also the feature information is dynamically processed, thereby realizing the change of the feature information of the facial feature part. Accurately analyze the emotional state of the user and bring a better experience to the user.

The information processing device provided by the embodiment of the present invention is the device to which the information processing method is applied, and the implementation manner of the embodiment of the information processing method is applicable to the device provided by the embodiment of the present invention, and the same technical effect can be achieved.

The embodiment of the present invention further provides a mobile terminal, comprising: the information processing apparatus provided by any of the foregoing embodiments.

When the mobile terminal provided by the embodiment detects that the user performs an operation on itself, that is, when the user is using the user, the mobile terminal automatically starts the camera (for example, adopts a front camera) to collect user image information according to a preset period; Then, according to the user image information, the feature information of the facial feature part of the user is acquired; since the camera is dynamically collected, the user image information at different time points can be collected, and therefore, the feature information of the acquired facial feature part of the user is correspondingly The feature information is also at different time points; subsequently, the feature information can be continuously compared and analyzed, and matched with the template feature information of the pre-stored corresponding facial feature parts to determine the emotional state of the user. In the embodiment of the present invention, not only the static processing of the acquired feature information of the facial feature part and the pre-stored template feature information is performed, but also the feature information is dynamically processed, thereby realizing the change of the feature information of the facial feature part. Accurately analyze the emotional state of the user and bring a better experience to the user.

The embodiment of the present invention provides a mobile terminal to which the mobile terminal is applied. The implementation manner of the foregoing information processing method is applicable to the mobile terminal, and the same technical effect can be achieved.

It should be noted that the mobile terminals described in the specification herein include, but are not limited to, a smartphone, a tablet, etc., and many of the functional components described are referred to as modules to more particularly emphasize the independence of their implementation.

In an embodiment of the invention, the modules may be implemented in software for execution by various types of processors. For example, an identified executable code module can comprise one or more physical or logical blocks of computer instructions, which can be constructed, for example, as an object, procedure, or function. Nonetheless, the executable code of the identified modules need not be physically located together, but may include different instructions stored in different bits that, when logically combined, constitute a module and implement the provisions of the module. purpose.

In practical applications, the above executable code modules may be a single instruction or a plurality of instructions, and may even be distributed over a plurality of different code segments, distributed among different programs, and distributed across multiple memory devices. Additionally, operational data may be identified within the modules and may be implemented in any suitable form and organized within any suitable type of data structure. The above operational data may be collected as a single data set or may be distributed at different locations, for example, on different storage devices, and at least in part may exist as an electronic signal only on the system or network.

On the basis that the module can be implemented by software, considering the level of the hardware process in the related technology, the module that can be implemented by software can be constructed by a person skilled in the art without corresponding consideration of cost. The hardware circuit includes conventional Very Large Scale Integration (VLSI) circuits or gate arrays and semiconductors such as logic chips, transistors, or other discrete components. The above modules can also be implemented with programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, and the like.

One of ordinary skill in the art will appreciate that all or a portion of the steps of the above-described embodiments can be implemented using a computer program flow, which can be stored in a computer readable storage medium on a corresponding hardware platform (according to The system, device, device, device, etc. are executed, and when executed, include one or a combination of the steps of the method embodiments.

Alternatively, all or part of the steps of the above embodiments may also be implemented by using an integrated circuit. These steps may be separately fabricated into individual integrated circuit modules, or multiple modules or steps may be fabricated into a single integrated circuit module. achieve.

The devices/function modules/functional units in the above embodiments may be implemented by a general-purpose computing device, which may be centralized on a single computing device or distributed over a network of multiple computing devices.

The device/function module/function unit in the above embodiment is implemented in the form of a software function module and When sold or used as a stand-alone product, it can be stored on a computer readable storage medium. The above mentioned computer readable storage medium may be a read only memory, a magnetic disk or an optical disk or the like.

Industrial applicability

In the embodiment of the present invention, when detecting that the user performs an operation on the mobile terminal, the user image information is collected according to a preset period by automatically starting the camera; and then, according to the user image information, the feature information of the facial feature portion of the user is acquired; Dynamic acquisition, that is, user image information at different time points can be collected. Therefore, the feature information corresponding to the acquired facial feature parts of the user is also feature information at different time points; subsequently, the feature information can be continuously compared. The analysis is performed and matched with the pre-stored template feature information of the corresponding facial feature portion to determine the emotional state of the user. In the embodiment of the present invention, not only the static processing of the acquired feature information of the facial feature part and the pre-stored template feature information is performed, but also the feature information of the facial feature part is dynamically processed, and the feature information of the facial feature part is realized. The change more accurately analyzes the user's emotional state and gives the user a better experience.

Claims

An information processing method includes:

When detecting that the user performs an operation on the mobile terminal, the startup camera dynamically collects the user image information according to the preset period;

Obtaining feature information of the facial feature part of the user according to the user image information;

The feature information is subjected to continuous comparative analysis, and matched with template feature information of the pre-stored corresponding facial feature parts to determine the emotional state of the user.
The information processing method according to claim 1, wherein the acquiring the feature information of the facial feature of the user according to the user image information comprises:

Determining a face area in the user image information by face detection;

Collecting facial feature information in the face area;

And distinguishing, according to the facial feature information, a facial feature part of the user, and extracting feature information corresponding to the facial feature part.
The information processing method according to claim 2, wherein the distinguishing the facial feature parts of the user according to the facial feature information and extracting the feature information corresponding to the facial feature parts comprises:

Distinguishing the mouth area of the user according to the facial feature information;

Obtaining a feature point location of the mouth region;

According to the feature point position, feature information of the mouth of the user is obtained.
The information processing method according to claim 1, wherein the feature information is continuously compared and analyzed, and matched with template feature information of a pre-stored corresponding facial feature portion to determine an emotional state of the user. include:

Matching the feature information with the template feature information of the pre-stored corresponding facial feature part to obtain a matching result;

Comparing the feature information with the feature information of the corresponding facial feature portion acquired in the previous cycle to obtain an analysis result;

And determining an emotional state of the user according to the matching result and the analysis result.
The information processing method according to claim 1, wherein after the detecting that the user performs an operation on the mobile terminal, the method further comprises:

Collect user voice information;

Determining the user's voice emotion according to the user voice information and the preset emotional sound template.
The information processing method according to claim 5, further comprising:

The emotional state of the user is verified in conjunction with the voice emotion.
The information processing method according to claim 1, further comprising:

The feature information of the determined emotional state of the user is saved in the template feature information corresponding to the facial feature portion.
The information processing method according to claim 1, further comprising:

Obtain a correspondence between a preset emotional state and a session template;

Determining, according to the correspondence, a session template corresponding to the determined emotional state of the user;

Initiating a session with the user according to the session template.
An information processing apparatus comprising:

The first processing module is configured to: when detecting that the user performs an operation on the mobile terminal, start the camera to dynamically collect user image information according to a preset period;

The first obtaining module is configured to: acquire feature information of the facial feature part of the user according to the user image information collected by the first processing module;

The second processing module is configured to: perform continuous comparative analysis on the feature information acquired by the first acquiring module, and match the pre-stored template feature information of the corresponding facial feature part to determine the emotional state of the user .
The information processing device according to claim 9, wherein the first acquisition module comprises:

a determining unit, configured to: determine, by the face detection, a face area in the user image information collected by the first processing module;

The collecting unit is configured to: collect the facial feature information in the facial region determined by the determining unit;

The extracting unit is configured to: according to the facial feature information collected by the collecting unit, distinguish the facial feature part of the user, and extract feature information corresponding to the facial feature part.
A mobile terminal comprising: the information processing apparatus according to claim 9 or 10.