JP2024028023A - Facial expression processing device, facial expression processing method, and facial expression processing program - Google Patents

Abstract

The present invention provides an expression processing program, an expression processing device, and an expression processing method for appropriately processing a user's expression in an image. The facial expression processing program of the present disclosure includes an emotion input step of receiving emotional information input from a user, a first image including the user's face, and the emotional information, and a facial expression processing program based on the emotional information. A computer is caused to execute an expression processing method including an image processing step of generating a second image from the first image by processing the facial expression of the user in the first image. [Selection diagram] Figure 3

Description

The present disclosure relates to an expression processing device, an expression processing method, and an expression processing program.

When conducting an online meeting using a computer's built-in camera or an external webcam, the amount of nonverbal information conveyed is smaller than in a face-to-face meeting, so small changes in facial expressions are difficult to communicate. It's difficult to tell. The same applies to streaming distribution. Therefore, in online meetings and streaming distribution, it is possible for the other party to receive information that you did not intend.

In addition, in recent years, online conferences and streaming distribution using avatars have also been conducted. Similarly to the above, in videos using avatars, the amount of nonverbal information transmitted is small, so it is difficult to convey minute changes in facial expressions to the other person.

Furthermore, an information processing device is known that aims to improve the emotional state of a user in a captured image. In this information processing device, for example, a user's smile level included in a captured image is measured, and the captured image is processed to have a smile level one level higher and output.

JP 2017-182594 Publication

The above-mentioned information processing device specializes in facial expressions such as smiling faces, and aims to improve the emotional state of the user in the captured image. However, since the above-mentioned information processing apparatus is not a device that has sufficiently focused on improving communication, it has not been considered to handle facial expressions other than smiling faces.

Therefore, the present disclosure provides a facial expression processing program, a facial expression processing device, and a facial expression processing method that appropriately process a user's facial expression in an image.

The facial expression processing program according to the first aspect of the present disclosure includes an emotion input step of receiving emotional information from a user, a first image including the user's face, and the emotional information, and a facial expression processing program based on the emotional information. A computer is caused to execute an expression processing method including an image processing step of generating a second image from the first image by processing the facial expression of the user in the first image. As a result, for example, the user can use this program to compensate for his or her own lacking emotional expression skills, and can prevent the user from conveying content that is not intended by the user to the other party.

The first aspect may further include an avatar generation step of generating an avatar, which is a character to be displayed as an alter ego of the user in a virtual space on a network, based on the second image. As a result, for example, by communicating through avatars with rich facial expressions, users can communicate their intentions without discrepancies in online meetings.

Further, in this first aspect, the facial expression of the avatar may be generated to correspond to the facial expression of the user in the second image. As a result, for example, by communicating through avatars with rich facial expressions, users can communicate their intentions without discrepancies in online meetings.

Further, in this first aspect, the image processing step may process the facial expression of the user in the first image into an expression corresponding to the emotional information. As a result, for example, the user can use this program to compensate for his or her own lacking emotional expression skills, and can prevent the user from conveying content that is not intended by the user to the other party.

Further, in this first aspect, the image processing step acquires a teacher image including a human facial expression, and compares the first image and the teacher image to obtain a The facial expression of the user may be processed. This makes it possible, for example, to analyze the user's facial movements that are insufficient to express emotions.

Moreover, in this first aspect, the image processing step analyzes the user's emotion from the first image, and based on the result of the analysis, the emotion information, and the result of the comparison, The facial expression of the user in one image may be processed. Based on this comparison result, by processing the image, the user can convey emotions correctly.

Moreover, in this first aspect, the first image may be an image captured by an imaging device. This makes it possible, for example, to accurately detect line of sight and analyze emotions.

Further, in this first aspect, the imaging device may be an RGB imaging device or an RGBIR imaging device. This makes it possible, for example, to accurately detect line of sight and analyze emotions.

Further, in this first aspect, the step further includes a scene input step of receiving scene information input from the user, and the image processing step receives the first image, the emotional information, and the scene information, and the image processing step receives the first image, the emotional information, and the scene information, and The second image may be generated from the first image by processing the facial expression of the user in the first image based on emotional information and the scene information. Thereby, for example, it is possible to output an image in which the user's facial expression has been processed to match the atmosphere of the scene. Therefore, the user can express the emotions he/she wants to express more accurately and in a way that can be conveyed to the other party, and communication can be further improved.

Further, in this first aspect, the image processing method may further include a feedback input step of receiving feedback input regarding the second image from the user, and the image processing step may perform learning based on the feedback. This allows, for example, an image obtained by processing an avatar or user's facial expression based on feedback input by the user to be reprocessed to achieve the optimal expression, allowing the user to express the image they want to express. Emotions can be expressed more accurately and conveyed to the other person, and communication can be further improved.

Further, in this first aspect, the second image may be output as an output image in an online conference. As a result, for example, rich facial expressions can be felt even through the screen, so the user can smoothly encourage the other user to change the conversation, and communication can proceed smoothly.

Further, in this first aspect, the second image may be output to the user's display device or the other party's display device. As a result, for example, rich facial expressions can be felt even through the screen, so that the user can smoothly encourage the other user to take turns in speaking and facilitate communication.

Further, in this first aspect, the avatar may be output as an output image in an online conference. As a result, for example, by talking to the other user through an avatar that conveys emotions, the user can create a sense of intimacy with the other user even if the other user does not know the other user.

Moreover, in this first aspect, the avatar may be output to the display device of the user or the display device of the other user. As a result, for example, by talking to the other user through an avatar that conveys emotions, the user can create a sense of intimacy with the other user even if the other user does not know the other user.

In addition, in this first aspect, the image processing step further includes an emotional information acquisition step of acquiring emotional information of the other user, and the image processing step includes the first image, the emotional information, and the emotional information of the other user. and generates the second image from the first image by processing the facial expression of the user in the first image based on the emotional information and the emotional information of the other user. You can. Thereby, for example, the user can express the emotion he/she wants to express more accurately and in a way that is conveyed to the other party according to the emotion of the other user, and communication can be further improved.

Further, in this first aspect, the emotional information obtaining step includes receiving a third image including the face of the other user from the other user, and based on the third image, emotional information of the other user. may be obtained. Thereby, for example, the user can express the emotion he/she wants to express more accurately and in a way that is conveyed to the other party according to the emotion of the other user, and communication can be further improved.

Moreover, in this first aspect, the emotional information acquisition step may acquire the emotional information of the other user by analyzing the emotions of the other user from the third image. Thereby, for example, the user can express the emotion he/she wants to express more accurately and in a way that is conveyed to the other party according to the emotion of the other user, and communication can be further improved.

Moreover, in this first aspect, the third image may be an image captured by an imaging device. This makes it possible, for example, to accurately detect line of sight and analyze emotions.

A facial expression processing device according to a second aspect of the present disclosure includes an emotion input unit that receives input of emotional information from a user, a first image including the user's face, and the emotional information, and receives an input of emotional information from a user based on the emotional information. and an image processing unit that generates a second image from the first image by processing the facial expression of the user in the first image. As a result, for example, the user can use this device to compensate for his or her own lacking emotional expression skills, and can prevent the user from conveying content that is not intended by the user to the other party.

A facial expression processing method according to a third aspect of the present disclosure includes an emotion input step of receiving an input of emotional information from a user, receiving a first image including the user's face, and the emotional information, and processing based on the emotional information. and an image processing step of generating a second image from the first image by processing facial expressions of the user in the first image. As a result, for example, the user can use this method to compensate for his or her lacking emotional expression skills, and can prevent the user from conveying content that is not intended by the user to the other party.

An example of a system configuration diagram when holding an online conference using the facial expression processing program in the first embodiment An example of a system configuration diagram when performing streaming distribution using the facial expression processing program in the first embodiment System block diagram of an information processing device in which the facial expression processing program in the first embodiment is installed Flowchart of the facial expression processing program in the first embodiment Example of displaying an image including the user's face captured by the imaging device 2 on the display device 4' Example of displaying an image processed with the user's facial expression on the display device 4' in the first embodiment Example of displaying an avatar corresponding to a user with “no expression” on the display device 4’ Example of displaying the avatar in the first embodiment on the display device 4' System block diagram of an information processing device installed with an expression processing program in the second embodiment Flowchart of the facial expression processing program in the second embodiment Example of displaying an image processed with the user's facial expression on the display device 4' in the second embodiment Example of displaying the avatar on the display device 4' in the second embodiment System block diagram of an information processing device installed with an expression processing program according to the third embodiment Flowchart of the facial expression processing program in the third embodiment System block diagram of an information processing device installed with an expression processing program in the fourth embodiment Flowchart of facial expression processing program in the fourth embodiment An example of the hardware configuration of the information processing device 1 in the fifth embodiment

Embodiments of the present disclosure will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is an example of a system configuration diagram when holding an online conference using the facial expression processing program according to the first embodiment.

FIG. 1 shows an information processing device 1 used by users participating in an online conference, and an imaging device 2 of this information processing device 1. A user participating in an online conference can, for example, install and execute a facial expression processing program on the information processing device 1. An example of the information processing device 1 is a computer device such as a PC (Personal Computer). Examples of the imaging device 2 include an RGB imaging device including pixels for red light (R), green light (G), and blue light (B); ) and pixels for infrared light (IR). The imaging device 2 is, for example, a camera or an image sensor. The imaging device 2 may be an external device connected to the information processing device 1, or may be a built-in device integrated with the information processing device 1.

FIG. 1 further shows an information processing device 1' used by a user on the other side participating in the online conference, and an imaging device 2' of this information processing device 1'. In FIG. 1, in order to make the explanation easier to understand, the information processing device of the other user is expressed as "information processing device 1'", and the imaging device of the other user is expressed as "imaging device 2'". The details of the information processing device 1' and the imaging device 2' are the same as the information processing device 1 and the imaging device 2. Note that the above online conference may be held by three or more users using three or more information processing devices.

Further details of the information processing device 1 and the imaging device 2 shown in FIG. 1 will be described below, but the following description is also applicable to the information processing device 1' and the imaging device 2'.

The facial expression processing program is a computer program for processing the facial expression of a user in an image. The facial expression processing program receives an image including the user's face from the imaging device 2, and also receives emotional information input from the user. Furthermore, the facial expression processing program processes the facial expression of the user in the image based on the emotional information. The image captured by the imaging device 2 is an example of the first image of the present disclosure. Further, an image obtained by processing the user's facial expression is an example of the second image of the present disclosure. Details of facial expression processing will be described later. Further, the facial expression processing program does not need to be used by all users who hold online meetings, and may be used only by users who wish to process their own facial expressions. For example, the facial expression processing program may be installed in each of the information processing device 1 and the information processing device 1', or may be installed only in the information processing device 1.

The image created by the facial expression processing program can be used as an input image in, for example, an online conference program. Instead of accepting an image captured by the imaging device 2 as an input image, the online conference program can accept an image generated by a facial expression processing program as an input image. In this case, by outputting images generated by the facial expression processing program to each user, facial expression processing in an online conference can be realized. Further, the facial expression processing program may be implemented as part of the functions of an online conference program. For example, in an online conference program, it is conceivable to generate images obtained by processing users' facial expressions and output them to each user.

In this online conference, the information processing device 1 transmits an image with processed facial expressions to the information processing device 1' via the network 100. The network 100 may be configured as either a wired network or a wireless network, and various networks 100 are constructed depending on the purpose of using the facial expression processing program.

FIG. 2 is an example of a system configuration diagram when performing streaming distribution using the facial expression processing program in the first embodiment. The system shown in FIG. 2 will be explained focusing on the differences from the system shown in FIG. 1.

A user who is a distributor can, for example, install and execute a facial expression processing program on the information processing device 1. FIG. 1 shows one information processing device 1 used by a distributor and an imaging device 2 of this information processing device 1. In this streaming distribution, the distributor transmits an image with processed facial expressions to the streaming distribution server 300 via the network 100. The streaming distribution server 300 can stream-distribute this image to the information processing device group 500 including the plurality of information processing devices 1'. Each information processing device 1' of the information processing device group 500 is used by a user who is a viewer.

An image created by the facial expression processing program can be used as an input image in a streaming distribution program, for example. The relationship between the facial expression processing program and the program for streaming distribution is the same as the relationship between the facial expression processing program and the online conference program described above.

The system configuration in the first embodiment is not limited to the configuration shown in FIG. 1 or 2, and can be configured in various ways depending on the purpose of using the facial expression processing program. For example, communication is not limited to one-to-one communication, but can be performed by one-to-N information processing apparatuses 1 (N is an integer of 1 or more). Hereinafter, in order to make the explanation easier to understand, an example in which one-to-one communication is performed will be described, but the following explanation is also applicable to a case in which one-to-N communication is performed.

FIG. 3 is a system block diagram of the information processing device 1 in which the facial expression processing program according to the first embodiment is installed.

FIG. 3 shows an information processing device 1 used by a user participating in an online conference, and an imaging device 2, an input device 3, and a display device 4 of this information processing device 1. FIG. 3 further shows an information processing device 1' used by a user on the other side participating in this online conference, and an imaging device 2', an input device 3', and a display device 4' of this information processing device 1'. There is. In FIG. 3, in order to make the explanation easier to understand, the information processing device, imaging device, input device, and display device of the other party's user are respectively referred to as "information processing device 1'," "imaging device 2'," "input device 3'," and "display device." "Device 4'".

In the present embodiment, the image processing unit 8 in the information processing device 1 converts an image including the user's face captured by the imaging device 2 into emotional information input by the user via the input device 3 and the emotion input unit 5. Process based on. The image processing unit 8 can output an image obtained by processing the user's facial expression to the display device 4 or the display device 4'. The image processing unit 8 can also generate an avatar from this image using the avatar generator 11 and output it to the display device 4 or the display device 4'. An avatar is a character displayed as a user's alter ego in a virtual space on a network. Further, each of these functions can be realized by a facial expression processing program. Each functional block will be explained below.

The input device 3 is, for example, a mouse. The input device 3 includes a mouse, a keyboard, a microphone, and the like. Further, the input device 3 may be an external device connected to the information processing device 1, or may be a built-in device integrated with the information processing device 1.

The display device 4 is, for example, a display. The user can output and display on the display device 4 or 4' an image or an avatar in which the user's facial expression has been processed using the facial expression processing program. For example, the user may output and display this image or avatar on the display device 4 or 4' via an online conference or streaming distribution. The display device 4 may be an external device connected to the information processing device 1, or may be a built-in device integrated with the information processing device 1.

The emotion input unit 5 provides a user interface (not shown) represented by a pull-down menu, for example. The emotion input unit 5 receives input of emotion information from the user using, for example, the input device 3. Emotional information is, for example, information related to emotions such as "happy" and "sad" and can be selected from a plurality of categories. Furthermore, the input to the emotion input section 5 is not limited to selection using a pull-down menu, and various input methods can be employed, such as selecting an icon using a mouse, inputting characters using a keyboard, and inputting voice using a microphone.

The storage unit 6 stores an image including a human facial expression serving as a model of emotional information as a teacher image 7. The storage unit 6 is constructed, for example, on an auxiliary storage device such as a hard disk. The storage unit 6 may be constructed not only on an internal device of the information processing device 1 but also on an external device such as an external hard disk or a cloud server. The teacher image 7 can employ various data such as an image including a human facial expression captured by the imaging device 2 or a digital camera, as well as illustration data or 3D data including a human facial expression.

The image processing unit 8 includes an AI (Artificial Intelligence) 9, a processor 10, and an avatar generator 11. The image processing unit 8 receives an image including the user's face and emotional information as input, and generates an image in which the facial expression of the user is processed from this image based on the emotional information. The processed image can be output to the display device 4 or 4'. Furthermore, an avatar can be generated from an image in which the user's facial expression has been processed, and can be output to the display device 4 or 4'.

The AI 9 receives an image including the user's face and emotional information as input. Furthermore, based on these inputs, the AI 9 outputs the results of analyzing facial movements that are insufficient to express emotions. AI9 uses the teacher image 7 to determine the relationship between an image including the user's face, emotional information, and facial movements that are insufficient to express emotion, using an algorithm such as a convolutional neural network (CNN), for example. It is possible to learn by using As an example of processing by the AI 9, the AI 9 extracts features of the user's face included in the image by performing a convolution operation on the received image. The AI 9 detects the user's facial expressions and line of sight based on the extracted features, and analyzes the user's emotions. The emotion analyzed here may be quantified in some form, or may be qualitatively expressed such as "expressionless." The AI 9 compares the emotional information, the emotion-analyzed image, and the teacher image 7, and analyzes facial movements that are insufficient to express the emotion. For example, AI9 compares an image in which emotions are analyzed with the teacher image 7 that corresponds to the emotion information regarding facial expressions around the mouth, and compares the characteristics of each to numerically evaluate facial movements that are insufficient to express emotions. may be converted into Furthermore, the AI 9 may use a plurality of teacher images 7 for comparison in order to confirm facial movements that are insufficient for emotional expression.

AI9 may include detecting facial orientation and hand gestures for emotional analysis. Furthermore, AI9 may include analyzing the line of sight from the received image. When analyzing the line of sight, the AI 9 checks the direction of the face and line of sight from the received image.

The processor 10 processes the expression on the user's face based on the emotion-analyzed image, the emotion information, and the comparison results so that the emotional expression is accurately conveyed. For example, when the user inputs "fun" as emotional information, the processor 10 processes the facial expression so that the emotional expression "fun" is accurately conveyed. This makes it possible to process facial expressions to complement the emotional expression of "fun." Additionally, existing algorithms can be used to process facial expressions. The processing device 10 may also process the line of sight based on the direction of the face and line of sight confirmed by the AI 9. For example, it is conceivable that the processing device 10 may synthesize the images so that the line of sight is facing forward, or process the images so that the user's line of sight matches that of the other user at an appropriate frequency or time. Appropriate frequency or time is, for example, setting the length of eye contact to 2 to 3 seconds, keeping the length of eye contact within 20 to 30 seconds per minute, or setting the time when you start talking and when you finish speaking. Processing such as making eye contact can be considered.

The avatar generator 11 generates a user's avatar from the image whose facial expression has been processed by the processing device 10. Existing algorithms can be used to generate avatars. The avatar's facial expression is generated to correspond to the processed image of the user's face. The avatar to be generated may be either a two-dimensional or three-dimensional character. Furthermore, the user can select whether or not to generate an avatar. For example, a user interface (not shown) represented by a button icon may allow the user to switch between outputting an image with processed facial expressions or an avatar.

FIG. 4 shows a flowchart of the facial expression processing program in the first embodiment. Here, a flow for outputting an avatar to the display device 4' based on an image including the user's face and emotional information will be described.

In step S11, the user inputs emotional information to the emotional input section 5 using the input device 3.

In step S12, the AI 9 in the image processing unit 8 receives an image including the user's face from the imaging device 2. In step S13, the AI 9 detects facial expressions and line of sight from the image received in step S12. In step S14, the AI 9 analyzes emotions from the image in which facial expressions and line of sight are detected.

In step S15, the AI 9 compares the emotion-analyzed image, the emotion information, and the teacher image 7 stored in the storage unit 6, and analyzes facial movements that are insufficient to express emotion. In step S16, the processor 10 processes the image including the user's face so that it corresponds to the emotional information based on the comparison result. In step S17, the avatar generator 11 generates the user's avatar based on the image with processed facial expressions. In step S18, the avatar generator 11 outputs the generated avatar image to the display device 4'.

In this flowchart, steps S12 to S14 are described as being performed after the flow of step S11, but they may be performed before the flow of step S11. In other words, a method may be adopted in which emotion analysis is performed after the input of emotional information, or a method may be adopted in which emotion analysis is completed first and then input of emotional information is received from the user.

Next, with reference to FIGS. 5 to 8, examples of input and output images when the user inputs emotional information such as "fun" into the emotion input section 5 will be described.

FIG. 5 is an example in which an image including the user's face captured by the imaging device 2 is displayed on the display device 4'. Further, FIG. 6 is an example in which an image obtained by processing the user's facial expression in the first embodiment is displayed on the display device 4'. The image in FIG. 5 represents an example of the user 20 who is "expressionless". In this example, the AI 9 detects facial expressions and line of sight from the image including the user's face shown in FIG. 5, and analyzes the user's emotion as being "expressionless." The AI9 also compares images analyzed as ``expressionless'', emotional information such as ``fun'', and a teacher image that serves as a model, and analyzes facial movements that are insufficient to express emotion. In FIG. 5, it is confirmed that the facial movement that is insufficient to express the emotion is the mouth, and the processing device 10 adjusts the user's facial expression so that the mouth expresses the emotion "fun" as shown in FIG. Process. Instead of displaying the user 20 with "no expression" on the display device 4', the processing device 10 outputs an image including the user 21 corresponding to the emotional information "fun".

Next, an example will be shown in which the avatar generator 11 outputs an avatar with processed facial expressions. FIG. 7 is an example in which an avatar corresponding to a user with "no expression" is displayed on the display device 4'. If an avatar is generated for the user 20 who is "expressionless" without processing the user's facial expression using a facial expression processing program, it will be represented as shown in FIG. In FIG. 7, the avatar 22 corresponding to the facial expression "no expression" is output as the expression corresponding to the user 20 who is "no expression". On the other hand, FIG. 8 is an example in which the avatar in the first embodiment is displayed on the display device 4'. The avatar generator 11 receives input of an image including the user 21 corresponding to the emotional information "fun", as shown in FIG. The avatar generator 11 generates an avatar 23 corresponding to the facial expression of "fun" as shown in FIG. Then, the avatar generator 11 outputs an image including the generated avatar to the display device 4'. In this example, a two-dimensional dog character is generated as an avatar, but the present invention is not limited to two-dimensional characters, and various avatars such as three-dimensional characters can be generated.

Note that the facial expression processing program may be executed by an external server (not shown) instead of being installed and executed in the information processing device 1. For example, after receiving input from the input device 3 and the imaging device 2 via the browser, the external server may generate an image or avatar in which the user's facial expression is processed and display it on the user's display device 4'. .

According to the present embodiment, for example, the user can compensate for his or her lacking emotional expression skills with the facial expression processing program, and can prevent the user from conveying content that is not intended by the user to the other party. Furthermore, according to the present embodiment, rich facial expressions can be felt even through the screen, so the user can smoothly encourage the other party to change the conversation in an online meeting, etc., thereby facilitating smooth communication. .

In addition, by communicating through avatars with rich facial expressions, users can communicate their intentions seamlessly in online meetings. Furthermore, by talking to the other user through an avatar that conveys emotions, the user can create a sense of intimacy with the other user even if the other user does not know the user.

Further, according to the present embodiment, in streaming distribution, the user who is the distributor can more clearly convey emotions to the viewers.

Furthermore, by using the AI 9, an image including the user's face can be compared with the teacher image 7, and the user can analyze facial movements that are insufficient for emotional expression. From this comparison result, by processing the image including facial expressions using the processing device 10, the user can accurately convey emotions.

Furthermore, by using an RGB imaging device, it is possible to accurately detect line of sight and analyze emotions, unlike a monochrome imaging device. By using an RGBIR imaging device, depth information can be obtained in addition to RGB information, so minute changes in facial expressions can be captured. Additionally, it is possible to detect the subject's line of sight and analyze their emotions, regardless of the subject's color or illuminance. Furthermore, in a near-infrared image, the pupil can be tracked, making it possible to stably detect the line of sight.

(Second embodiment)
FIG. 9 is a system block diagram of the information processing device 1 in which the facial expression processing program according to the second embodiment is installed.

In the second embodiment, the information processing device 1 includes a scene input section 12 in addition to the configuration in the first embodiment. The scene input unit 12 provides a user interface (not shown) represented by a pull-down menu, for example. The scene input unit 12 receives input of scene information from the user using, for example, the input device 3. The scene information is information representing the atmosphere of the scene, such as "discussion," "presentation," or "chat," and can be selected from a plurality of categories. Further, the scene input unit 12 is not limited to selection using a pull-down menu, and can employ various input methods such as selecting an icon using a mouse, inputting characters using a keyboard, inputting voice using a microphone, or inputting using a sensor. In this way, input to the scene input section 12 may be performed using the input device 3 (for example, a keyboard or a microphone) other than the mouse.

The image processing unit 8 in this embodiment receives an image including the user's face, emotional information from the user, and scene information, and analyzes the user's face from this image based on the emotional information and scene information. Generates an image with processed facial expressions.

FIG. 10 shows a flowchart of the facial expression processing program in the second embodiment. Here, a flow will be described in which an image in which the user's facial expression is processed is generated based on an image including the user's face, emotional information, and scene information, and is output as an avatar to the display device 4'. Furthermore, descriptions of steps S21 and S23 to S25 will be omitted.

In step S22, the user inputs scene information into the scene input section 12 using the input device 3. In step S26, the AI 9 compares the emotion-analyzed image, emotion information, scene information, and the teacher image 7 stored in the storage unit 6, and analyzes facial movements that are insufficient to express emotion. In step S27, the processor 10 processes the image including the user's face so that it corresponds to the emotional information based on the comparison result. In step S28, the avatar generator 11 generates the user's avatar based on the image with processed facial expressions. The avatar's facial expression is generated to correspond to the processed image of the user's face. In step S29, the avatar generator 11 outputs the generated avatar image to the display device 4'.

In this flowchart, steps S23 to S25 are described as being performed after steps S21 and S22, but may be performed before steps S21 and S22. That is, a method may be adopted in which emotion analysis is performed after the input of emotional information and scene information, or a method may be adopted in which emotion analysis is completed first and then input of emotional information and scene information is received from the user.

Next, an example of an output image when the user inputs emotional information such as "fun" into the emotional input section 5 and scene information such as "chat" into the scene input section 12 will be described. The image including the user's face will be described using FIG. 5 as in the first embodiment. FIG. 11 is an example in which an image processed with a user's facial expression according to the second embodiment is displayed on the display device 4'. In this example, considering the informal communication of "chat" which is situational information, more emphasis is placed on the emotional expression of fun. Unlike the first embodiment, the AI9 analyzes facial movements that are insufficient to express emotion as being in the eyes in addition to the mouth. As shown in FIG. 11, the processing device 10 processes the mouth and eyes so that they express the emotion of "fun." Instead of displaying the user 20 with a "neutral" expression, the display device 4' outputs an image including the user 24 with an emphasized "happy" expression.

When outputting as an avatar, the avatar generator 11 can generate the avatar from an image processed with the user's facial expression and output it to the display device 4'. FIG. 12 is an example in which the avatar according to the second embodiment is displayed on the display device 4'. An image including the avatar 25 with an emphasized "happy" expression corresponding to the facial expression of the user in FIG. 11 is output.

According to the present embodiment, the information processing device 1 is able to output an image in which the user's facial expression has been processed to match the atmosphere of the scene by considering not only emotional information by the user but also scene information. . Therefore, the user can express the emotions he/she wants to express more accurately and in a way that can be conveyed to the other party, and communication can be further improved.

(Third embodiment)
FIG. 13 is a system block diagram of the information processing device 1 in which the facial expression processing program according to the third embodiment is installed.

In the third embodiment, the information processing device 1 includes a feedback input unit 13 in addition to the configuration in the first embodiment. The feedback input unit 13 provides a user interface (not shown) represented by a pull-down menu, for example. The feedback input unit 13 receives feedback input from the user regarding whether the image obtained by processing the user's facial expression generated by the processor 10 or the expression of the avatar generated by the avatar generator 11 is optimal. . The feedback is, for example, information such as "good" or "fail" indicating the quality of the image obtained by processing the user's facial expression, and the user can select the feedback from a plurality of categories. Further, the feedback input unit 13 is not limited to selection using a pull-down menu, and can employ various input methods such as selecting an icon using a mouse, inputting characters using a keyboard, or inputting voice using a microphone. In this way, input to the feedback input unit 13 may be performed using an input device other than the mouse (for example, a keyboard or a microphone).

In this embodiment, the user gives feedback as to whether the image obtained by processing the user's facial expression generated by the processing device 10 or the expression of the avatar generated by the avatar generator 11 is optimal. Thereby, the image processing unit 8 can regenerate the processing of the user's facial expression or regenerate the avatar. Additionally, the AI 9 can perform additional learning based on feedback.

FIG. 14 shows a flowchart of the facial expression processing program in the third embodiment. This flowchart shows an example in which feedback is given to the avatar generated by the avatar generator 11.

In S31 to S38, avatar output is performed according to a flow similar to that in FIG. In step S39, the user inputs feedback into the feedback input section 13 regarding whether the expression of the avatar displayed on the display device 4' is optimal.

In step S39, if the user inputs feedback that the expression of the avatar is not optimal, the flow from steps S36 to S38 is repeated again to regenerate and reoutput the avatar. Here, in step S36, when processing the facial expression, the processor 10 may refer to another teacher image 7 to process the facial expression.

Further, in step S39, if the user inputs feedback indicating that the expression of the avatar is optimal, in step S40, the output of the avatar created in step S38 is continued. In step S41, the AI 9 performs additional learning based on the feedback obtained from the user in step S39. By performing additional learning based on feedback, the AI 9 can improve the accuracy of facial expression processing.

In addition, in this flowchart, an example was explained in which the user inputs feedback regarding whether the expression of the avatar is optimal, but if the avatar is not output, feedback is provided regarding the image obtained by processing the user's facial expression. Enter.

Further, the image obtained by processing the facial expression of the avatar or the user may be output to the display device 4 for the user to check the facial expression before being output to the display device 4'. In this case, feedback is input based on the avatar etc. output to the display device 4.

According to this embodiment, the information processing device 1 receives input of feedback from the user in addition to emotional information. By reprocessing images obtained by processing avatars and user's facial expressions to achieve optimal expression, users can more accurately convey the emotions they want to express to others. You can express yourself and improve your communication.

(Fourth embodiment)
FIG. 15 is a system block diagram of the information processing device 1 in which the facial expression processing program according to the fourth embodiment is installed.

In the fourth embodiment, the information processing device 1 includes an emotional information acquisition unit 14 in addition to the configuration in the first embodiment. The emotional information acquisition unit 14 acquires emotional information of the other user based on an image including the other user's face in an online conference or the like. The image including the other user's face is an example of the third image of the present disclosure. Hereinafter, in order to make the explanation easier to understand, the user who processes facial expressions will be referred to as "user A", and the other user will be referred to as "user B". It is conceivable that the image including the user B's face is received by the emotional information acquisition unit 14 of the information processing device 1 after the information processing device 1 receives the image captured by the imaging device 2' in an online conference or the like. It will be done.

The image processing unit 8 of the information processing device 1 in this embodiment receives an image including the user A's face captured by the imaging device 2, user A's emotional information, and user B's emotional information, and receives user A's emotional information. Generates an image with processed facial expressions.

FIG. 16 shows a flowchart of the facial expression processing program in the fourth embodiment. Here, a flow will be described in which an avatar of user A is output to the display device 4' of user B based on an image including the face of user A, emotional information of user A, and emotional information of user B. Furthermore, descriptions of steps S51 and S55 to S57 will be omitted.

In step S52, the emotional information acquisition unit 14 of the information processing device 1 receives the image of user B. In step S53, the emotional information acquisition unit 14 detects user B's facial expression and line of sight from the received image. In step S54, the emotion information acquisition unit 14 analyzes the emotion of user B based on the detected facial expression and line of sight. It is conceivable that the emotion information acquisition unit 14 analyzes emotions using a method similar to the AI 9, for example. The emotional information acquisition unit 14 extracts features in the image by performing a convolution operation on the received image using a convolutional neural network. The emotional information acquisition unit 14 detects the facial expression and line of sight of user B based on the extracted features, and analyzes the emotion. The emotion of user B analyzed here may be expressed numerically in some form, or may be qualitatively expressed such as "expressionless." In this way, the emotional information acquisition unit 14 acquires user B's emotional information through steps S52 to S54.

In step S58, the AI 9 of the information processing device 1 analyzes the image of the user A's emotions, the emotional information of the user A, the emotional information of the user B, and the teacher image stored in the storage unit 6 of the information processing device 1. 7 and analyze user A's facial movements that are insufficient for emotional expression. In step S59, the processing device 10 of the information processing device 1 processes the image including the user A's face so as to correspond to the emotional information based on the comparison result. In step S60, the avatar generator 11 of the information processing device 1 generates an avatar of the user A based on the facial expression-processed image. The facial expression of the avatar is generated to correspond to the processed image of user A's face. In step S61, the generated avatar image is output to the display device 4'.

According to this embodiment, the information processing device 1 processes the user's facial expression by reflecting not only the emotional information of the user A but also the emotional information of the user B. Therefore, user A can express the emotions he/she wants to express more accurately and in a way that can be conveyed to the other party according to user B's emotions, and communication can be further improved.

Other possible applications of the facial expression processing program include the following. For example, it is possible to apply facial expression processing programs to handling complaints at call centers. By conveying emotions through avatars, operators can prevent customers from making unnecessary offensive comments.

Another example would be to apply the facial expression processing program to crime prevention in unmanned stores. By having avatars with facial expressions similar to those of humans watching over stores, it is possible to curb shoplifting and abnormal behavior.

Another example would be to apply the facial expression processing program to remote support using digital signage in unmanned stores. By expressing the store clerk's facial expressions richly through the avatar and displaying it remotely on digital signage, the avatar can provide the same level of customer service as in-person customer service.

(Fifth embodiment)
FIG. 17 shows an example of the hardware configuration of the information processing device 1 in the fifth embodiment. The information processing device 1 of the fifth embodiment corresponds to an example of the information processing device 1 of the first embodiment.

The information processing device 1 according to this embodiment is configured by a computer device 200. The computer device 200 includes a CPU (Central Processing Unit) 201, a main storage device 202, an auxiliary storage device 203, a communication interface 204, and an input/output interface 205, which are interconnected by a bus 206.

The CPU 201 executes computer programs that implement the above-described functional configurations of the information processing device 1 on the main storage device 202 . When the CPU 201 executes a computer program, the functions of the emotion input section 5 and the image processing section 8 shown in FIG. 3 are realized. This computer program is, for example, a facial expression processing program, an online conference program, or a streaming distribution program.

The main storage device 202 stores programs that implement the processing of this embodiment, data necessary for executing the programs, data generated by executing the programs, and the like. The program is expanded on the main storage device 202 and executed. The main storage device 202 is, for example, a RAM (Random Access Memory), but is not limited to this.

The auxiliary storage device 203 stores the program, data necessary for executing the program, data generated by executing the program, and the like. These programs and data are read into the main storage device 202 during processing of this embodiment. The auxiliary storage device 203 is, for example, a hard disk, an optical disk, a flash memory, and a magnetic tape, but is not limited thereto. The storage unit 6 in FIG. 3 may be constructed on the auxiliary storage device 203.

The communication interface 204 is a circuit for communicating by wire or wirelessly with an external computer device during an online conference or the like.

The input/output interface 205 is a circuit for connecting with the imaging device 2 and the mouse, which is an example of the input device 3, as well as input devices such as a keyboard and microphone, and output devices such as the display device 4.

Bus 206 is a circuit for interconnecting CPU 201, main storage 202, auxiliary storage 203, communication interface 204, and input/output interface 205.

Note that the above-mentioned program may be installed in advance on the computer device 200, or may be stored in a storage medium such as a CD-ROM. Further, the program may be uploaded onto the Internet.

Note that the computer device 200 may include one or more of each of a CPU 201, a main storage device 202, an auxiliary storage device 203, a communication interface 204, and an input/output interface 205.

Further, the information processing device 1 may be configured with a single computer device 200, or may be configured as a system consisting of a plurality of mutually connected computer devices 200.

According to this configuration, the functions of the facial expression processing program in the first embodiment can be realized by software. Further, although FIG. 16 shows an example of the hardware configuration of the information processing device 1 in FIG. 3, it is possible to implement the functions of the facial expression processing program by software in other embodiments with a similar configuration. .

Although the embodiments of the present disclosure have been described above, these embodiments may be implemented with various changes within the scope of the gist of the present disclosure. For example, two or more embodiments may be combined and implemented.

Note that the present disclosure can also have the following configuration.

(1)
an emotion input step of accepting emotional information input from the user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing step for generating
A facial expression processing program that causes a computer to execute facial expression processing methods including

(2)
The facial expression processing program according to (1), further comprising an avatar generation step of generating an avatar, which is a character to be displayed as an alter ego of the user in a virtual space on a network, based on the second image.

(3)
The facial expression processing program according to (2), wherein the facial expression of the avatar is generated to correspond to the facial expression of the user in the second image.

(4)
The facial expression processing program according to (1), wherein the image processing step processes the facial expression of the user in the first image into an expression corresponding to the emotional information.

(5)
The image processing step processes the user's facial expression in the first image by acquiring a teacher image including a human facial expression and comparing the first image and the teacher image. , the facial expression processing program described in (1).

(6)
The image processing step analyzes the user's emotion from the first image, and analyzes the user's face in the first image based on the analysis result, the emotion information, and the comparison result. The facial expression processing program described in (5) that processes facial expressions.

(7)
The facial expression processing program according to (1), wherein the first image is an image captured by an imaging device.

(8)
The facial expression processing program according to (7), wherein the imaging device is an RGB imaging device or an RGBIR imaging device.

(9)
further comprising a scene input step of receiving scene information input from the user,
The image processing step receives the first image, the emotional information, and the scene information, and processes the facial expression of the user in the first image based on the emotional information and the scene information. The facial expression processing program according to (1), wherein the second image is generated from the first image by doing so.

(10)
further comprising a feedback input step of receiving feedback input regarding the second image from the user;
The facial expression processing program according to (1), wherein the image processing step performs learning based on the feedback.

(11)
The facial expression processing program according to (1), wherein the second image is output as an output image in an online conference.

(12)
The facial expression processing program according to (1), wherein the second image is output to the user's display device or the other user's display device.

(13)
The facial expression processing program according to (2), wherein the avatar is output as an output image in an online conference.

(14)
The facial expression processing program according to (2), wherein the avatar is output to the user's display device or the other user's display device.

(15)
Further comprising an emotional information acquisition step of acquiring emotional information of the other user,
The image processing step receives the first image, the emotional information, and the emotional information of the other user, and processes the image in the first image based on the emotional information and the emotional information of the other user. The facial expression processing program according to (1), wherein the second image is generated from the first image by processing the facial expression of the user.

(16)
The emotional information acquisition step is described in (15), wherein a third image including the face of the other user is received from the other user, and emotional information of the other user is obtained based on the third image. Facial expression processing program.

(17)
The facial expression processing program according to (16), wherein the emotion information acquisition step acquires the emotion information of the other party user by analyzing the emotion of the other party user from the third image.

(18)
The facial expression processing program according to (16), wherein the third image is an image captured by an imaging device.

(19)
an emotion input section that accepts input of emotional information from a user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing unit that generates
An expression processing device equipped with.

(20)
an emotion input step of accepting emotional information input from the user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing step for generating
Facial expression processing methods including.

1: Information processing device, 2: Imaging device, 3: Input device, 4, Display device, 5: Emotion input unit,
1': information processing device of the other user, 2': imaging device of the other user,
3': input device of the other user; 4', display device of the other user;
6: Storage unit, 7: Teacher image, 8: Image processing unit, 9: AI, 10: Processing device,
11: Avatar generator, 12: Scene input unit, 13: Feedback input unit,
14: Emotional information acquisition unit, 20: User who is "expressionless",
21: User corresponding to the emotional information “fun”,
22: Avatar that corresponds to the facial expression “no expression”,
23: Avatar corresponding to the facial expression of “fun”,
24: User whose “happy” expression was emphasized,
25: Avatar with emphasized “fun” expression, 100: Network,
200: Computer device, 201: CPU, 202: Main storage device,
203: Auxiliary storage device, 204: Communication interface,
205: input/output interface, 206: bus,
300: Streaming distribution server, 500: Information processing device group

Claims (20)

an emotion input step of accepting emotional information input from the user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing step for generating
A facial expression processing program that causes a computer to execute facial expression processing methods including The facial expression processing program according to claim 1, further comprising an avatar generation step of generating an avatar, which is a character to be displayed as an alter ego of the user in a virtual space on a network, based on the second image. The facial expression processing program according to claim 2, wherein the facial expression of the avatar is generated to correspond to the facial expression of the user in the second image. The facial expression processing program according to claim 1, wherein the image processing step processes the facial expression of the user in the first image into an expression corresponding to the emotional information. The image processing step processes the user's facial expression in the first image by acquiring a teacher image including a human facial expression and comparing the first image and the teacher image. The facial expression processing program according to claim 1. The image processing step analyzes the user's emotion from the first image, and analyzes the user's face in the first image based on the analysis result, the emotion information, and the comparison result. The facial expression processing program according to claim 5, which processes facial expressions. Emotion input step, image processing step The facial expression processing program according to claim 1, wherein the first image is an image captured by an imaging device. The facial expression processing program according to claim 7, wherein the imaging device is an RGB imaging device or an RGBIR imaging device. further comprising a scene input step of receiving scene information input from the user,
The image processing step receives the first image, the emotional information, and the scene information, and processes the facial expression of the user in the first image based on the emotional information and the scene information. The facial expression processing program according to claim 1, wherein the second image is generated from the first image by doing so. further comprising a feedback input step of receiving feedback input regarding the second image from the user;
The facial expression processing program according to claim 1, wherein the image processing step performs learning based on the feedback. The facial expression processing program according to claim 1, wherein the second image is output as an output image in an online conference. The facial expression processing program according to claim 1, wherein the second image is output to the display device of the user or the display device of the other user. The facial expression processing program according to claim 2, wherein the avatar is output as an output image in an online conference. 3. The facial expression processing program according to claim 2, wherein the avatar is output to the user's display device or the other user's display device. Further comprising an emotional information acquisition step of acquiring emotional information of the other user,
The image processing step receives the first image, the emotional information, and the emotional information of the other user, and processes the image in the first image based on the emotional information and the emotional information of the other user. The facial expression processing program according to claim 1, wherein the second image is generated from the first image by processing the facial expression of the user. 16. The emotional information acquisition step includes receiving a third image including the face of the other user from the other user, and acquiring emotional information of the other user based on the third image. Facial expression processing program. 17. The facial expression processing program according to claim 16, wherein the emotion information acquisition step acquires the emotion information of the other party user by analyzing the emotion of the other party user from the third image. The facial expression processing program according to claim 16, wherein the third image is an image captured by an imaging device. an emotion input section that accepts input of emotional information from a user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing unit that generates
An expression processing device equipped with. an emotion input step of accepting emotional information input from the user;
A first image including the user's face and the emotional information are received, and the facial expression of the user in the first image is processed based on the emotional information, thereby converting the first image into a second image. an image processing step for generating
Facial expression processing methods including.

Images