JP5720595B2 - Network system, server device, information processing device, and program - Google Patents

Description

  The present invention relates to a network system, a server device, an information processing device, and a program.

  When business personnel are far away, telephone, e-mail, video conferencing, etc. are used as means of communication. When using telephone, e-mail, video conferencing, etc. as a means of communication, it is difficult to directly hear the other person's facial expression and appearance, and it becomes difficult to know changes and situations of motivation.

  For example, it is difficult to detect the emotions and situations of the other party in sentences such as e-mails and letters. In addition, although the voice of the telephone, the video conference, and the video of the Web camera can temporarily know the other party's state, only the other party's state during communication can be known.

  Conventionally, for example, a happy calendar is already known as a tool that shows the user's mood with an icon and can be shared between users. The happy calendar is a tool that “visualizes” the user's mood. In addition, a state display system that transmits and displays state information such as personal feelings between state display terminals is conventionally known (see, for example, Patent Document 1).

  However, the conventional tool to “visualize” the mood of the user manually updates the user's mood, so the user forgets to update or cannot update due to busy time. There was a problem to do.

  In addition, the conventional tool to “visualize” the mood of the user decides the mood of the user to update by self-judgment, so the user is troublesome to update and always updates in the same mood or feels down There was a problem that even when I was on the road, I couldn't update the icon to indicate that I was feeling depressed because I refrained from surroundings.

  Conventional tools for “visualizing” the user's mood have the problem that the user's mood may not be updated accurately as described above, and the expected effect cannot be obtained and cannot be effectively utilized.

  An object of the present invention is to provide a network system, a server device, an information processing device, and a program that can automate updating of display information related to facial expressions.

  In order to solve the above problem, a network system according to claim 1 is a network system in which a first device and a second device are connected, and user information and image data are transmitted by the first device. An acquiring means for acquiring, a face area extracting means for extracting a face area included in the image data from the image data acquired by the acquiring means, and a face extracted by the face area extracting means from a plurality of facial expressions. Identification means for identifying facial expressions in the area; storage means for storing facial expression identification information indicating facial expressions in association with display information for each of a plurality of facial expressions; and display stored in the storage means A specifying means for specifying display information stored in association with facial expression identification information corresponding to the facial expression identified by the identification means, the display information specified by the specifying means, and the acquiring means. It associates the acquired user information; and a display means for displaying on the display unit of the second device.

  In addition, what applied the component, expression, or arbitrary combination of the component of this invention to a method, an apparatus, a system, a computer program, a recording medium, a data structure, etc. is also effective as an aspect of this invention.

  According to the present invention, updating of display information related to facial expressions can be automated.

It is a block diagram of an example of the motivation visualization system. It is a block diagram of the other example of the motivation visualization system. It is a hardware block diagram of an example of a server apparatus. It is a hardware block diagram of an example of a terminal device. It is a functional block diagram of an example of a motivation visualization system. It is a block diagram of an example of facial expression DB. It is a block diagram of an example of index DB. It is a block diagram of an example of history DB. It is a block diagram of an example of icon DB. It is an image figure of an example of icon DB. It is a flowchart which shows an example of the process sequence of the server apparatus contained in a motivation visualization system. It is a flowchart of an example of a process of step S5. It is an image figure of an example of a process of step S5. It is an image figure of an example of UI displayed on a terminal device. It is an image figure of the other example of UI displayed on a terminal device. It is a functional block diagram of another example of a terminal device.

  Hereinafter, embodiments of the present invention will be described in detail. Note that the motivation in the present embodiment is treated as including the meaning of the user's feelings, motivation, mood, state, situation, emotion, and the like.

  FIG. 1 is a configuration diagram of an example of a motivation visualization system. A motivation visualization system (hereinafter simply referred to as a system) 1 in FIG. 1 includes a server device 10 and one or more terminal devices 11 connected via a network 12 such as the Internet or a LAN. The terminal device 11 is, for example, a PC (Personal Computer), a PDA (Personal Digital Assistant), a mobile phone, a dedicated terminal, or the like.

  The terminal device 11 is equipped with a camera. The camera mounted on the terminal device 11 may be either internal or external. The terminal device 11 is operated by a user who performs visualization of motivation. The terminal device 11 is connected to the server device 10 and performs processing to be described later, and updates information representing the user's motivation. In addition, since various processes of the server apparatus 10 and the terminal apparatus 11 in the system 1 of FIG.

  FIG. 2 is a configuration diagram of another example of the motivation visualization system. In the motivation visualization system (hereinafter simply referred to as a system) 2 in FIG. 2, a plurality of terminal devices 11 are connected via a network 12 such as the Internet or a LAN. The terminal device 11 is, for example, a PC, a PDA, a mobile phone, a dedicated terminal, or the like.

  The terminal device 11 is equipped with a camera. The camera mounted on the terminal device 11 may be either internal or external. The terminal device 11 is operated by a user who performs visualization of motivation. The terminal device 11 is connected to the other terminal device 11 to perform processing described later, and updates information representing the user's motivation. 2 includes a form in which the system 2 is connected via a server device as a simple relay device included in the network 12.

  FIG. 3 is a hardware configuration diagram of an example of the server apparatus. The server device 10 includes an input device 21, an output device 22, a recording medium reading device 23, an auxiliary storage device 24, a main storage device 25, an arithmetic processing device 26, and an interface device 27 that are connected to each other via a bus 28. .

  The input device 21 is a keyboard or a mouse. The input device 21 is used for inputting various signals. The output device 22 is a display device or the like. The output device 22 is used to display various windows and data. The interface device 27 is a modem, a LAN card, or the like. The interface device 27 is used for connecting to the network 12.

  A program for realizing the system 1 such as a motivation visualization program is provided by, for example, distribution of the recording medium 29 or download from the network 12. A program for realizing the system 1 such as a motivation visualization program is at least a part of a program executed by the server device 10.

  The recording medium 29 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor that electrically records information, such as a ROM or flash memory. Various types such as a memory can be used. The recording medium 29 is set in the recording medium reader 23. A program for realizing the system 1 such as a motivation visualization program is installed in the auxiliary storage device 24 from the recording medium 29 via the recording medium reader 23.

  A program for realizing the system 1 such as a motivation visualization program downloaded from the network 12 is installed in the auxiliary storage device 24 via the interface device 27.

  The auxiliary storage device 24 stores a program for realizing the system 1 such as a motivation visualization program, necessary files, data, and the like. The main storage device 25 reads and stores a program for realizing the system 1 such as a motivation visualization program from the auxiliary storage device 24 when the program is started. The arithmetic processing unit 26 implements various functions to be described later according to a program for realizing the system 1 such as a motivation visualization program stored in the main storage device 25.

  FIG. 4 is a hardware configuration diagram of an example of the terminal device. Here, a hardware configuration when the terminal device 11 is a PC will be described as an example. The terminal device 11 includes an input device 31, an output device 32, a recording medium reading device 33, an auxiliary storage device 34, a main storage device 35, an arithmetic processing device 36, an interface device 37, and a camera 38 that are connected to each other via a bus 39. It is a configuration. The camera 38 may be built in the terminal device 11 or may be externally attached.

  The input device 31 is a keyboard or a mouse. The input device 31 is used for inputting various signals. The output device 32 is a display device or the like. The output device 32 is used to display various windows and data. The interface device 37 is a modem, a LAN card, or the like. The interface device 37 is used for connecting to the network 12. The camera 38 captures the user's face.

  A program for realizing the system 1 or 2 such as a motivation visualization program is provided by, for example, distribution of the recording medium 40 or downloading from the network 12. A program for realizing the system 1 such as a motivation visualization program is at least a part of a program executed by the terminal device 11.

  The recording medium 40 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor that electrically records information, such as a ROM or flash memory. Various types such as a memory can be used. The recording medium 40 is set in the recording medium reader 33. A program for realizing the system 1 such as a motivation visualization program is installed from the recording medium 40 to the auxiliary storage device 34 via the recording medium reader 33.

  A program for realizing the system 1 such as a motivation visualization program downloaded from the network 12 is installed in the auxiliary storage device 34 via the interface device 37.

  The auxiliary storage device 34 stores a program for realizing the system 1 such as a motivation visualization program, necessary files, data, and the like. The main storage device 35 reads and stores a program for realizing the system 1 such as a motivation visualization program from the auxiliary storage device 34 when the program is started. The arithmetic processing unit 36 realizes various functions to be described later according to a program for realizing the system 1 such as a motivation visualization program stored in the main storage device 35.

  FIG. 5 is a functional block diagram of an example of the motivation visualization system. FIG. 5 is a functional block diagram of an example in the system 1 of FIG. In FIG. 5, the terminal device 11 includes an imaging unit 51, a transmission unit 52, a UI processing unit 53, and a reception unit 54. The server device 10 includes a receiving unit 61, a still image converting unit 62, a face extracting unit 63, a part extracting unit 64, a facial expression recognizing unit 65, an indexing unit 66, an icon determining unit 67, a transmitting unit 68, a facial expression DB 69, and an index DB 70. A history DB 71 and an icon DB 72.

  The photographing unit 51 of the terminal device 11 photographs the user's face. The photographing unit 51 may photograph a moving image of the user's face or a still image. When the imaging unit 51 captures a still image, the processing of the later-described still image converting unit 62 becomes unnecessary. Here, the description is continued assuming that the photographing unit 51 has photographed a moving image. The transmission unit 52 transmits the moving image captured by the imaging unit 51 to the server device 10.

  The UI processing unit 53 displays the UI on the output device 32. The receiving unit 54 receives a later-described icon update request from the server device 10. The icon update request is a request for displaying an icon representing the user's motivation described later on the UI. Based on the icon update request, the UI processing unit 53 displays or updates an icon representing the user's motivation on the UI.

  The receiving unit 61 of the server device 10 receives a moving image of the user's face. The still image converting unit 62 converts the moving image received by the receiving unit 61 into a still image. The face extraction unit 63 extracts a face from the still image that the still image converting unit 62 converts to a still image. The part extracting unit 64 extracts each part from the face on the still image extracted by the face extracting unit 63. The part extracted here may be a part useful for facial expression recognition. Here, eyes, eyebrows and mouth are extracted as parts useful for facial expression recognition.

  The facial expression recognition unit 65 recognizes the facial expression of each part from the shape (contour etc.) of the facial part on the still image extracted by the part extraction unit 64. A facial expression DB 69 described later is used for recognizing facial expressions of each part. The indexing unit 66 indexes the facial expression of each part recognized by the facial expression recognition unit 65, and calculates the average of the indexes. An index DB 70, which will be described later, is used for indexing the facial expression of each part. The indexing unit 66 stores the average of the calculated indices in the history DB 71 described later.

  The icon determination unit 67 refers to an icon DB 72 described later, and determines an icon corresponding to the average of the indexes calculated by the indexing unit 66. The icon is an example of information indicating user motivation. The transmission unit 68 transmits an icon update request for an icon representing the user's motivation determined by the icon determination unit 67 to the terminal device 11.

  The user index history stored in the history DB 71 is transmitted from the server device 10 to the terminal device 11 when a reference request is received from the terminal device 11. For example, the user can refer to the history of the index of the user displayed on the UI.

  In the system 1 of FIG. 5, the expression DB 69, the index DB 70, the history DB 71, and the icon DB 72 are provided in the server device 10, but at least one of the expression DB 69, the index DB 70, the history DB 71, and the icon DB 72 is replaced with another one. You may make it provide in two or more server apparatuses.

  Further, the system 1 of FIG. 5 may be provided with, for example, the icon determination unit 67 and the icon DB 72 in the terminal device 11. In the system 1 in which the icon determination unit 67 and the icon DB 72 are provided in the terminal device 11, instead of the icon update request, the average of the index calculated by the indexing unit 66 is transmitted from the transmission unit 68 to the terminal device 11. Furthermore, although the system 1 of FIG. 5 shows an example constituted by one server device 10, it may be constituted by a plurality of server devices.

  FIG. 6 is a configuration diagram of an example of the facial expression DB. The expression DB 69 in FIG. 6 includes an expression table, an eye table, an eyebrow table, and a mouth table. In the expression DB 69, an eye table, an eyebrow table, and a mouth table, which are tables of each part, are associated with an expression table of each expression (for example, emotions).

  In the expression table, an expression ID (eyes), an expression ID (eyebrows), and an expression ID (mouth) are set for each expression. The expression ID (eyes), the expression ID (eyebrow), and the expression ID (mouth) are set by numbers, for example. In the case of the facial expression ID (eyes), for example, numbers such as e1 and e2 are assumed.

  In the eye table, facial expression IDs (eyes), features (eyes), and trajectories (eyes) are set. In the eyebrow table, an expression ID (eyebrow), a characteristic (eyebrow), and a locus (eyebrow) are set. In the mouth table, an expression ID (mouth), a feature (mouth), and a locus (mouth) are set. In the eye table, eye characteristics and trajectories are set. In the eyebrow table, the characteristics and locus of the eyebrows are set. The mouth table is set with mouth features and trajectories.

  With respect to the expression table, the eye table, the eyebrow table, and the mouth table, which are tables of each part, are associated with each other by an expression ID (eye), an expression ID (eyebrow), and an expression ID (mouth). In the eye table, eyebrow table, and mouth table, which are tables of each part, for example, the position and angle of the vertex of the outline of each part are set as features, and a collection of points that border the outline of each part is set as a trajectory Assuming that For example, in the eye table, the positions and angles of the vertices of the eye outline are set as features, and a collection of points that border the eye outline is assumed as a trajectory.

  When referring to the facial expression DB 69, each feature table having similar features and trajectories is compared with the features and trajectories of each part table using the facial features and trajectories on the still image extracted by the part extracting unit 64 as keys. The facial expression ID is acquired. For example, when the eye feature and trajectory are used as a key and compared with the feature and trajectory of the eye table, the facial expression ID (eye) of the eye table similar to the feature and trajectory is acquired.

  The facial expression recognition unit 65 can recognize the facial expression of the user by searching the facial expression ID in the facial expression table using the acquired facial expression ID as a key. For example, the facial expression recognition unit 65 can recognize the facial expression of the user's eyes by searching the facial expression ID (eyes) in the facial expression table using the facial expression ID (eyes) as a key.

  FIG. 7 is a configuration diagram of an example of the index DB. The index DB 70 in FIG. 7 has an index table. The index DB 70 has an index table for each part. The index table for each part is set in association with facial expressions and indices (values). That is, the index DB 70 sets an index for each facial expression in association with each part.

  When referring to the index DB 70, the facial expression of each part of the face on the still image recognized by the facial expression recognition unit 65 is compared with the facial expression in the index table of each part, and matched with the matching facial expression from the index table of each part. Get the index.

  FIG. 8 is a configuration diagram of an example of the history DB. The history DB 71 in FIG. 8 stores a date, a user ID, and an index (value) in association with each other. That is, the history DB 71 sets a user index for each date in association with the user ID. The history DB 71 is associated with user information by a user ID. Note that the user information in FIG. 8 includes information about the user such as a user ID and a user name. By using the user information, the user name can be converted into a user ID. When referring to the history DB 71, the user index is acquired from the history DB 71 using the user ID as a key.

  FIG. 9 is a configuration diagram of an example of an icon DB. The icon DB 72 of FIG. 9 stores icon IDs, index ranges, and images in association with each other. That is, the icon DB 72 sets an icon ID and an image in association with the index range. When referring to the icon DB 72, an icon ID and an image are acquired from the icon DB 72 using a user index as a key. In other words, different icon IDs and images are acquired from the icon DB 72 depending on where the index of the user is included in the range of the index of the icon DB 72.

  FIG. 10 is an image diagram of an example of the icon DB. The icon DB 72 in FIG. 10 is an example in which three types of icons are stored. When the user index is A1 to A2, the icon DB 72 of FIG. 10 acquires an image with the icon ID “iconA”. When the user index is B1 to B2, the icon DB 72 of FIG. 10 acquires an image with the icon ID “iconB”. Further, when the user index is C1 to C2, the icon DB 72 in FIG. 10 acquires an image with the icon ID “iconC”.

  In FIG. 10, the image with the icon ID “iconA” is an example of a smiled face. The image with the icon ID “iconB” is an example of an ordinary face. The image of the icon ID “iconC” is an example of an angry face.

  FIG. 11 is a flowchart illustrating an example of a processing procedure of the server device included in the motivation visualization system. 11 is started at a predetermined time interval when the terminal device 11 is activated, when a program related to the motivation visualization system is activated on the terminal device 11, or when an update button or the like is pressed on the terminal device 11. And so on.

  In step S <b> 1, the receiving unit 61 of the server device 10 receives a moving image of the user's face from the terminal device 11. In step S2, the still image converting unit 62 converts the moving image received by the receiving unit 61 into a plurality of still images. In the process of step S2, a plurality of still images can be created from a moving image by using, for example, OpenCV (Open Computer Vision Library).

  OpenCV is a library of functions related to image recognition. OpenCV is assumed to be mainly used for computing technology related to image recognition and analysis called computer vision. By using OpenCV, the server device 10 can use advanced image processing functions related to computer vision.

  Note that in step S2, the still image creating unit 62 assumes that the relationship between N and M is N ≧ k × M (k> 1) where the moving image is N seconds and the sampling interval is M seconds. The processes in steps S3 to S6 are performed for each still image created in step S2.

  In step S <b> 3, the face extraction unit 63 extracts a face outline from the still image that the still image converting unit 62 has converted into a still image. Even in the process of step S3, the face contour can be extracted from the still image by using OpenCV.

  In step S <b> 4, the part extracting unit 64 extracts the outlines of eyes, eyebrows, and mouths as respective parts from the face outlines on the still image extracted by the face extracting unit 63. For example, the part extraction unit 64 extracts the outline of the mouth from the lower half of the face outline, and extracts the outlines of the eyes and eyebrows from the upper half of the face outline. Even in the process of step S4, the contours of eyes, eyebrows, and mouth can be extracted from the contours of the face by using OpenCV.

  In step S5, the facial expression recognition unit 65 uses the eyes, eyebrows, and mouth contours of the user's face on the still image extracted by the part extraction unit 64 to use the eye table, eyebrow table, mouth table features and locus of the facial expression DB 69. And pattern matching to recognize facial expressions. In pattern matching, in addition to the features and trajectories of each part, the distance to each part may be used. Details of the processing in step S5 will be described later.

  In step S <b> 6, the indexing unit 66 uses the facial expression recognition result performed by the facial expression recognition unit 65 to index the facial expression of the user's face, eyebrows, and mouth on a single still image. The index of facial expression of the upper user's face, eyebrows, and mouth is averaged and calculated as the index of the user's face on one still image. The indexing unit 66 uses the index DB 70 for indexing the facial expressions of the user's face, eyebrows, and mouth. Regarding the index, for example, X> Y when the facial expression is “joy”, and X is Y when the facial expression is “anger”.

  After performing the processing of Steps S3 to S6 for each still image created in Step S2, the indexing unit 66 calculates all the still image user face indices calculated in Steps S3 to S6 from Step S7. The average of the face index of the user of the still image is calculated. In step S8, the indexing unit 66 stores the average of the user's face indices of all still images calculated in step S7 in the history DB 71 together with the date and user ID.

  In step S <b> 9, the icon determination unit 67 refers to the icon DB 72 and determines an icon corresponding to the average of the indexes calculated by the indexing unit 66. In step S <b> 10, the transmission unit 68 transmits an icon update request for an icon representing the user's motivation determined by the icon determination unit 67 to the terminal device 11.

  Thereafter, the receiving unit 54 of the terminal device 11 receives an icon update request from the server device 10. The UI processing unit 53 of the terminal device 11 can display or update an icon representing the user's motivation on the UI based on the icon update request.

  FIG. 12 is a flowchart of an example of the process in step S5. The processing in steps S21 to S23 is performed for each of the eyes, eyebrows, and mouth contours of the user's face on the still image extracted by the part extraction unit 64.

  In step S21, the facial expression recognition unit 65 performs shape matching of the three methods as described later. Proceeding to step S22, the facial expression recognition unit 65 sums up the results of shape matching in step S21. In step S23, the facial expression recognizing unit 65 sets a table (eye table) in which features and trajectories of the user's face (such as eyes) on the still image extracted by the region extracting unit 64 and features and trajectories that are very close are set. Etc.) is found.

  If a table of a part for which a very close feature and locus is set is not found, the facial expression recognition unit 65 returns to step S21 and continues processing for the table of the next part. When a table of a part where a very close feature and locus are set is found, the facial expression recognition unit 65 performs the process of step S24 at that time. The facial expression recognition unit 65 also performs the process of step S24 when shape matching has been completed for all the parts of the table.

  In step S24, the facial expression recognition unit 65 acquires the facial expression ID set in the table of the part where the closest feature and locus are set, and searches the facial expression ID in the facial expression table using the acquired facial expression ID as a key, thereby Recognize facial expressions on the face.

  Here, the shape matching in step S21 will be further described. Shape matching is to perform shape comparison on two input images. In the shape matching in step S21, it is assumed that the shape matching is performed by three kinds of methods as in the later-described OpenCV sample code.

opencv.jp-OpenCV: Matching sample code-(see "Matching shapes"): http://opencv.jp/sample/matching.html
The general flow of the above-described OpenCV sample code is as follows.
(1) Read an input image and a comparison image.
(2) Shape comparison is performed by three methods.
(3) Output the result of the shape comparison to an image.
(4) Read the next comparison image and return to (2).

  Here, the function called cvMatchShapes used in the above-described sample code flow (2) is a function that returns a numerical value close to “0” when the shapes of two images are more similar. The shape comparison technique used by the cvMatchShapes function described above is described in the following link.

Structural analysis and shape descriptors-opencv v2.1 documentation (see cvMatchShapes)
http://opencv.jp/opencv-2.1/c/structural_analysis_and_shape_descriptors.html#cv-matchshapes
In the present embodiment, shape matching between a user's face part (input image) extracted from a still image and a pattern (comparison image) such as a feature or locus set in the part table of the facial expression DB 69 is performed (cvMatchShapes function). 3 types of shape comparison). In the present embodiment, the facial expression associated with the table of the part where the pattern that is closer to 0 is obtained by averaging the results of the shape comparisons is used as the facial expression of the user's face.

  FIG. 13 is an image diagram of an example of the process of step S5. When the facial expression recognition unit 65 determines facial expressions using the contours of the user's face eyes, eyebrows, and mouth on the still image extracted by the region extraction unit 64, the eye table, eyebrow table, mouth as shown in FIG. Performs pattern matching with table features and trajectory patterns.

  For example, FIG. 13 shows an example of determining the facial expression of the user's face on a still image. The facial expression recognition unit 65 is a user's eye pattern that is the contour of the user's face on the still image extracted by the part extraction unit 64, and patterns such as eye features and trajectories set in the eye table of the facial expression DB 69. And pattern matching. Then, the facial expression recognition unit 65 determines the facial expression of the eyes from the eye pattern set in the eye table closer to the eye pattern of the user's face on the still image extracted by the part extraction unit 64. In the example of FIG. 13, “Easy” is determined as the facial expression of the eyes.

  FIG. 14 is an image diagram of an example of a UI displayed on the terminal device. The UI in FIG. 14 displays an icon representing user motivation for each date for each user. For example, when the mouse cursor is overlaid on the icon, the UI display unit 53 may display supplementary information such as a comment input by the user. If supplementary information such as comments input by the user can be displayed, the UI can be expected to further promote the sharing of motivation among users.

  FIG. 15 is an image diagram of another example of a UI displayed on the terminal device. The UI of FIG. 15 displays an icon representing the current user's motivation as an auxiliary screen. The UI of FIG. 15 can be expected to improve the convenience of the user when it is assumed that the current user's motivation is to be confirmed.

  Note that the system 1 in FIG. 5 is an example, and a configuration in which at least a part of the functions of the server device 10 is provided on the terminal device 11 side is also possible. For example, the system 1 may have a configuration in which the server apparatus 10 has the history DB 71 of the functions of the server apparatus 10 in FIG. 5 and the terminal apparatus 11 has functions other than the history DB 71 in the functions of FIG.

  In the system 1 in which the server apparatus 10 has the history DB 71 among the functions of the server apparatus 10 in FIG. 5 and the terminal apparatus 11 has functions other than the history DB 71 in the functions in FIG. 5, the server apparatus 10 receives from the terminal apparatus 11. The server device 10 provides a history of the user index based on a request from the terminal device 11.

  As described above, according to the system 1 of the present embodiment, the user's motivation is obtained by photographing the user's face with the terminal device 11 and indexing the user's facial expression based on the recognition result of the photographed facial expression of the user. Can be displayed on the UI. As described above, in the system 1 according to the present embodiment, it is possible to automate the icon update that has been performed manually.

  Also, according to the system 1 of the present embodiment, the user automatically updates the icon representing the user's motivation that has been determined by the user's self-judgment based on the facial expression of the photographed user's face. The problem is that you always update with the same mood, even if you are feeling depressed, withholding the surroundings and not updating the icon to indicate that you are feeling depressed Can be solved. That is, according to the system 1 of the present embodiment, it can be updated to an icon representing more accurate user motivation.

  The terminal device 11 included in the motivation visualization system 2 of FIG. 2 is configured as shown in FIG. 16, for example. FIG. 16 is a functional block diagram of another example of the terminal device. The terminal device 11 of FIG. 16 includes an imaging unit 51, a UI processing unit 53, a receiving unit 61, a still image converting unit 62, a face extracting unit 63, a part extracting unit 64, a facial expression recognizing unit 65, an indexing unit 66, and an icon determining unit. 67, a transmission unit 68, an expression DB 69, an index DB 70, a history DB 71, and an icon DB 72.

  The terminal device 11 in FIG. 16 has a configuration in which the functional blocks of the server device 10 and the terminal device 11 in FIG. 5 are combined. The functional blocks of the terminal device 11 in FIG. 16 perform the same processing as the functional blocks of the server device 10 and the terminal device 11 in FIG.

  The photographing unit 51 of the terminal device 11 photographs the user's face. The UI processing unit 53 displays the UI on the output device 32. The receiving unit 54 receives the icon update request described above from another terminal device 11. The UI processing unit 53 displays or updates an icon representing the user's motivation.

  The still image converting unit 62 converts the moving image captured by the image capturing unit 51 into a still image. The face extraction unit 63 extracts a face from the still image that the still image converting unit 62 converts to a still image. The part extracting unit 64 extracts each part from the face on the still image extracted by the face extracting unit 63.

  The facial expression recognition unit 65 refers to the facial expression DB 69 and recognizes the facial expression of each part from the shape (contour etc.) of the facial part on the still image extracted by the part extraction unit 64. The indexing unit 66 refers to the index DB 70, indexes the facial expressions of each part recognized by the facial expression recognition unit 65, and calculates the average of the indexes. The indexing unit 66 stores the average of the calculated indices in the history DB 71 described above.

  The icon determination unit 67 refers to the icon DB 72 and determines an icon corresponding to the average of the indexes calculated by the indexing unit 66. The icon determined by the icon determination unit 67 is displayed on the UI by the UI processing unit 53. In addition, the transmission unit 68 transmits an icon update request for an icon representing the user's motivation determined by the icon determination unit 67 to the other terminal device 11.

  Note that the user index history stored in the history DB 71 is transmitted to another terminal device 11 when a reference request is received from another terminal device 11. Then, the UI processing unit 53 of the other terminal device 11 displays the received user index history. Further, the history of the user index stored in the history DB 71 is displayed on the UI when a reference request is received from the UI processing unit 53. For example, the user can refer to the history of the index of the user displayed on the UI.

  The present invention is not limited to the specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims.

  The device described in the claims corresponds to the device terminal 11. The server device corresponds to the server device 10. The network system corresponds to the motivation visualization system 1. The acquisition unit corresponds to the photographing unit 51 and the UI processing unit 53. The face area extracting means corresponds to the face extracting unit 63. The identification means corresponds to the part extraction unit 64, facial expression recognition unit 65, and indexing unit 66. The storage means corresponds to the icon DB 72. The specifying means corresponds to the icon determining unit 67. The display means corresponds to the UI processing unit 53. The part extracting means corresponds to the part extracting unit 64. The facial expression storage means corresponds to the facial expression DB 69. The recognition means corresponds to the facial expression recognition unit 65 and the indexing unit 66. The first transmission means corresponds to the transmission unit 52. The first receiving unit corresponds to the receiving unit 61. The second transmission means corresponds to the transmission unit 68. The second receiving unit corresponds to the receiving unit 54. The transmission means corresponds to the transmission unit 52. The receiving means corresponds to the receiving unit 54.

1, 2 Motivation visualization system 10 Server device 11 Terminal device 12 Network 21, 31 Input device 22, 32 Output device 23, 33 Recording medium reader 24, 34 Auxiliary storage device 25, 35 Main storage device 26, 36 Arithmetic processing device 27 37 Interface device 28, 39 Bus 29, 40 Recording medium 38 Camera 51 Imaging unit 52 Transmission unit 53 UI processing unit 54, 61 Reception unit 62 Still image forming unit 63 Face extraction unit 64 Site extraction unit 65 Expression recognition unit 66 Indexing Part 67 icon determination part 68 transmission part 69 facial expression DB
70 Indicator DB
71 History DB
72 Icon DB

JP 2004-192297 A

Claims (12)

A network system in which a first device and a second device are connected,
First storage means for storing facial expression identification information indicating the facial expression and display information in association with each other for each facial expression;
Obtaining means for obtaining user information and image data in the first device;
Face area extraction means for extracting a face area included in the image data from the image data acquired by the acquisition means;
Identifying means for identifying facial expressions of the face area extracted by the face area extracting means from a plurality of facial expressions;
Second storage means for storing facial expression identification information corresponding to the facial expression identified by the identification means, user information obtained by the obtaining means, and date information indicating a date in association with each other;
From among the first of the plurality of display information relevant to stored respectively for each facial expression in the storage unit, the display being stored in association with the expression identification information stored in said second storage means specifying means for specifying information respectively,
Display means for displaying the display information specified by the specifying means and the user information acquired by the acquisition means in association with the display unit of the second device, wherein each of the display information And a display unit that displays the data in a time series based on the date information stored in the second storage unit in association with the network. The identification means includes
Part extraction means for extracting the shape of the part from the face of the face area extracted by the face area extraction means;
Facial expression storage means for storing the shape of the part and information indicating the facial expression in association with each other;
Recognizing means for recognizing information indicating the facial expression stored in association with the shape of the part extracted by the part extracting means from information indicating the facial expression stored in the facial expression storage means The network system according to claim 1, further comprising: The recognizing means stores at least one outline of eyes, eyebrows, and mouth as the shape of the part and information indicating at least one expression of the eyes, eyebrows, and mouth as information indicating facial expressions. The information indicating at least one facial expression of the eyes, eyebrows, and mouth stored in association with at least one contour of the eyes, eyebrows, and mouth extracted by the part extracting means using the facial expression storage means The network system according to claim 2, wherein information indicating facial expressions is recognized. The network system further includes a server device,
  The first device and the second device communicate via the server device.
The network system according to any one of claims 1 to 3, wherein: A network system in which a first device and a second device are connected,
The first device includes:
Acquisition means for acquiring user information and image data;
Face area extraction means for extracting a face area included in the image data from the image data acquired by the acquisition means;
An identification means for identifying facial expressions of the face area extracted by the face area extraction means from a plurality of facial expressions;
Transmission means for transmitting information indicating the facial expression identified by the identification means and user information acquired by the acquisition means to the second device;
The second device includes:
For each expression of the multiple faces, a first storage means for storing in association with expression identification information and the display information indicating a facial expression of the face,
Second storage means for storing facial expression identification information and user information corresponding to facial expressions transmitted from the first device in association with date information indicating a date;
The display stored in association with the facial expression identification information stored in the second storage means from among the display information stored in association with each of the plurality of facial expressions in the first storage means A specific means of identifying information,
Display means for displaying the display information specified by the specifying means and the user information acquired by the acquisition means in association with the display unit of the second device, wherein each of the display information And a display means for displaying the images in time series based on the date information stored in the second storage means in association with the network. The identification means includes
Part extraction means for extracting the shape of the part from the face of the face area extracted by the face area extraction means;
Facial expression storage means for storing the shape of the part and information indicating the facial expression in association with each other;
Recognizing means for recognizing information indicating the facial expression stored in association with the shape of the part extracted by the part extracting means from information indicating the facial expression stored in the facial expression storage means The network system according to claim 5, further comprising: The recognizing means stores at least one outline of eyes, eyebrows, and mouth as the shape of the part and information indicating at least one expression of the eyes, eyebrows, and mouth as information indicating facial expressions. The information indicating at least one facial expression of the eyes, eyebrows, and mouth stored in association with at least one contour of the eyes, eyebrows, and mouth extracted by the part extracting means using the facial expression storage means The network system according to claim 6, wherein information indicating facial expressions is recognized. The display information is an icon
The network system according to claim 1, wherein: A first device that acquires user information and image data, and display information that indicates facial expressions of a facial area included in the image data acquired by the first device and the user information are associated with each other on the display unit. An information processing apparatus having any function with the second apparatus to be displayed,
When functioning as the first device,
Acquisition means for acquiring user information and image data;
Face area extraction means for extracting a face area included in the image data from the image data acquired by the acquisition means;
An identification means for identifying facial expressions of the face area extracted by the face area extraction means from a plurality of facial expressions;
Transmitting means for transmitting the information indicating the facial expression identified by the identifying means and the user information acquired by the acquiring means to another information processing device functioning as the second device;
When functioning as the second device,
For each expression of the multiple faces, a first storage means for storing in association with expression identification information and the display information indicating a facial expression of the face,
Receiving means for receiving facial information and user information transmitted from another information processing apparatus functioning as the first apparatus;
Second storage means for storing facial expression identification information corresponding to the facial expression identified by the identification means, user information obtained by the obtaining means, and date information indicating a date in association with each other;
From among the first of the plurality of display information relevant to stored respectively for each facial expression in the storage unit, the display being stored in association with the expression identification information stored in said second storage means and identifying means for identifying the information, respectively,
Display means for displaying the display information specified by the specifying means and the user information acquired by the acquisition means in association with the display unit of the second device, wherein each of the display information An information processing apparatus comprising: a display unit configured to display the data in time series based on date information stored in the second storage unit in association with The display information is an icon
The information processing apparatus according to claim 9. An information processing apparatus that associates display information indicating facial expressions of a face area included in image data acquired by the first apparatus that acquires user information and image data and the user information with the user information A program that functions as any of the second devices displayed on the display unit,
The program includes an acquisition unit that acquires user information and image data from the information processing apparatus,
A face area extracting means for extracting a face area included in the image data from the image data acquired by the acquiring means;
Identifying means for identifying facial expressions of the facial area extracted by the facial area extracting means from a plurality of facial expressions;
1st apparatus provided with the transmission means which transmits the information which shows the facial expression identified by the said identification means, and the user information acquired by the said acquisition means to the other information processing apparatus which functions as said 2nd apparatus Function as
Further, the program causes the information processing apparatus to
First storage means for storing, in association with each other, facial expression identification information indicating the facial expression and display information for each facial expression;
Receiving means for receiving information indicating facial expressions and user information transmitted from another information processing device functioning as the first device;
Second storage means for storing the facial expression identification information corresponding to the facial expression identified by the identification means, the user information acquired by the acquisition means, and date information indicating the date in association with each other;
From among the first storage means in the plurality of display information relevant to stored respectively for each facial expression, from the displayed information stored in the second storage means, said second storage specifying means for specifying respective display information stored in association with the expression identification information stored in means,
Display means for displaying the display information specified by the specifying means and the user information acquired by the acquisition means in association with the display unit of the second device, wherein each of the display information A program that functions as the second device including display means that displays the data in time series based on the date information stored in the second storage means in association with the second storage means . The display information is an icon
The program according to claim 11.

Images