JP2021157681A - Simplified communication system, simplified communication method and program - Google Patents

Abstract

To properly transmit information under a situation in which a user is not able to use his/her hand.SOLUTION: A simplified communication system 1 comprises a display device, when being mounted on a head of the user, having: a first photographing section capable of photographing a first image including at least a lip of the user; a second photographing section capable of photographing a second image including an eye of the user; and a display section for displaying an image, and a portable terminal 10. The portable terminal 10 comprises: a word identification section 151 for detecting movement of the lip on the basis of the first image and identifying a word indicated by the detected movement of the lip; an expression identification section 152 for identifying expression of the user on the basis of the first image; a recommendation processing section 153 for acquiring candidates of stamps from a candidate storage section for storing the word, the expression and the stamps in association with each other on the basis of the identified word and expression, and displaying an image including the candidates of stamps on the display section; and a transmission processing section 156 for selecting a stamp from among the candidates of stamps according to a line of vision of the user detected on the basis of the second image and transmitting the selected stamp.SELECTED DRAWING: Figure 2

Description

The present invention relates to a simple communication system, a simple communication method, and a program.

In recent years, a technique for detecting voice and lip movement and inputting a message or the like has been known (see, for example, Patent Document 1). In such a conventional technique, it is possible to input a message or the like by using, for example, a microphone and a camera of a mobile terminal such as a smartphone held in the hand.

Japanese Unexamined Patent Publication No. 2019-208138

However, in the above-mentioned conventional technology, it is necessary to hold a mobile terminal such as a smartphone in front of the user's face, and it is not possible to input a message or the like in a situation where space is limited such as a crowded train. In some cases, it was difficult for the user to send the information.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a simple communication system, a simple communication method, and a program capable of appropriately transmitting information in a situation where the user's hand cannot be used. To do.

In order to solve the above problem, one aspect of the present invention is a display device that can be worn on the user's head, and when the display device is worn on the user's head, at least the user's lips are worn. A display device having a first image pickup unit capable of capturing a first image including the first image, a second image pickup unit capable of capturing a second image including the user's eyes, and a display unit for displaying an image is connected to a network. The mobile terminal includes a possible mobile terminal, and the mobile terminal detects the movement of the lip based on the first image captured by the first imaging unit, and based on the detected movement of the lip, the mobile terminal of the lip. An illustration expressing the word, the facial expression, and the emotion of the user, the word identification unit that identifies the word indicated by the movement, the facial expression identification unit that identifies the facial expression of the user based on the first image, and the word, the facial expression, and the emotion of the user. From the candidate storage unit that stores the stamp in association with the stamp, the stamp candidate is acquired based on the word identified by the word identification unit and the facial expression identified by the facial expression identification unit, and the stamp is obtained. It is displayed on the display unit by the recommended processing unit for displaying the image including the candidate of the above on the display unit and the line of sight of the user detected based on the second image captured by the second imaging unit. It is a simple communication system including a transmission processing unit that selects the stamp from the candidates for the stamp and transmits the selected stamp to a predetermined destination via the network.

Further, one aspect of the present invention is a transmission that stores past transmission history information in which the transmission date and time, the transmitted stamp, the transmission destination, the word, and the facial expression are associated with each other in the simple communication system. The recommended processing unit includes a history storage unit, determines the priority of the stamp candidates based on the transmission history information stored in the transmission history storage unit, arranges the stamp candidates according to the priority, and arranges the stamp candidates. It is characterized in that stamp candidates are displayed on the display unit.

Further, in one aspect of the present invention, in the above-mentioned simple communication system, the transmission history information includes the place information where the stamp is transmitted, and the recommended processing unit is currently based on the transmission history information. It is characterized in that the priority of the stamp candidate corresponding to the location of the mobile terminal is raised.

Further, in one aspect of the present invention, in the simple communication system, the transmission history information includes schedule information indicating a situation when the stamp is transmitted, and the recommended processing unit is the user. Based on the transmission history information, the priority of the stamp candidate corresponding to the user's situation based on the current schedule is raised.

Further, in one aspect of the present invention, in the simple communication system, the transmission processing unit additionally stores the transmission history information corresponding to the transmitted stamp in the transmission history storage unit, and stores the recommended processing. The unit is characterized in that the priority of the stamp candidate with respect to the most recently transmitted stamp is lowered.

Further, in one aspect of the present invention, in the simple communication system, the recommended processing unit gives priority to the stamp candidate as the frequency of use of the stamp increases based on the transmission history information of the user. It is characterized by raising the ranking.

Further, in one aspect of the present invention, in the simple communication system, the candidate storage unit stores the word, the facial expression, the stamp, and the group information indicating the group of the stamps in association with each other. The recommended processing unit is characterized in that, based on the transmission history information of the user, the priority of the stamp candidate, which matches the group information of the stamp transmitted last time, is raised.

Further, in one aspect of the present invention, in the above-mentioned simple communication system, the candidate storage unit stores stamp table information in which the word, the expression, and the stamp are associated with each user, and the above-mentioned The candidate storage unit is provided with a registration processing unit that stores and registers the stamp table information of the user.

Further, one aspect of the present invention includes a reception processing unit for displaying a message received via the network on the display unit in the simple communication system, and the recommended processing unit is for the received message. The display unit displays the image including the stamp candidate, and the transmission processing unit transmits the selected stamp to a predetermined destination including the source of the received message.

Further, one aspect of the present invention is a display device that can be worn on the user's head, and when worn on the user's head, captures a first image including at least the user's lips. A display device having a possible first image pickup unit, a second image pickup unit capable of capturing a second image including the user's eyes, a display unit for displaying an image, and a portable terminal connectable to a network. A simple communication method of a simple communication system provided, wherein the mobile terminal detects the movement of the lip based on the first image captured by the first image pickup unit, and based on the detected movement of the lip. , A word identification step for identifying a word indicated by the movement of the lips, an emotion identification step for the mobile terminal to identify the facial expression of the user based on the first image, and the mobile terminal with the word. , The word identified by the word identification step and the word identified by the emotion identification step from a candidate storage unit that stores the facial expression and a stamp which is an illustration expressing the user's feelings in association with each other. A recommended processing step of acquiring a stamp candidate based on a facial expression and displaying the image including the stamp candidate on the display unit, and the second image captured by the second imaging unit by the mobile terminal. The stamp is selected from the stamp candidates displayed on the display unit based on the line of sight of the user detected based on the image, and the selected stamp is sent to a predetermined destination via the network. This is a simple communication method including a transmission processing step of transmitting the image.

Further, one aspect of the present invention is a display device that can be worn on the user's head, and when worn on the user's head, images a first image including at least the user's lips. A display device having a possible first image pickup unit, a second image pickup unit capable of capturing a second image including the user's eyes, a display unit for displaying an image, and a portable terminal connectable to a network. A computer of a simple communication system provided with the device detects the movement of the lip based on the first image captured by the first image pickup unit, and outputs a word indicated by the movement of the lip based on the detected movement of the lip. A word identification step for identifying, an emotion identification step for identifying the user's facial expression based on the first image, the word, the facial expression, and a stamp which is an illustration expressing the user's feelings. From the candidate storage unit that is stored in association with each other, the stamp candidate is acquired based on the word identified by the word identification step and the facial expression identified by the emotion identification step, and the stamp candidate is obtained. The stamp displayed on the display unit by the recommended processing step of displaying the including image on the display unit and the line of sight of the user detected based on the second image imaged by the second imaging unit. This is a program for selecting the stamp from the candidates of the above and executing a transmission processing step of transmitting the selected stamp to a predetermined destination via the network.

According to the present invention, information can be appropriately transmitted in a situation where the user's hand cannot be used.

It is a system block diagram which shows an example of the simple communication system by 1st Embodiment. It is a functional block diagram which shows an example of the simple communication system by 1st Embodiment. It is a figure which shows the data example of the stamp storage part in 1st Embodiment. It is a flowchart which shows an example of the operation of the simple communication system by 1st Embodiment. It is a figure which shows an example of the message reception screen in 1st Embodiment. It is a figure which shows an example of the stamp candidate selection screen in 1st Embodiment. It is a figure which shows an example of the transmission completion screen of a stamp in 1st Embodiment. It is a functional block diagram which shows an example of the simple communication system by 2nd Embodiment. It is a figure which shows the data example of the transmission history storage part in 2nd Embodiment. It is a flowchart which shows an example of the operation of the simple communication system by 2nd Embodiment. It is a figure which shows the modification of the stamp storage part.

Hereinafter, a simple communication system and a simple communication method according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a system configuration diagram showing an example of a simple communication system 1 according to the first embodiment. Further, FIG. 2 is a functional block diagram showing an example of the simple communication system 1 according to the first embodiment.

As shown in FIGS. 1 and 2, the simple communication system 1 includes a mobile terminal 10, a glasses-type display device 20, and a server device 30.
In the simple communication system 1 according to the present embodiment, an example of using a message application (hereinafter referred to as a message application) for transmitting and receiving messages between users will be described. Further, in the present embodiment, in addition to the message, the message application can send and receive stamps, which are illustrations expressing the feelings of the users, to and from each other. Here, the emotion indicates, for example, the thoughts and feelings in the mind.

As shown in FIG. 1, the spectacle-type display device 20 (an example of the display device) is a spectacle-type display with a camera that can be worn on the head of the user U1, and displays the lip camera 21 and the line-of-sight camera 22. A unit 23 and a NW (network) communication unit 24 are provided. The eyeglass-type display device 20 can be connected to the network NW1 by wireless communication, for example, and communicates with the mobile terminal 10 via the network NW1.

The lip camera 21 (an example of the first imaging unit) is a camera for detecting the movement and facial expression of the lips of the user U1 and is used when the eyeglass-type display device 20 is attached to the head of the user U1. The image around the lips of the person U1 (for example, the lower half of the face) is arranged so as to be able to be captured. When the eyeglass-type display device 20 is attached to the head of the user U1, the lip camera 21 captures at least an image around the lips (first image) including the lips of the user U1. The lip image is, for example, an image obtained by capturing the lower half of the face of the user U1.

The line-of-sight camera 22 (an example of the second image pickup unit) is, for example, an eye tracking camera for detecting the line of sight of the user U1, and is used when the eyeglass-type display device 20 is attached to the head of the user U1. The image around the eyes of the person U1 (for example, the upper half of the face) is arranged so as to be able to be captured. When the eyeglass-type display device 20 is attached to the head of the user U1, the line-of-sight camera 22 captures an image around the eyes (second image) including the eyes of the user U1.

The display unit 23 is, for example, a transmissive liquid crystal display and displays various information. The display unit 23 is arranged so that the user U1 can visually recognize the display when the eyeglass-type display device 20 is attached to the head of the user U1.

The NW communication unit 24 connects to the network NW1 by wireless communication such as a wireless LAN (Local Area Network), and performs various communications via the network NW1. For example, the NW communication unit 24 transmits the images (image around the lips and the image around the eyes) captured by the lip camera 21 and the line-of-sight camera 22 to the mobile terminal 10, and also transmits the information to be displayed on the display unit 23 from the mobile terminal 10. Receive.

The server device 30 is an application server that provides services such as a message application, and stores various information. The server device 30 connects to the mobile terminal 10 via, for example, the network NW1 and communicates various information with the mobile terminal 10. The server device 30 includes, for example, a NW communication unit 31 and a stamp storage unit 32.

The NW communication unit 31 connects to the network NW1 and performs various communications via the network NW1. For example, the NW communication unit 24 transmits / receives information stored in the stamp storage unit 32, which will be described later, to / from the mobile terminal 10.

The stamp storage unit 32 (an example of the candidate storage unit) stores the registered stamp information (stamp table information). The stamp storage unit 32 stores, for example, a word, a facial expression, and a stamp in association with each other. Here, a data example of the stamp storage unit 32 will be described with reference to FIG.

FIG. 3 is a diagram showing a data example of the stamp storage unit 32 in the present embodiment.
As shown in FIG. 3, the stamp storage unit 32 stores stamp table information in which "user ID", "stamp ID", "stamp information", "word", and "facial expression" are associated with each other. ..

Here, the "user ID" is an example of user identification information that identifies the user U1, and the "stamp ID" is an example of stamp identification information that identifies the stamp. Further, the "stamp information" indicates the illustration image information which is the content of the stamp. Further, "word" indicates a word related to the stamp, and "facial expression" indicates a facial expression related to the stamp.

In the example shown in FIG. 3, for example, in the stamp whose "user ID" is "U001", the "word" corresponding to the stamp whose "stamp ID" is "AAA" and whose "stamp information" is "stamp A" is It is "like" and shows that "expression" is "joy".
The stamp storage unit 32 stores stamp table information for each user U1 (“user ID”).

Returning to the description of FIGS. 1 and 2, the mobile terminal 10 is, for example, a terminal device such as a smartphone or a tablet terminal, and is a terminal device that can be carried by the user U1. The mobile terminal 10 includes, for example, a NW communication unit 11, an input unit 12, a display unit 13, a terminal storage unit 14, and a terminal control unit 15.

The NW communication unit 11 connects to the network NW1 by wireless communication such as a wireless LAN or 4G (4th generation mobile communication system), and performs various communications via the network NW1. For example, the NW communication unit 11 receives the images (image around the lips and the image around the eyes) captured by the lip camera 21 and the line-of-sight camera 22 of the spectacle-type display device 20 from the spectacle-type display device 20 and displays the image on the display unit 23. The information to be displayed is transmitted to the glasses-type display device 20.

Further, the NW communication unit 11 receives, for example, the stamp table information stored in the stamp storage unit 32 of the server device 30 from the server device 30, and transmits the registration information of the stamp storage unit 32 to the server device 30.

The input unit 12 is, for example, an input device such as a touch sensor, and receives input of various information from the user U1.
The display unit 13 is a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays various information of the mobile terminal 10.
The input unit 12 and the display unit 13 are configured as, for example, a touch screen.

The terminal storage unit 14 stores various information used by the mobile terminal 10. The terminal storage unit 14 stores, for example, messages and stamps sent and received by the message application, various display information, and the like.

The terminal control unit 15 is, for example, a processor including a CPU (Central Processing Unit) and the like, and controls the mobile terminal 10 in an integrated manner. The terminal control unit 15 includes, for example, a word identification unit 151, a facial expression identification unit 152, a recommended processing unit 153, a line-of-sight detection unit 154, a reception processing unit 155, a transmission processing unit 156, and a stamp registration processing unit 157. It has.

The word identification unit 151 detects the movement of the lips based on the lip image captured by the lip camera 21, and identifies the word indicated by the movement of the lips based on the detected movement of the lips. The word identification unit 151 acquires, for example, a lip image captured by the lip camera 21 from the eyeglass-type display device 20 via the NW communication unit 11, and based on the acquired lip image, the movement of the lips of the user U1 is performed. To detect. The word identification unit 151 identifies a word indicated by the movement of the lips based on the detected movement of the lips by using, for example, an existing technique such as Patent Document 1. Here, the movement of the lips of the user U1 does not have to be accompanied by the utterance that the user U1 utters, and may be the movement of the lips that moves the mouth without making the mouth crisp and uttering.

The facial expression identification unit 152 identifies the facial expression of the user U1 based on the lip image captured by the lip camera 21. Here, the facial expression is a manifestation of the user's emotions, and indicates, for example, anger, joy, sadness, and the like. The facial expression identification unit 152 identifies the facial expression of the user U1 based on the acquired lip image using the existing technology.

The recommended processing unit 153 acquires stamp candidates from the stamp storage unit 32 described above based on the word identified by the word identification unit 151 and the facial expression identified by the facial expression identification unit 152, and an image including the stamp candidate. Is displayed on the display unit 23.

The recommended processing unit 153 refers to, for example, the stamp storage unit 32 of the server device 30, and extracts and extracts stamps matching the words and facial expressions identified by the word identification unit 151 and the facial expression identification unit 152 as stamp candidates. The stamp ID and stamp information are acquired from the stamp storage unit 32. The recommended processing unit 153 generates display image data including the acquired stamp information, transmits the display image data to the eyeglass-type display device 20 via the NW communication unit 11, and causes the display unit 23 to display the display image data. .. For example, when the reception processing unit 155, which will be described later, receives a message, the recommended processing unit 163 causes the display unit 23 to display an image including stamp candidates for the received message.

The line-of-sight detection unit 154 detects the line of sight of the user U1 based on the image around the eyes captured by the line-of-sight camera 22. The line-of-sight detection unit 154 acquires, for example, an image around the eyes captured by the line-of-sight camera 22 from the eyeglass-type display device 20 via the NW communication unit 11, and moves the pupil (eyeball) included in the acquired image around the eyes. Therefore, the line of sight of the user U1 is detected by using the existing technology.

The reception processing unit 155 causes the display unit 23 to display the message received via the network NW1. When the reception processing unit 155 receives a message via the NW communication unit 11, for example, the message application transmits the received message to the glasses-type display device 20 via the NW communication unit 11. It is displayed on the display unit 23.

The transmission processing unit 156 selects a stamp from the stamp candidates displayed on the display unit 23 based on the line of sight of the user U1 detected based on the second image captured by the line-of-sight camera 22, and the selected stamp. Is transmitted to a predetermined destination via the network NW1. The transmission processing unit 156 causes, for example, a mark such as a cross mark to be displayed at a position corresponding to the tip of the line of sight of the image displayed on the display unit 23 by the line of sight detected by the line of sight detection unit 154.

The transmission processing unit 156 selects a stamp by gazing at the stamp to be selected from the stamp candidates displayed on the display unit 23 for a predetermined period (for example, 3 seconds). The transmission processing unit 156 transmits the selected stamp to a predetermined destination (for example, a source of the received message or a predetermined group including the source) via the NW communication unit 11.

The stamp registration processing unit 157 (an example of the registration processing unit) stores and registers the stamp table information of the user U1 in the stamp storage unit 32. As shown in FIG. 3 described above, the stamp storage unit 32 stores stamp table information in which words, facial expressions, and stamps are associated with each user U1 (for example, for each user ID). .. When registering a unique stamp for each user U1, the stamp registration processing unit 157 transfers the stamp table information of the designated stamp to the stamp table information of the designated stamp via the NW communication unit 11, for example, in response to the operation of the input unit 12. It is transmitted to the server device 30, and the transmitted stamp table information is added and stored in the stamp storage unit 32.

Next, the operation of the simplified communication system 1 according to the present embodiment will be described with reference to the drawings.
FIG. 4 is a flowchart showing an example of the operation of the simple communication system 1 according to the present embodiment.

As shown in FIG. 4, when the mobile terminal 10 of the simple communication system 1 receives the notification of the message by the message application, the mobile terminal 10 executes the message reception process (step S101). The reception processing unit 155 of the mobile terminal 10 receives a message from, for example, the server device 30 via the NW communication unit 11.

Next, the reception processing unit 155 displays the received message on the display unit 23 of the eyeglass-type display device 20 (step S102). The reception processing unit 155 transmits the received message to the eyeglass-type display device 20 via the NW communication unit 11, and causes the display unit 23 to display the message, for example, as shown in FIG.

FIG. 5 is a diagram showing an example of a message reception screen in the present embodiment.
The display screen G1 shown in FIG. 5 is a screen displayed by the display unit 23 of the eyeglass-type display device 20. The display unit 23 displays the message received by the mobile terminal 10 by the message application like the display screen G1.

Next, the mobile terminal 10 acquires an image of the lip camera 21 (step S103). The word identification unit 151 (or facial expression identification unit 152) of the mobile terminal 10 acquires the lip image captured by the lip camera 21 from the glasses-type display device 20 via the NW communication unit 11.

Next, the word identification unit 151 identifies the word indicated by the movement of the lips from the image (step S104). The word identification unit 151 detects the movement of the lips from the lip image acquired from the eyeglass-type display device 20, and identifies the word indicated by the movement of the lips based on the detected movement of the lips.

Next, the facial expression identification unit 152 of the mobile terminal 10 identifies the facial expression of the user U1 from the image (step S105). The facial expression identification unit 152 identifies the facial expression of the user U1 who shows emotions such as anger, joy, and sadness from the lip image acquired from the eyeglass-type display device 20 by using the existing technology.

Next, the recommended processing unit 153 of the mobile terminal 10 acquires stamp candidates from the stamp storage unit 32 based on the words and facial expressions (step S106). The recommended processing unit 153 refers to, for example, the stamp storage unit 32 of the server device 30, and extracts and extracts stamps matching the words and facial expressions identified by the word identification unit 151 and the facial expression identification unit 152 as stamp candidates. The stamp ID and stamp information are acquired from the stamp storage unit 32.

Next, the recommended processing unit 153 causes the display unit 23 of the eyeglass-type display device 20 to display the stamp candidates (step S107). The recommended processing unit 153 generates display image data including the acquired stamp information, transmits the display image data to the eyeglass-type display device 20 via the NW communication unit 11, and causes the display unit 23 to display the display image data. .. The display unit 23 of the eyeglass-type display device 20 displays stamp candidates, for example, as shown in FIG.

FIG. 6 is a diagram showing an example of a stamp candidate selection screen in the present embodiment.
The display screen G2 shown in FIG. 6 is a screen displayed by the display unit 23 of the eyeglass-type display device 20. The display unit 23 displays a plurality of stamp candidates (for example, stamp STP1, stamp STP2, stamp STP3, ...) Received from the mobile terminal 10 like the display screen G2.

Next, the line-of-sight detection unit 154 of the mobile terminal 10 acquires the image of the line-of-sight camera 22 and detects the line of sight of the user U1 (step S108). The line-of-sight detection unit 154 acquires an image around the eyes captured by the line-of-sight camera 22 from the eyeglass-type display device 20 via the NW communication unit 11. The line-of-sight detection unit 154 detects the line of sight of the user U1 by using the existing technology from the movement of the pupil (eyeball) included in the acquired image around the eyes.

Next, the transmission processing unit 156 of the mobile terminal 10 displays the line-of-sight mark on the image ahead of the line of sight of the user U1 (step S109). For example, as shown in FIG. 6, the transmission processing unit 156 displays the cross mark MK1 as a line-of-sight mark on the display screen G2 of the display unit 23 ahead of the line of sight of the user U1.

Next, the transmission processing unit 156 determines whether or not the stamp determination operation is performed (step S110). The transmission processing unit 156 determines whether or not the stamp determination operation is performed by detecting the line of sight by the line-of-sight detection unit 154, for example, depending on whether or not the stamp candidate has been watched for 3 seconds. The transmission processing unit 156 advances the processing to step S111, for example, when the stamp candidate is watched for 3 seconds (step S110: YES). Further, the transmission processing unit 156 returns the processing to step S108, for example, when the stamp candidate is not watched for 3 seconds (step S110: NO).

In step S111, the transmission processing unit 156 transmits the selected stamp to a destination (predetermined destination) including the source of the message. The recommended processing unit 153 returns the selected stamp to a destination (predetermined destination) including the source of the message via the NW communication unit 11. After the processing of step S111, the transmission processing unit 156 ends the processing.

Note that FIG. 7 is a diagram showing an example of a stamp transmission completion screen in the present embodiment.
The display screen G3 shown in FIG. 7 is a screen displayed by the display unit 23 of the eyeglass-type display device 20, and the display unit 23 displays the stamp STP4 transmitted by the mobile terminal 10 like the display screen G2.

As described above, the simple communication system 1 according to the present embodiment includes a mobile terminal 10 that can be connected to the network NW1 and a glasses-type display device 20 (display device) that can be worn on the head of the user U1. The eyeglass-type display device 20 includes a lip camera 21 (first imaging unit) capable of capturing at least a first image including the lips of the user U1 when worn on the head of the user U1 and the user U1. It has a line-of-sight camera 22 (second imaging unit) capable of capturing a second image including eyes, and a display unit 23 for displaying an image. The mobile terminal 10 includes a word identification unit 151, a facial expression identification unit 152, a recommended processing unit 153, and a transmission processing unit 156. The word identification unit 151 detects the movement of the lips based on the first image captured by the lip camera 21, and identifies the word indicated by the movement of the lips based on the detected movement of the lips. The facial expression identification unit 152 identifies the facial expression of the user U1 based on the first image. The recommended processing unit 153 acquires stamp candidates from the stamp storage unit 32 (candidate storage unit) based on the word identified by the word identification unit 151 and the facial expression identified by the facial expression identification unit 152, and obtains stamp candidates. The image including the above is displayed on the display unit 23. Here, the stamp storage unit 32 stores words, facial expressions, and stamps, which are illustrations expressing the emotions of the user U1, in association with each other. The transmission processing unit 156 selects a stamp from the stamp candidates displayed on the display unit 23 based on the line of sight of the user U1 detected based on the second image captured by the line-of-sight camera 22, and the selected stamp. Is transmitted to a predetermined destination (for example, the source of the received message) via the network NW1.

As a result, the simple communication system 1 according to the present embodiment identifies the word intended by the user U1 from the movement of the lips of the user U1 in a situation where the hand of the user U1 cannot be used, for example, in a crowded train. At the same time, the facial expression is identified from the image around the lips, and stamp candidates are displayed on the display unit 23 based on the word and the facial expression. Then, the simple communication system 1 according to the present embodiment selects a stamp from the stamp candidates by detecting the line of sight, and transmits the selected stamp to a predetermined destination, so that an appropriate stamp can be transmitted. can. Therefore, the simple communication system 1 according to the present embodiment can appropriately transmit information (for example, a stamp) in a situation where the hand of the user U1 cannot be used.

Further, in the present embodiment, the stamp storage unit 32 stores stamp table information in which words, facial expressions, and stamps are associated with each user U1. The simple communication system 1 (mobile terminal 10) includes a stamp registration processing unit 157 (registration processing unit) in which the stamp storage unit 32 stores and registers the stamp table information of the user U1.

As a result, the simple communication system 1 according to the present embodiment can register a unique stamp for each user U1. For example, in a situation where the user U1's hand cannot be used, the user U1's own unique stamp can be registered. You can send a stamp. Therefore, the simple communication system 1 according to the present embodiment can increase the degree of freedom in selecting stamps and can improve convenience.

Further, the simple communication system 1 according to the present embodiment includes a reception processing unit 155 that displays a message received via the network NW1 on the display unit 23. The recommended processing unit 163 causes the display unit 23 to display an image including stamp candidates for the received message. The transmission processing unit 156 transmits the selected stamp to a predetermined destination including the source of the received message.

As a result, the simple communication system 1 according to the present embodiment can appropriately return a stamp to the received message in a situation where the hand of the user U1 cannot be used, for example.

Further, the simple communication method according to the present embodiment is a simple communication method of the simple communication system 1 including the above-mentioned mobile terminal 10 and the glasses-type display device 20, and is a word identification step, an emotion identification step, and a recommended process. Includes a step and a transmission processing step. In the word identification step, the mobile terminal 10 detects the movement of the lips based on the first image captured by the lip camera 21, and identifies the word indicated by the movement of the lips based on the detected movement of the lips. In the emotion identification step, the mobile terminal 10 identifies the facial expression of the user U1 based on the first image. In the recommended processing step, the mobile terminal 10 acquires stamp candidates from the stamp storage unit 32 based on the words identified by the word identification step and the facial expressions identified by the emotion identification step, and obtains stamp candidates. The including image is displayed on the display unit 23. In the transmission processing step, the mobile terminal 10 selects a stamp from the stamp candidates displayed on the display unit 23 based on the line of sight of the user U1 detected based on the second image captured by the line-of-sight camera 22. , The selected stamp is transmitted to a predetermined destination via the network NW1.

As a result, the simple communication method according to the present embodiment has the same effect as the simple communication system 1 described above, and can appropriately transmit information (for example, a stamp) in a situation where the hand of the user U1 cannot be used. ..

[Second Embodiment]
Next, the simple communication system 1a according to the second embodiment will be described with reference to the drawings. In the present embodiment, a modified example of prioritizing stamp candidates by using the past stamp transmission history will be described.

FIG. 8 is a functional block diagram showing an example of the simple communication system 1a according to the second embodiment.
As shown in FIG. 8, the simple communication system 1a includes a mobile terminal 10a, a glasses-type display device 20, and a server device 30a.
In this figure, the same reference numerals are given to the same configurations as those in FIGS. 1 and 2, and the description thereof will be omitted here.

The server device 30a is an application server that provides services such as a message application, and stores various information. The server device 30a connects to the mobile terminal 10a via, for example, the network NW1 and communicates various information with the mobile terminal 10a. The server device 30a includes, for example, a NW communication unit 31, a stamp storage unit 32, and a transmission history storage unit 33.

The transmission history storage unit 33 stores the transmission history information of the stamp by the mobile terminal 10a. The transmission history storage unit 33 stores, for example, past transmission history information in which a transmission date and time, a transmission stamp, a transmission destination, a word, and a facial expression are associated with each other. Here, a data example of the transmission history storage unit 33 will be described with reference to FIG.

FIG. 9 is a diagram showing a data example of the transmission history storage unit 33 in the present embodiment.
As shown in FIG. 9, the transmission history storage unit 33 has a "user ID", a "date and time", a "stamp ID", a "destination", a "word", a "facial expression", and a "location". And the history information associated with the "schedule" is stored.

Here, the "date and time" indicates the date and time when the stamp was transmitted, and the "stamp ID" is an example of stamp identification information for identifying the stamp. Further, "word" and "facial expression" indicate the recognition result of the word and facial expression by the mobile terminal 10a when the stamp is transmitted, and "place" is the position related to the place where the user U1 was when the stamp was transmitted. Shows information. Further, the "schedule" is schedule information indicating the status of the user U1 based on the schedule of the user U1 when the stamp is transmitted. The "schedule" information is set based on the information registered in the schedule storage unit 141 of the mobile terminal 10a, which will be described later. The "schedule" information may be the registered information as it is.

In the example shown in FIG. 9, for example, the "date and time" is "2020/01/20 10:00" (January 20, 2020 10:00), the "stamp ID" is "AAA", and the stamp is ". Indicates that the message is being sent to "○○" in "Destination". Also, when this stamp is sent, the "word" is "like", the "expression" is "boring", the "location" is "XXX city", and the "schedule" is "work". .. That is, this stamp indicates that the user U1 was transmitted during work when he was in the "XXX city".

Returning to the description of FIG. 8, the mobile terminal 10a is, for example, a terminal device such as a smartphone or a tablet terminal, and is a terminal device that can be carried by the user U1. The mobile terminal 10a includes, for example, a NW communication unit 11, an input unit 12, a display unit 13, a terminal storage unit 14a, a terminal control unit 15a, and a position information detection unit 16.

The terminal storage unit 14a stores various information used by the mobile terminal 10a. The terminal storage unit 14a includes a schedule storage unit 141.
The schedule storage unit 141 stores the schedule of the user U1 registered in advance. The schedule storage unit 141 stores, for example, the date and time information and the schedule contents (information indicating the status of the user U1 such as "work" and "hobby activity") in association with each other. The schedule stored in the schedule storage unit 141 is used when prioritizing the above-mentioned transmission history information and stamp candidates.

The position information detection unit 16 detects the position information of the mobile terminal 10a (user U1) by, for example, GPS (Global Positioning System). The position information detected by the position information detection unit 16 is used when prioritizing the above-mentioned transmission history information and stamp candidates.

The terminal control unit 15a is, for example, a processor including a CPU and the like, and controls the mobile terminal 10a in an integrated manner. The terminal control unit 15a includes, for example, a word identification unit 151, a facial expression identification unit 152, a recommended processing unit 153a, a line-of-sight detection unit 154, a reception processing unit 155, a transmission processing unit 156a, and a stamp registration processing unit 157. It has. The function of the terminal control unit 15a is basically the same as that of the terminal control unit 15 of the first embodiment described above, but in the present embodiment, a part of the processing of the recommended processing unit 153a and the transmission processing unit 156a is performed. different.

The recommended processing unit 153a determines the priority of stamp candidates based on the transmission history information stored in the transmission history storage unit 33, arranges the stamp candidates according to the priority, and displays the stamp candidates on the display unit 23. Let me. For example, the recommended processing unit 153a sets the upper left of the display as the highest priority, and the stamp has a lower priority from the left to the right of the display and a lower priority from the top to the bottom of the display. The candidate is displayed on the display unit 23.

Here, as for the priority, the first is the highest priority, and the larger the number of the rank, the lower the priority. That is, it is assumed that the higher priority has a smaller rank number than the lower priority, and the lower priority has a higher rank number than the higher priority.
Further, in the present embodiment, "raising the priority" means raising the priority, and "lowering the priority" means lowering the priority.

For example, the recommended processing unit 153a places the highest priority on the stamp candidates in which both the identified "word" and "facial expression" match, and only one of the "word" and "facial expression" matches. The priority of stamp candidates may be lowered (lowered). In addition, it may be decided in advance which of "word" and "facial expression" is prioritized.

Further, the transmission history information of the transmission history storage unit 33 described above includes the location information at which the stamp was transmitted, and the recommended processing unit 153a has the stamp corresponding to the current location of the mobile terminal 10a based on the transmission history information. Executes the process of raising (raising) the priority of the candidates. The recommended processing unit 153a determines, for example, the current location of the mobile terminal 10a based on the location information detected by the location information detection unit 16, and the determined location matches the "location" included in the transmission history information. Or try to raise (higher) the priority of those that are close.

Further, the transmission history information of the transmission history storage unit 33 described above includes schedule information indicating the status when the stamp is transmitted, and the recommended processing unit 153a is based on the transmission history information of the user U1 at present. A process of raising (higher) the priority of stamp candidates corresponding to the situation of the user U1 based on the schedule is executed. The recommended processing unit 153a determines the current schedule information based on the schedule of the current user U1 registered in the schedule storage unit 141, for example, and the "schedule" included in the determined schedule information and the transmission history information. Priority is given (higher) to those that match or are close to.

Further, the recommended processing unit 153a may raise (higher) the priority of stamp candidates as the stamp usage frequency increases, based on the transmission history information of the user U1, or the same continuously. In order not to send stamps, the priority of stamp candidates for the most recently sent stamp may be lowered (lowered).

For example, when the destination to which the stamp is to be transmitted is "XX", the word is "like", and the facial expression is "boring", the recommended processing unit 153a has the transmission history storage unit 33. Based on the transmission history information to be stored, the priority order is determined as follows, and the stamp candidates are displayed on the display unit 23. The recommended processing unit 153a is used, for example, when the destination is "XX" and the top XX (for example, 5) stamp candidates that are frequently used and the word is "Like". The top XX candidates for stamps with high frequency (for example, 5) and the top XX candidates for stamps with high frequency of use when the facial expression is "boring" (for example, 5) are in order. Is displayed on the display unit 23 as a stamp candidate.

The basic function of the transmission processing unit 156a is the same as that of the transmission processing unit 156 of the first embodiment described above. The transmission processing unit 156a adds a process of adding and storing the transmission history information corresponding to the transmitted stamp to the transmission history storage unit 33. After transmitting the stamp, the transmission processing unit 156a transmits, for example, transmission history information corresponding to the transmitted stamp as shown in FIG. 9 to the server device 30a via the NW communication unit 11, and the transmission history information. Is stored in the transmission history storage unit 33.

Next, the operation of the simplified communication system 1a according to the present embodiment will be described with reference to the drawings.
FIG. 10 is a flowchart showing an example of the operation of the simple communication system 1a according to the present embodiment.

Since the processing from step S201 to step S206 shown in FIG. 10 is the same as the processing from step S101 to step S106 shown in FIG. 4 described above, the description thereof will be omitted here.
In step S207, the recommended processing unit 153a of the mobile terminal 10a determines the priority of stamp candidates based on the transmission history information. That is, the recommended processing unit 153a determines the priority of stamp candidates as described above based on the transmission history information stored in the transmission history storage unit 33.

Next, the recommended processing unit 153a causes the display unit 23 of the eyeglass-type display device 20 to display the stamp candidates (step S208). The recommendation processing unit 153a arranges the stamp candidates according to the priority (for example, the priority is lowered as the stamp candidates are moved from the left to the right of the display, and the priority is lowered as the stamp candidates are moved from the top to the bottom of the display. It is displayed on the display unit 23 in the above arrangement).

Since the subsequent processing from step S209 to step S212 is the same as the processing from step S108 to step S111 shown in FIG. 4 described above, the description thereof will be omitted here.
Further, in step S213, the transmission processing unit 156a adds transmission history information. The transmission processing unit 156a adds and stores the transmission history information corresponding to the transmitted stamp to the transmission history storage unit 33. After the process of step S213, the transmission processing unit 156a ends the process.

As described above, the simple communication system 1a according to the present embodiment is a transmission history storage unit that stores past transmission history information in which a transmission date and time, a transmission stamp, a transmission destination, a word, and a facial expression are associated with each other. 33 is provided. The recommended processing unit 153a determines the priority of stamp candidates based on the transmission history information stored in the transmission history storage unit 33, arranges the stamp candidates according to the priority, and displays the stamp candidates on the display unit 23. Let me.

As a result, the simple communication system 1a according to the present embodiment determines the priority of stamp candidates based on the transmission history information, arranges the stamp candidates according to the priority, and displays the stamp candidates on the display unit 23. Therefore, it is possible to recommend (recommend) to the user U1 an appropriate stamp candidate according to the usage tendency of the user U1.

Further, in the present embodiment, the transmission history information of the transmission history storage unit 33 described above includes the location information where the stamp is transmitted. The recommended processing unit 153a raises (increases) the priority of stamp candidates corresponding to the current location of the mobile terminal 10a based on the transmission history information.
Thereby, the simple communication system 1a according to the present embodiment can recommend (recommend) an appropriate stamp candidate to the user U1 according to the place of use.

Further, in the present embodiment, the transmission history information of the transmission history storage unit 33 described above includes schedule information indicating a situation when the stamp is transmitted. The recommended processing unit 153a raises (increases) the priority of stamp candidates corresponding to the situation of the user U1 based on the current schedule based on the transmission history information of the user U1.
As a result, the simple communication system 1a according to the present embodiment recommends (recommends) an appropriate stamp candidate to the user U1 according to the situation of the user U1 such as work (business) or hobby activity (private). )be able to.

Further, in the present embodiment, the transmission processing unit 156a adds and stores the transmission history information corresponding to the transmitted stamp to the transmission history storage unit 33. The recommended processing unit 153a lowers (lowers) the priority of stamp candidates for the most recently transmitted stamp.
As a result, the simple communication system 1a according to the present embodiment can suppress, for example, continuously sending the most recently transmitted stamps, and recommends an appropriate stamp candidate to the user U1 (recommended). )be able to.

Further, in the present embodiment, the recommended processing unit 153a raises (increases) the priority of stamp candidates as the stamp usage frequency increases, based on the transmission history information of the user U1.
As a result, in the simple communication system 1a according to the present embodiment, since the stamp candidates with high frequency of use are preferentially displayed on the display unit 23, the user U1 is provided with an appropriate stamp candidate according to the user's usage tendency. Can be recommended (recommended).

The stamp storage unit 32 may store words, facial expressions, stamps, and group information indicating stamp groups in association with each other. Here, the group of stamps is, for example, a set of stamps using the same character, a set of stamps used for the same purpose, and the like.

FIG. 11 is a diagram showing a modified example of the stamp storage unit 32.
As shown in FIG. 11, in the modified example of the stamp storage unit 32, the "user ID", the "group ID", the "stamp ID", the "stamp information", the "word", and the "facial expression" are used. Store the associated stamp table information. Here, the "group ID" is identification information that identifies a group of stamps, and is an example of group information.

When the stamp table information includes the "group ID" in this way, the recommended processing unit 153a gives priority to stamp candidates that match the group information of the stamp transmitted last time based on the transmission history information of the user U1. You may try to raise (raise) the ranking.
As a result, the simple communication system 1a according to the present embodiment can make the stamps to be transmitted consistent by preferentially recommending the stamp candidates with the same group information, and improve the convenience. Can be done.

The present invention is not limited to each of the above embodiments, and can be modified without departing from the spirit of the present invention.
For example, in the above embodiment, the example in which the server device 30 (30a) includes the stamp storage unit 32 and the transmission history storage unit 33 has been described, but the present invention is not limited to this, and the mobile terminal 10 (10a) may be used. These storage units may be provided.

Further, in the above embodiment, an example in which communication between the mobile terminal 10 (10a) and the eyeglass-type display device 20 is performed via the network NW1 has been described, but the present invention is not limited to this. Communication may be performed by a wired interface or a wireless interface without going through the network NW1. In this case, an interface such as Bluetooth (registered trademark) may be used for wireless communication between the mobile terminal 10 (10a) and the eyeglass-type display device 20.

Further, in the above embodiment, each functional unit included in the mobile control unit 15 (15a) may be provided outside the mobile control unit 15 (15a). For example, the word identification unit 151, the facial expression identification unit 152, and a part or all of the line-of-sight detection unit 154 may be provided in the eyeglass-type display device 20, or may be provided in the server device 30 (30a). good.

Each configuration included in the simple communication system 1 (1a) described above has a computer system inside. Then, a program for realizing the functions of each configuration included in the simple communication system 1 (1a) described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system. By executing the processing, the processing in each configuration included in the simple communication system 1 (1a) described above may be performed. Here, "loading and executing a program recorded on a recording medium into a computer system" includes installing the program in the computer system. The term "computer system" as used herein includes hardware such as an OS and peripheral devices.
Further, the "computer system" may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and a dedicated line. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. As described above, the recording medium in which the program is stored may be a non-transient recording medium such as a CD-ROM.

The recording medium also includes an internal or external recording medium that can be accessed from the distribution server to distribute the program. Even if the program is divided into a plurality of programs, downloaded at different timings, and then combined with each configuration provided in the simple communication system 1 (1a), or the distribution server that distributes each of the divided programs is different. good. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network, and holds the program for a certain period of time. It shall also include things. Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned function in combination with a program already recorded in the computer system.

Further, a part or all of the above-mentioned functions may be realized as an integrated circuit such as LSI (Large Scale Integration). Each of the above-mentioned functions may be made into a processor individually, or a part or all of them may be integrated into a processor. Further, the method of making an integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

1, 1a Simple communication system 10, 10a Mobile terminal 11, 24, 31 NW communication unit 12 Input unit 13, 23 Display unit 14, 14a Terminal storage unit 15, 15a Terminal control unit 16 Position information detection unit 20 Glass-type display device 21 Lip camera 22 Line-of-sight camera 30, 30a Server device 32 Stamp storage unit 33 Transmission history storage unit 141 Schedule storage unit 151 Word recognition unit 152 Expression identification unit 153, 163a Recommended processing unit 154 Line-of-sight detection unit 155 Reception processing unit 156, 156a Transmission processing Department 157 Stamp registration processing department NW1 network

Claims (11)

A first imaging unit that is a display device that can be worn on the user's head and that can capture at least a first image including the user's lips when worn on the user's head, and the above. A display device having a second image pickup unit capable of capturing a second image including the user's eyes and a display unit for displaying the image, and a display device.
Equipped with a mobile terminal that can connect to the network
The mobile terminal
A word identification unit that detects the movement of the lips based on the first image captured by the first imaging unit and identifies the word indicated by the movement of the lips based on the detected movement of the lips.
A facial expression identification unit that identifies the user's facial expression based on the first image,
The word identified by the word identification unit and the facial expression identification unit were identified from a candidate storage unit that stores the word, the facial expression, and a stamp which is an illustration expressing the user's emotion in association with each other. A recommended processing unit that acquires the stamp candidate based on the facial expression and displays the image including the stamp candidate on the display unit.
The stamp is selected from the stamp candidates displayed on the display unit based on the line of sight of the user detected based on the second image captured by the second imaging unit, and the selected stamp is selected. A simple communication system including a transmission processing unit that transmits an image to a predetermined destination via the network. A transmission history storage unit that stores past transmission history information in which the transmission date and time, the transmitted stamp, the transmission destination, the word, and the facial expression are associated with each other is provided.
The recommended processing unit determines the priority of the stamp candidates based on the transmission history information stored in the transmission history storage unit, arranges the stamp candidates according to the priority, and displays the stamp candidates. The simple communication system according to claim 1, wherein the communication system is displayed on a unit. The transmission history information includes location information at which the stamp was transmitted.
The simple communication system according to claim 2, wherein the recommended processing unit raises the priority of the stamp candidate corresponding to the current location of the mobile terminal based on the transmission history information. The transmission history information includes schedule information indicating the situation when the stamp is transmitted.
Claim 2 or claim, wherein the recommended processing unit raises the priority of the stamp candidate corresponding to the user's situation based on the current schedule based on the transmission history information of the user. Item 3. The simple communication system according to item 3. The transmission processing unit adds and stores the transmission history information corresponding to the transmitted stamp to the transmission history storage unit.
The simple communication system according to any one of claims 2 to 4, wherein the recommended processing unit lowers the priority of the stamp candidate with respect to the stamp transmitted most recently. A second to a fifth aspect of the present invention, wherein the recommended processing unit raises the priority of the stamp candidate as the stamp is used more frequently based on the transmission history information of the user. The simple communication system according to any one of the above. The candidate storage unit stores the word, the facial expression, the stamp, and the group information indicating the group of the stamps in association with each other.
From claim 2, the recommended processing unit raises the priority of the stamp candidate, which matches the group information of the stamp transmitted last time, based on the transmission history information of the user. The simple communication system according to any one of claims 6. The candidate storage unit stores stamp table information in which the word, the facial expression, and the stamp are associated with each user.
The simple communication system according to any one of claims 1 to 7, wherein the candidate storage unit includes a registration processing unit that stores and registers the stamp table information of the user. A reception processing unit for displaying a message received via the network on the display unit is provided.
The recommended processing unit causes the display unit to display the image including the stamp candidate for the received message.
The simple communication according to any one of claims 1 to 8, wherein the transmission processing unit transmits the selected stamp to a predetermined destination including the source of the received message. system. A first imaging unit that is a display device that can be worn on the user's head and that can capture at least a first image including the user's lips when worn on the user's head, and the above. A simple communication method of a simple communication system including a second imaging unit capable of capturing a second image including the user's eyes, a display device having a display unit for displaying an image, and a mobile terminal connectable to a network. There,
The mobile terminal detects the movement of the lips based on the first image captured by the first imaging unit, and identifies the word indicated by the movement of the lips based on the detected movement of the lips. Identification step and
An emotion identification step in which the mobile terminal identifies the facial expression of the user based on the first image,
The word identified by the word identification step and the word identified by the mobile terminal from a candidate storage unit that stores the word, the facial expression, and a stamp which is an illustration expressing the user's emotion in association with each other. A recommended processing step of acquiring a stamp candidate based on the facial expression identified by the emotion identification step and displaying the image including the stamp candidate on the display unit.
The mobile terminal selects the stamp from the stamp candidates displayed on the display unit based on the line of sight of the user detected based on the second image captured by the second imaging unit. , A simple communication method comprising a transmission processing step of transmitting the selected stamp to a predetermined destination via the network. A first imaging unit that is a display device that can be worn on the user's head and that can capture at least a first image including the user's lips when worn on the user's head, and the above. A computer of a simple communication system including a second imaging unit capable of capturing a second image including the user's eyes, a display device having a display unit for displaying an image, and a mobile terminal connectable to a network.
A word identification step of detecting the movement of the lips based on the first image captured by the first imaging unit and identifying a word indicated by the movement of the lips based on the detected movement of the lips.
An emotion identification step for identifying the user's facial expression based on the first image, and
The word identified by the word identification step is identified by the emotion identification step from a candidate storage unit that stores the word, the facial expression, and a stamp which is an illustration expressing the user's emotion in association with each other. A recommended processing step of acquiring a stamp candidate based on the facial expression and displaying the image including the stamp candidate on the display unit, and
The stamp is selected from the stamp candidates displayed on the display unit based on the line of sight of the user detected based on the second image captured by the second imaging unit, and the selected stamp is selected. A program for executing a transmission processing step of transmitting an image to a predetermined destination via the network.

Images