WO2018061871A1

WO2018061871A1 - Terminal device, information processing system, processing method, and program

Info

Publication number: WO2018061871A1
Application number: PCT/JP2017/033734
Authority: WO
Inventors: 久美子高塚; 山賀　宏之; 伊藤　真由美; 康一森川
Original assignee: 日本電気株式会社
Priority date: 2016-09-29
Filing date: 2017-09-19
Publication date: 2018-04-05

Abstract

This terminal device is provided with: a transmission destination-specifying information detection unit for searching, from an image of an object to be read, in which transmission destination-specifying information and message information are described, a predetermined region where the transmission destination-specifying information is described and for detecting the transmission destination-specifying information included in the predetermined region; a message information detection unit for detecting, from the image, the message information; and a message transmission processing unit for transmitting the message information to a transmission destination terminal device specified on the basis of the transmission destination-specifying information.

Description

Terminal device, information processing system, processing method, program

The present invention relates to a terminal device, an information processing system, a processing method, and a program.

There are portable terminals such as tablet terminals that simplify functions and GUIs and display display information in a large size targeting elderly people who are unfamiliar with the use of ICT (Information and Communications Technology) devices. In such an apparatus, the user-friendly interface is improved by using an interactive UI (User Interface) using a character. In order to perform dialogue processing using a character or the like, an ICT device such as a terminal needs to detect the user's voice with high accuracy. A related technique is disclosed in Patent Document 1.

JP 2008-108133 A

In the technique of Patent Document 1 described above, a transmission terminal device reads a transmission sheet with an ID sticker on which a sender, destination, and an identifier representing a type of fixed sentence constituting a part of communication contents are attached, by a scanner. Then, the transmitting terminal device recognizes each identifier from the digitized data, and transmits the ID code together with the digitized data to the server device.

However, the technique of Patent Document 1 is not a technique for a user unfamiliar with an ICT device to easily send a message to a communication destination using a mobile terminal.

Therefore, an object of the present invention is to provide a terminal device, an information processing system, a processing method, and a program that solve the above-described problems.

According to the first aspect of the present invention, the terminal device searches the predetermined area in which the transmission destination specifying information is described from the image of the reading target object in which the transmission destination specifying information and the message information are described. A transmission destination identification information detection unit for detecting transmission destination identification information included in the area; a message information detection unit for detecting the message information from the image; and a transmission destination terminal device identified based on the transmission destination identification information. A message transmission processing unit for transmitting message information.

According to the second aspect of the present invention, the processing method searches for a predetermined area in which the transmission destination specifying information is described from an image of a reading object in which the transmission destination specifying information and message information are described, and The destination identification information included in the area is detected, the message information is detected from the image, and the message information is transmitted to the destination terminal device identified based on the transmission destination identification information.

According to the third aspect of the present invention, the program searches a computer for a predetermined area in which the transmission destination specifying information is described from an image of a reading object in which the transmission destination specifying information and message information are described. Detecting transmission destination specifying information included in the predetermined area, detecting the message information from the image, and transmitting the message information to the transmission destination terminal device specified based on the transmission destination specifying information. .

According to the present invention, a user unfamiliar with the ICT device can easily transmit a message.

It is a 1st figure which shows the terminal device by one Embodiment of this invention, and its image display example. It is a hardware block diagram of the terminal device by one Embodiment of this invention. It is a functional block diagram of the terminal device by one Embodiment of this invention. It is a 2nd figure which shows the terminal device by one Embodiment of this invention, and its image display example. It is a figure which shows the operation outline of message transmission using the terminal device by one Embodiment of this invention. It is a figure which shows the structure of the information processing system by one Embodiment of this invention. It is a figure which shows the processing flow of the terminal device by one Embodiment of this invention. It is a 1st figure which shows the functional block of the terminal device and server apparatus of the information processing system by one Embodiment of this invention. It is a figure which shows the processing flow of the information processing system by one Embodiment of this invention. It is a 2nd figure which shows the functional block of the terminal device and server apparatus of the information processing system by one Embodiment of this invention. It is a figure which shows the robot provided with the function of the terminal device by one Embodiment of this invention. It is a figure which shows the minimum structure of the terminal device by one Embodiment of this invention.

<First Embodiment>
The terminal device according to the first embodiment will be described below with reference to the drawings.
FIG. 1 is a first diagram illustrating a terminal device and an image display example thereof.
As shown in this figure, the terminal device 1 has a display screen 16. The terminal device 1 is a tablet terminal, for example. A tablet terminal is an embodiment of an ICT device. The terminal device 1 displays the character image 100 and the auxiliary image 101 on the display screen 16 and displays operation buttons on the display screen 16 that are simplified so that even a user unaccustomed to an ICT device such as an elderly person can easily operate them. Display in area 110. In the present embodiment, an example in which only icon images of three operation buttons are displayed in the operation button display area 110 is shown. The terminal device 1 includes a camera 18.

FIG. 2 is a hardware configuration diagram of the terminal device.
The terminal device 1 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an SSD (Solid State Drive) 14, a communication module 15, a display screen 16, an IF (Interface) 17, The camera 18 and the microphone 19 are provided. The display screen 16 is configured by a liquid crystal monitor, a touch panel, or the like, and may have an input function for a user to input an operation by touching the touch panel in addition to a display function. Information input on the touch panel of the display screen 16 is processed by the IF 17 and output to the CPU 11.

FIG. 3 is a functional block diagram of the terminal device.
When the power is turned on, the CPU 11 (FIG. 2) of the terminal device 1 starts the interactive processing program recorded in the ROM 13 (FIG. 2) or the SSD 14 (FIG. 2). Thereby, the CPU 11 of the terminal device 1 includes the functions of the control unit 111, the transmission destination specifying information detection unit 112, the message information detection unit 113, and the dialogue processing unit 114. The CPU 11 of the terminal device 1 has a function of a communication application processing unit 115 (message transmission processing unit) by starting a communication application program.

The control unit 111 controls other functional units.
The transmission destination specifying information detecting unit 112 searches a predetermined area in which the transmission destination specifying information is described from the image of the reading target object in which the transmission destination specifying information and the message information are described, and the transmission destination specifying information included in the predetermined area Is detected. The transmission destination specifying information detecting unit 112 searches for a predetermined area where a destination card pasted on the object to be read and in which the transmission destination specifying information is described is stretched.
The message information detection unit 113 detects message information from the image.
The dialogue processing unit 114 performs dialogue processing such as outputting information that prompts the user to adjust the shooting position of the whiteboard W that is the reading target, in accordance with the shooting state of the transmission destination specifying information included in the predetermined area of the image.
The communication application processing unit 115 performs predetermined application processing such as a mail function, a message processing function, and an SNS (Social Networking Service) function. The communication application processing unit 115 transmits message information to the transmission destination (transmission destination terminal device) specified based on the transmission destination specification information. The communication application processing unit 115 has the information amount (first information amount) read based on the transmission destination specifying information included in the predetermined area of the image as all information indicated by the transmission destination specifying information stored in the storage unit such as the SSD 14. When the amount is equal to or greater than a predetermined ratio of the amount (second information amount), the message information is transmitted based on the transmission destination identifier corresponding to the transmission destination specifying information. The communication application processing unit 115 performs processing for transmitting predetermined message information such as a mail function, a message processing function, and an SNS (Social Networking Service) function.
Details of the processing of each of the other processing units will be described below.

FIG. 4 is a second diagram illustrating the terminal device and an image display example thereof.
As shown in FIG. 4, after the power is turned on, the terminal device 1 displays the character image 100, and displays a plurality of operation buttons in a predetermined operation button display area 110 in the screen area. In principle, the terminal device 1 does not change the position of the operation button display area 110. As a result, a user unfamiliar with the ICT device can be operated without thinking about many operations. The terminal device 1 may give an action to the character image 100 to display a gesture such as the character image 100 walking on the screen or a gesture for performing a conversation. Further, the terminal device 1 may display an auxiliary image 101 representing the emotion of the character image 100 as shown in FIG. In FIG. 1, a heart mark is displayed as the auxiliary image 101. The character image 100 shown in FIG. 4 shows a movement that walks to the left and right, and a display in which the character walks between the character image 100a and the character image 100b is performed.

FIG. 5 is a diagram showing an outline of message transmission operation using the terminal device of this embodiment.
The user A attaches the destination card C to the whiteboard W of a size (size) that can be carried (FIG. 5, step a). The destination card C is a card received in advance from the user B. The user A takes out the destination card of the user B from the destination cards C of many friends and pastes the destination card C on the whiteboard W. It is assumed that information (image information and code information) of the destination card C is recorded in the storage unit of the terminal device 1 in advance. For example, the destination card C is provided with a magnet on the back surface, and a magnet is provided on the entire surface of the whiteboard W, so that the user can attach the destination card C to the whiteboard W. The destination card C displays the face image, name, telephone number, and code information of the user B as the transmission destination. The code information includes transmission destination specifying information. The code information is specifically a two-dimensional code or the like. The color of the character or figure displayed on the destination card C, the position of the color, the position of the character, the number, the position of the number, etc. may be used as the code information.

User A enters a message with a marker on whiteboard W (FIG. 5, step b).
The user A operates the operation button to instruct the terminal device 1 to send a message (FIG. 5, step c). The user may instruct message transmission by sounding “message transmission”. In this case, the terminal device 1 has a voice analysis function.
The user A puts the whiteboard W attached with the destination card C in front of the camera 18 of the terminal device 1 and adjusts the position of the whiyboard so that the camera 18 can photograph the whiteboard W (FIG. 5, step). d). Accordingly, the terminal device 1 analyzes the image and transmits the message information detected from the whiteboard W to the destination specified based on the code information.
The other user B receives the message and displays it on his terminal device 1 (step e in FIG. 5).

FIG. 6 is a diagram showing the configuration of the information processing system.
In the information processing system 200 of the present embodiment, a plurality of terminal devices 1 are communicatively connected via the server device 20.
The user carries the terminal device 1 and transmits / receives message information to / from other users using the terminal device 1.

FIG. 7 is a diagram showing a processing flow of the terminal device.
Next, the processing flow of the terminal device 1 in the message transmission processing described with reference to FIG. 5 will be described step by step.
The dialogue processing unit 114 of the terminal device 1 displays the character image 100, the auxiliary image 101, and operation buttons after activation (step S701). The dialogue processing unit 114 controls the display type and movement of the character image 100 and the auxiliary image 101. For example, the dialogue processing unit 114 displays an image that attracts the user's interest, such as moving the character indicated by the character image 100 on the screen or shaking the character's head. The dialogue processing unit 114 may change or move the color of the auxiliary image 101. The character image 100 is an aspect of a dialogue target image. Other examples of the dialogue target image may be an anthropomorphic image or an actual human image.

The dialogue processing unit 114 detects pressing of an instruction button for transmitting a message among the operation buttons (step S702). The dialogue processing unit 114 instructs the communication application processing unit 115 to start message transmission processing. Then, the communication application processing unit 115 activates the camera 18 (step S703). The camera 18 starts to acquire a captured image.

Note that the user may input a message transmission instruction by voice. In this case, the dialogue processing unit 114 acquires the user's voice through the microphone 19. The dialogue processing unit 114 has a voice analysis function, and converts the user's voice into character information or the like. The dialogue processing unit 114 may operate various functions based on the character information. For example, when the dialogue processing unit 114 converts the voice of the user into character information of an instruction to operate the communication application processing unit 115 based on the voice analysis, the dialogue processing unit 114 instructs the communication application processing unit 115 to operate based on the character information. For example, when the character information generated based on the voice analysis is “message transmission”, the dialogue processing unit 114 instructs the communication application processing unit 115 to start the message transmission process. Then, the communication application processing unit 115 activates the camera 18. As a result, the camera 18 starts to acquire a captured image.

After instructing the message transmission, the user enters a message with a marker on the above-described whiteboard W of a portable size (size) and attaches the destination card C. The user places the whiteboard W in front of the camera 18 of the terminal device 1 and adjusts the position of the whiteboard so that the camera 18 can photograph the whiteboard W. The camera 18 repeatedly outputs the captured image to the communication application processing unit 115. The communication application processing unit 115 sequentially acquires captured images (step S704). The communication application processing unit 115 outputs the captured image to the transmission destination specifying information detection unit 112 and the message information detection unit 113, and instructs an analysis request.

When receiving the analysis request, the transmission destination specifying information detecting unit 112 searches for an attached area (predetermined area) of the destination card C included in the captured image (step S705). The transmission destination specifying information detecting unit 112 may store an image pattern of the destination card C in advance and detect an attached area that matches the image pattern from the captured image. The transmission destination specifying information detecting unit 112 detects code information included in the attached area of the destination card C (step S706). The code information is information for specifying a transmission destination. The transmission destination specifying information detection unit 112 outputs the code information to the communication application processing unit 115. The transmission destination specifying information detection unit 112 may output the attached area of the destination card C to the message information detection unit 113. When the character or figure color displayed on the destination card C, the position of the color, the position of the character, the numeral, the position of the numeral, or the like is used as the code information, the transmission destination specifying information detecting unit 112 detects the character. Alternatively, one or more of the color of the figure, the position of the color, the position of the character, the number, and the position of the number may be read as code information. Which information is displayed on the destination card C is set in advance, and the transmission destination specifying information detecting unit 112 reads the code information. The transmission destination specifying information detection unit 112 has conditions that allow the code information to be recognized as matching even if the information amount of the code information displayed on the destination card C cannot be completely detected due to blurring of the captured image. For example, the code information may be output to the communication application processing unit 115.

The communication application processing unit 115 uses the code information read by the transmission destination specifying information detection unit 112 in the attachment area as one code information among a plurality of code information recorded in the SSD 14 or the ROM 13 (card information storage unit). Is determined. If the code information matches one of the code information recorded on the SSD 14 or the like, the communication application processing unit 115 reads the transmission destination identifier associated with the code information and recorded on the SSD 14 or the like ( Step S707). The communication application processing unit 115 determines the degree of coincidence of the code information even if the code information does not coincide with one of the code information recorded in the SSD 14 or the like. When the degree of coincidence of the code information is equal to or greater than a predetermined ratio, the communication application processing unit 115 determines the degree of coincidence (corresponding to the code information recorded in the SSD 14 or the like) and the transmission recorded in the SSD 14 or the like. Read the destination identifier. The transmission destination identifier may be a mail address in the present embodiment. The communication application processing unit 115 identifies the mail address as the destination mail address. If the degree of coincidence of the code information is equal to or higher than a predetermined ratio, the user can send mail even if the code information cannot be read with high accuracy by specifying the mail address as the transmission destination. Thus, even when a user unfamiliar with the operation of ICT causes the camera 18 to photograph the whiteboard W, mail can be easily transmitted even if the focus is not completely achieved.

When the message information detection unit 113 acquires the analysis request, the message information detection unit 113 detects the message information included in the captured image (step S708). Specifically, the message information detection unit 113 stores, for example, an image pattern when the whiteboard W is photographed, and detects a whiteboard area that matches the image pattern from the photographed image. The message information detection unit 113 acquires the attachment area of the destination card C from the transmission destination specifying information detection unit 112, and excludes the attachment area of the destination card C from the whiteboard area. The message information detection unit 113 detects characters described in the area excluding the attachment area of the destination card C from the whiteboard area. A known character detection technique in an image may be used for character detection. Thus, the message information detection unit 113 can detect the character information of the user's message entered on the whiteboard W with a marker or the like without erroneously recognizing the character information described on the destination card. The message information detection unit 113 outputs the detected message information to the communication application processing unit 115.

The communication application processing unit 115 generates the message detected in step S708 using the transmission destination identifier read in step S707 (step S709). That is, the communication application processing unit 115 generates a mail or a message that includes a mail address that is a transmission destination identifier as a destination and includes message information in the text. And the terminal device 1 transmits the produced | generated mail and message (step S710). As a result, the terminal device 1 used by the user A can send an email or message to the terminal device 1 used by the user B via the server device 20.

Through the above processing, the user can send a mail or a message to a desired destination without using the terminal device 1 to input a mail address or inputting a character string of a message body into the terminal device 1. As a result, a user unfamiliar with the ICT device can easily send e-mails and messages.

<Second Embodiment>
The terminal device 1 of the transmission source determines whether or not the destination card information including the transmission destination specifying information is recorded in the other terminal device 1 of the transmission destination, and the destination card information including the transmission destination specifying information is the destination Message information may be transmitted when recorded in another terminal device 1. This process is called a destination card registration status confirmation process.

In this case, as a first example of the confirmation process of the destination card registration status, the terminal device 1 of the transmission source used by the user A sends the destination of the user A to the terminal device 1 of the transmission destination used by the user B. Queries whether the card information is retained. The transmission source terminal device 1 transmits message information only when the transmission destination terminal device 1 holds information on the destination card of the user A.
Specifically, the communication application processing unit 115 of the terminal device 1 transmits the destination card information of the user A to the terminal device 1 used by the user B after step S709. The terminal device 1 of the user B determines whether or not to store the destination card information of the user A, and transmits information indicating whether or not to store the information to the terminal device 1 of the transmission source (user A). The terminal device 1 of the user A discards the generated message information when the destination terminal device 1 used by the user B does not store the destination card information of the user A. The terminal device 1 of the user A transmits the generated message information when the terminal device 1 of the transmission destination used by the user B stores the destination card information of the user A.

As a second example of destination card registration status confirmation processing, the terminal device 1 of the transmission source used by the user A transmits message information to the terminal device 1 of the user B. In this case, it is confirmed whether the terminal device 1 of the transmission destination used by the user B holds the information of the destination card of the user A using the terminal device 1 of the transmission source in its own device or the like. The terminal device 1 of the transmission destination receives the message information if the information of the destination card of the user A is held by the own device or the like, and discards the message information if not held.
Specifically, after the communication application processing unit 115 of the transmission source terminal device 1 performs the process of step S710, the transmission destination terminal device 1 used by the user B temporarily receives message information. Assume that the message information includes destination card information. The destination terminal device 1 reads the destination card information from the message information. The destination terminal device 1 determines whether or not to store the destination card information of the user A. The destination terminal device 1 receives the message information when storing the destination card information of the user A. If the terminal device 1 of the transmission destination does not store the destination card information of the user A, the message information is discarded.

According to the processing of the terminal device 1 of the second embodiment, it is possible to perform control so as not to transmit / receive a message to / from a terminal device used by another user who has not exchanged the destination card. As a result, it is possible to prevent erroneous transmission of message information of a user unfamiliar with the ICT device or transmission of unauthorized message information using the terminal device 1.

In the first embodiment and the second embodiment described above, the dialogue processing unit 114 provides information that prompts the user to adjust the shooting position of the whiteboard W according to the shooting state of the code information included in the attached area of the shot image. You may make it output.
Specifically, the communication application processing unit 115 determines the degree of coincidence between the code information acquired from the captured image and the code information stored in the storage unit such as the SSD 14 in advance. Alternatively, the message information detection unit 113 determines whether or not the character information has been determined and outputs it to the communication application processing unit 115. The communication application processing unit 115 determines that it is necessary to adjust the shooting position of the whiteboard W based on information indicating that the degree of coincidence of the code information is low or based on information indicating that character information obtained from another processing unit cannot be determined. To do. The communication application processing unit 115 outputs the determination result to the dialogue processing unit 114. Then, the dialogue processing unit 114 outputs information instructing the user to prompt adjustment of the shooting position of the whiteboard W. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a voice such as “Cannot confirm whiteboard message” from the speaker.

In the first and second embodiments described above, the dialogue processing unit 114 may output information that prompts the user to adjust the shooting position of the whiteboard W when the attached area cannot be detected in the shot image. Good.
Specifically, the transmission destination specifying information detecting unit 112 of the terminal device 1 determines whether or not the attached area has been searched in the process of step S705. If the attached area cannot be searched for in the captured image, the transmission destination specifying information detecting unit 112 notifies the communication application processing unit 115 that the attached area cannot be searched. Then, the dialogue processing unit 114 outputs information instructing the user to prompt adjustment of the shooting position of the whiteboard W. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a sound such as “cannot confirm destination card” from the speaker.

In the first and second embodiments described above, the dialogue processing unit 114 may output information that prompts the user to change the brightness of the room based on the luminance of the captured image.
Specifically, the communication application processing unit 115 of the terminal device 1 inputs a captured image from the camera 18. The communication application processing unit 115 determines the brightness of the captured image. When the luminance of the captured image is equal to or less than the threshold value, the communication application processing unit 115 outputs to the dialogue processing unit 114 so as to output information that prompts a change in room brightness. The dialogue processing unit 114 outputs information prompting to change the room brightness to a stronger (brighter). This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a sound such as “Please brighten the room” from the speaker.

In the first and second embodiments described above, when the code information included in the attached area of the destination card C cannot be detected in the captured image, the dialogue processing unit 114 provides information that prompts the user to change the room brightness. You may make it output.
Specifically, the transmission destination specifying information detection unit 112 of the terminal device 1 determines whether code information has been detected. If the code information cannot be detected, the transmission destination specifying information detection unit 112 outputs information indicating that the code information cannot be detected to the communication application processing unit 115. When the communication application processing unit 115 receives a notification that the code information cannot be detected, the communication application processing unit 115 outputs to the dialogue processing unit 114 so as to output information that prompts the user to change the brightness of the room. The dialogue processing unit 114 outputs information that prompts the user to strongly change the room brightness. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a sound such as “Please brighten the room” from the speaker.

In the first embodiment and the second embodiment described above, the dialogue processing unit 114 may output information for prompting a change in room brightness based on the brightness information acquired from the sensor device.
Specifically, the communication application processing unit 115 of the terminal device 1 acquires indoor brightness information from the sensor device. The sensor device may be a camera 18. Alternatively, the sensor device may be an illuminance detector provided in the terminal device 1. When the value of the brightness information acquired from the sensor device is equal to or less than the threshold value, the communication application processing unit 115 outputs to the dialogue processing unit 114 so as to output information that prompts a change in room brightness. The dialogue processing unit 114 outputs information that prompts the user to strongly change the room brightness. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a sound such as “Please brighten the room” from the speaker. Note that the terminal device 1 may increase the degree of brightness based on the brightness information in the brightness change described above.

In the first and second embodiments described above, the communication application processing unit 115 may determine whether the photographed image includes the face image of the user together with the whiteboard W that is the reading target. In this case, for example, the communication application processing unit 115 acquires a captured image. It is determined whether the face image of the user is included in the captured image together with the communication application processing unit 115 whiteboard W. If the communication application processing unit 115 determines that the user's face image is not included in the captured image together with the whiteboard W, the communication application processing unit 115 outputs information indicating that the face image is not included in the captured image to the dialogue processing unit 114. To do. When the dialogue processing unit 114 acquires information indicating that the face image is not included in the photographed image, the dialog processing unit 114 outputs information that prompts photographing including the face image. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a sound such as “Please photograph the face together” from the speaker.

In the first and second embodiments described above, if the communication application processing unit 115 determines that the photographed image does not include the face image of the user together with the whiteboard W, the communication application processing unit 115 stops transmitting message information. You may judge.
In this case, for example, the communication application processing unit 115 acquires a captured image. The communication application processing unit 115 determines whether the face image of the user is included in the captured image together with the whiteboard W. If the communication application processing unit 115 determines that the face image of the user is not included in the captured image together with the whiteboard W, the communication application processing unit 115 determines to stop transmission of the message information. The communication application processing unit 115 outputs information indicating suspension of message transmission to the dialogue processing unit 114. When the dialog processing unit 114 acquires information indicating that message transmission is stopped, the dialog processing unit 114 outputs the information. This instruction information may be output from a speaker by voice, or character information may be output to the display screen 16. For example, the dialogue processing unit 114 outputs a voice such as “Message transmission is stopped because the face cannot be photographed” from the speaker.

In the first and second embodiments described above, when the captured image includes the user's face image together with the whiteboard W, the communication application processing unit 115 sets both the face image and message information to the transmission destination. You may make it transmit.
Specifically, the communication application processing unit 115 generates the message information detected in step S708 using the transmission destination identifier read in step S707. That is, the communication application processing unit 115 generates a mail or message that includes a mail address that is a transmission destination identifier as a destination, message information including a text, and a detected face image of the user A. And the terminal device 1 transmits the produced | generated mail and message.

In the first embodiment and the second embodiment described above, the communication application processing unit 115 may perform authentication processing of the face image included in the photographed image and transmit message information when the authentication is successful. .
More specifically, information on the face image of user A is recorded in advance in a storage unit such as the SSD 14. The communication application processing unit 115 of the terminal device 1 performs an authentication process as to whether or not the face image included in the photographed image matches the face image recorded on the SSD 14. When the face image included in the captured image matches the face image recorded in the SSD 14, the communication application processing unit 115 transmits the generated mail or message. On the other hand, when the face image included in the captured image does not match the face image recorded in the SSD 14, the communication application processing unit 115 stops the transmission of the generated mail or message.

In the first and second embodiments described above, when the communication application processing unit 115 transmits message information to the transmission destination, the communication application processing unit 115 automatically transmits the message information to a predetermined specific device other than the transmission destination. You may make it do.
Specifically, the communication application processing unit 115 transmits message information to the transmission destination by the process of step S710. The terminal device 1 of the transmission source stores in advance a transmission destination identifier of a predetermined device other than the transmission destination that automatically transmits message information to a storage unit such as the SSD 14. The communication application processing unit 115 reads the transmission destination identifier of a predetermined device other than the transmission destination from the SSD 14. The communication application processing unit 115 uses the transmission destination identifier of a predetermined device other than the transmission destination, and transmits the same message information to another transmission destination indicated by the identifier.
Thereby, for example, when the user is an elderly person, it is possible to notify that the elderly person has transmitted the message information by setting the terminal device 1 of the family as a predetermined device other than the transmission destination.

<Third Embodiment>
FIG. 8 is a functional block diagram of a terminal device and a server device of the information processing system according to the third embodiment.
As shown in this figure, in the information processing system 200 according to the third embodiment, the terminal device 1 includes a control unit 111, a dialogue processing unit 114, and a communication application processing unit 115. The server device 20 includes a control unit 201, a transmission destination specifying information detection unit 202, a message information detection unit 203, and a message transmission processing unit 204.

In the first embodiment, an example in which the terminal device 1 that transmits the mail or message performs all the processes in steps S701 to S710 has been described. However, any one of the processes is performed in the server apparatus 20. Also good. For example, the processing of steps S701 to S704 in the first embodiment may be performed by the terminal device 1, and the processing of steps S705 to S710 may be performed by the server device 20.

In each of the above-described embodiments, the user attaches the destination card C to the whiteboard W and photographs the whiteboard W using the camera 18 of the terminal device 1. However, the information described in the destination card C may be printed on a predetermined area of a paper medium such as a flyer or an advertisement, and the paper medium may be photographed using the camera 18 of the terminal device 1. Also in this case, the terminal device 1 similarly searches for a predetermined area, detects code information and character information, and transmits message information generated using them to the transmission destination specified by the code information.

FIG. 9 is a diagram showing a processing flow of the information processing system according to the third embodiment.
As a specific example of the processing of the third embodiment, the terminal device 1 used by the user A performs the processing of corresponding steps S801 to S804 in the same manner as steps S701 to S704 in the first embodiment. The communication application processing unit 115 of the terminal device 1 transmits the captured image to the server device 20 (step S805).

When receiving the analysis request, the transmission destination specifying information detecting unit 202 of the server device 20 searches for an attached area (predetermined area) of the destination card C included in the photographed image (step S806). The transmission destination specifying information detection unit 202 may store an image pattern of the destination card C in advance and detect an attached area that matches the image pattern from the captured image. The transmission destination specifying information detection unit 202 detects code information included in the attached area of the destination card C (step S807). The code information is information for specifying a transmission destination. The transmission destination specifying information detection unit 202 outputs the code information to the message transmission processing unit 204. The transmission destination specifying information detection unit 202 may output the attached area of the destination card C to the message information detection unit 203. When the character or figure color displayed on the destination card C, the position of the color, the position of the character, the numeral, the position of the numeral, or the like is used as the code information, the transmission destination specifying information detection unit 202 uses the characters. Alternatively, one or more of the color of the figure, the position of the color, the position of the character, the number, and the position of the number may be read as code information. Which information is displayed on the destination card C is set in advance, and the transmission destination specifying information detecting unit 202 reads the code information. The transmission destination specifying information detection unit 202 outputs the code information to the message transmission processing unit 204 even if the information amount of the code information displayed on the destination card C cannot be completely detected due to blurring of the captured image. It's okay.

The message transmission processing unit 204 associates the code information read by the transmission destination specifying information detection unit 202 in the attached area with the IDs of a plurality of users A recorded in a database (card information storage unit) or the like provided in the own device. It is determined whether the code information matches one of the code information. When the code information matches one of the code information related to the user A recorded in the database or the like, the message transmission processing unit 204 stores the transmission destination associated with the code information and recorded in the database or the like. The identifier is read (step S808). The message transmission processing unit 204 determines the degree of coincidence of the code information even if the code information does not coincide with one of the code information recorded in the database or the like. When the degree of coincidence of the code information is equal to or greater than a predetermined ratio, the message transmission processing unit 204 is a transmission destination identifier recorded in the database or the like in association with the code information recorded in the database or the like for which the coincidence is determined Read. The transmission destination identifier may be a mail address in this embodiment. The message transmission processing unit 204 identifies the mail address as the destination mail address.

When acquiring the analysis request, the message information detection unit 203 detects message information included in the captured image (step S809). Specifically, the message information detection unit 203 stores, for example, an image pattern when the whiteboard W is photographed, and detects a whiteboard area that matches the image pattern from the photographed image. The message information detection unit 203 acquires the attachment area of the destination card C from the transmission destination identification information detection unit 202, and excludes the attachment area of the destination card C from the whiteboard area. The message information detection unit 203 detects characters described in the area excluding the attachment area of the destination card C from the whiteboard area.
Character detection may use a known character detection technique in an image. As a result, the message information detection unit 203 can detect the character information of the user's message entered on the whiteboard W with a marker or the like without erroneously recognizing the character information described on the destination card. The message information detection unit 203 outputs the detected message information to the message transmission processing unit 204.

The message transmission processing unit 204 generates the message detected in step S809 using the transmission destination identifier read in step S808 (step S810). That is, the message transmission processing unit 204 generates a mail or a message that includes a mail address that is a transmission destination identifier as a destination and includes message information in the text. Then, the server device 20 transmits the generated mail or message to the terminal device 1 of the user B (step S811). Thereby, the server apparatus 20 can transmit a mail and a message to the terminal device 1 which the user B uses using the picked-up image obtained from the terminal device 1 which the user A uses.

<Fourth Embodiment>
FIG. 10 is a functional block diagram of a terminal device and a server device of the information processing system according to the fourth embodiment.
As shown in this figure, in the information processing system 200 according to the fourth embodiment, the terminal device 1 includes the same functional units as those in the first embodiment. The server device 20 includes a control unit 201 and an information processing unit 205 (transaction processing unit).

In the information processing system 200, the terminal device 1 may transmit message information to the server device 20 as a destination. Further, the server device 20 may perform a commercial transaction process or other information processing based on message information from the terminal device 1.
In this case, the terminal device 1 transmits to the server device 20 message information in which the destination is set to the server device 20 by the processing of the first embodiment and the like. At this time, the user attaches the destination card of the server device 20 to the whiteboard W. Alternatively, an area (predetermined area in which transmission destination specifying information is described) that describes the same information as that described in the destination card is provided in a flyer or an advertisement. Are read by the camera 18 of the terminal device 1.

The information processing unit 205 of the server device 20 extracts a face image included in the message information transmitted from the terminal device 1. The information processing unit 205 of the server device 20 determines whether or not the face image matches the user's face image registered in the database. The information processing unit 205 performs predetermined processing using the message information when it is determined that the face images match. The predetermined process performed by the information processing unit 205 performs, for example, a commercial transaction process based on message information. Alternatively, various other processes may be performed based on the message information. The information processing unit 205 discards the message information when the face images do not match.

FIG. 11 is a diagram illustrating a robot having a function of a terminal device.
The robot 500 may have the function of the terminal device 1 described above. In this case, for example, the robot 500 may be provided with the display screen 16 shown by the terminal device 1 on the front surface. Further, the terminal device 1 provided in the robot 500 may control the robot 500 so that the robot 500 performs the motion of the character image 100 instead of displaying the character image 100. In this case, the terminal device 1 may control mechanical eye movements, mouth movements, foot movements, and the like included in the robot 500.

FIG. 12 is a diagram illustrating a minimum configuration of the terminal device.
As shown in this figure, the terminal device 1 includes at least a transmission destination specifying information detection unit 112, a message information detection unit 113, and a communication application processing unit 115. The transmission destination specifying information detecting unit 112 searches a predetermined area in which the transmission destination specifying information is described from the image of the reading target object in which the transmission destination specifying information and the message information are set, and the transmission destination specifying information included in the predetermined area Is detected. The message information detection unit 113 detects message information from the image. The communication application processing unit 115 transmits message information to the transmission destination specified based on the transmission destination specification information.

The terminal device 1 described above has a computer system inside. And the program for making the terminal device 1 perform each process mentioned above is memorize | stored in the computer-readable recording medium of the said terminal device 1, The computer of the terminal device 1 reads and runs this program The above processing is performed. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

Further, the program may be a program for realizing a part of the functions of each processing unit described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

A part or all of the above-described embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Appendix 1)
Destination specifying information for searching for a predetermined area in which the destination specifying information is described from an image of a reading object in which the destination specifying information and message information are described, and detecting the destination specifying information included in the predetermined area A detection unit;
A message information detector for detecting the message information from the image;
A message transmission processing unit that transmits the message information to the destination terminal device identified based on the destination identification information;
A terminal device comprising:

(Appendix 2)
The terminal device according to claim 1, wherein the transmission destination specifying information detecting unit searches for the predetermined area on which a destination card pasted on the reading object and describing the transmission destination specifying information is stretched.

(Appendix 3)
A card information storage unit for storing destination card information including transmission destination identification information described in the destination card and a transmission destination identifier corresponding to the transmission destination identification information;
The message transmission processing unit is configured such that the first information amount read based on the transmission destination specifying information included in the predetermined area is a second information amount indicated by the transmission destination specifying information stored in the card information storage unit. The terminal device according to claim 2, wherein when the ratio is equal to or greater than a predetermined ratio, the message information is transmitted based on the transmission destination identifier corresponding to the transmission destination specifying information.

(Appendix 4)
The message transmission processing unit determines whether or not the destination card information including the transmission destination specifying information is recorded in the transmission destination terminal device, and the destination card information including the transmission destination specifying information is the transmission destination The terminal device according to appendix 3, wherein the message information is transmitted when recorded in the terminal device.

(Appendix 5)
An interactive processing unit that outputs information for urging adjustment of the photographing position of the reading object in accordance with the photographing state of the transmission destination specifying information included in the predetermined area of the image;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 6)
An interactive processing unit that outputs information that prompts adjustment of the shooting position of the reading object when the predetermined area of the image cannot be detected;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 7)
An interactive processing unit that outputs information prompting a change in room brightness based on the brightness of the image;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 8)
An interactive processing unit that outputs information for prompting a change in indoor brightness when transmission destination specifying information included in the predetermined area of the image cannot be detected;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 9)
An interactive processing unit that outputs information for prompting a change in room brightness based on brightness information acquired from the sensor device;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 10)
A message transmission processing unit for determining whether the image includes a face image of the user together with the reading object;
A terminal device according to any one of appendix 1 to appendix 4, comprising:

(Appendix 11)
A message transmission processing unit for determining whether the image includes a user's face image together with the reading object;
The dialogue processing unit outputs information for prompting photographing including the face image when the image of the user and the face to be read are not included in the image, according to any one of appendix 5 to appendix 9. The terminal device described.

(Appendix 12)
The terminal device according to Supplementary Note 10 or Supplementary Note 11, wherein when it is determined that the user's face image is not included in the image together with the reading object, the message transmission processing unit stops transmission of the message information.

(Appendix 13)
The terminal device according to any one of Supplementary Note 10 to Supplementary Note 12, wherein the message transmission processing unit transmits both the face image and the message information.

(Appendix 14)
The terminal device according to any one of Supplementary Note 10 to Supplementary Note 13, wherein the message transmission processing unit performs authentication processing of the face image and transmits the message information when authentication is successful.

(Appendix 15)
The terminal device according to any one of appendix 1 to appendix 14, wherein the message transmission processing unit transmits the message information to a predetermined specific device other than the transmission destination terminal device.

(Appendix 16)
Destination specifying information for searching for a predetermined area in which the destination specifying information is described from an image of a reading object in which the destination specifying information and message information are described, and detecting the destination specifying information included in the predetermined area A detection unit;
A message information detector for detecting the message information from the image;
A message transmission processing unit that transmits the message information to the destination terminal device identified based on the destination identification information;
An information processing system comprising:

(Appendix 17)
A terminal device and a server device;
The server device acquires a face image included in the image, performs authentication processing based on the face image, and receives the message information transmitted from the terminal device when authentication is successful. Information processing system.

(Appendix 18)
The server device performs a transaction process after the authentication process is successful; and
18. The information processing system according to appendix 17.

(Appendix 19)
Search the predetermined area in which the transmission destination identification information is described from the image of the reading object in which the transmission destination identification information and the message information are described, and detect the transmission destination identification information included in the predetermined area;
Detecting the message information from the image;
A processing method for transmitting the message information to a transmission destination terminal device identified based on the transmission destination identification information.

(Appendix 20)
On the computer,
Search the predetermined area in which the transmission destination identification information is described from the image of the reading object in which the transmission destination identification information and the message information are described, and detect the transmission destination identification information included in the predetermined area;
Detecting the message information from the image;
Transmitting the message information to the destination terminal device identified based on the destination identification information;
A program that executes processing.

This application claims priority based on Japanese Patent Application No. 2016-191209 filed in Japan on September 29, 2016, the entire disclosure of which is incorporated herein.

DESCRIPTION OF SYMBOLS 1 ... Terminal device, 100 ... Character image, 101 ... Auxiliary image, 11 ... CPU, 12 ... RAM, 13 ... ROM, 14 ... SSD, 15 ... Communication Module, 16 ... display screen, 17 ... IF, 18 ... camera, 19 ... microphone, 111 ... control unit, 112 ... destination specific information detection unit, 113 ... message Information detecting unit 114 ... Dialog processing unit 115 ... Communication application processing unit 20 ... Server device 201 ... Control unit 202 ... Destination specifying information detecting unit 203 ... Message information detection unit, 204... Message transmission processing unit, 205.

Claims

Destination specifying information for searching for a predetermined area in which the destination specifying information is described from an image of a reading object in which the destination specifying information and message information are described, and detecting the destination specifying information included in the predetermined area A detection unit;
A message information detector for detecting the message information from the image;
A message transmission processing unit that transmits the message information to the destination terminal device identified based on the destination identification information;
A terminal device comprising:
The terminal device according to claim 1, wherein the transmission destination specifying information detecting unit searches for the predetermined area on which a destination card attached to the reading object and describing the transmission destination specifying information is stretched.
A card information storage unit for storing destination card information including transmission destination identification information described in the destination card and a transmission destination identifier corresponding to the transmission destination identification information;
The message transmission processing unit is configured such that the first information amount read based on the transmission destination specifying information included in the predetermined area is a second information amount indicated by the transmission destination specifying information stored in the card information storage unit. The terminal device according to claim 2, wherein the message information is transmitted based on the transmission destination identifier corresponding to the transmission destination specifying information when the ratio is equal to or greater than a predetermined ratio.
The message transmission processing unit determines whether or not the destination card information including the transmission destination specifying information is recorded in the transmission destination terminal device, and the destination card information including the transmission destination specifying information is the transmission destination The terminal device according to claim 3, wherein the message information is transmitted when recorded in the terminal device.
An interactive processing unit that outputs information for urging adjustment of the photographing position of the reading object in accordance with the photographing state of the transmission destination specifying information included in the predetermined area of the image;
The terminal device according to any one of claims 1 to 4, further comprising:
An interactive processing unit that outputs information that prompts adjustment of the shooting position of the reading object when the predetermined area of the image cannot be detected;
The terminal device according to any one of claims 1 to 4, further comprising:
An interactive processing unit that outputs information prompting a change in room brightness based on the brightness of the image;
The terminal device according to any one of claims 1 to 4, further comprising:
An interactive processing unit that outputs information for prompting a change in indoor brightness when transmission destination specifying information included in the predetermined area of the image cannot be detected;
The terminal device according to any one of claims 1 to 4, further comprising:
An interactive processing unit that outputs information for prompting a change in room brightness based on brightness information acquired from the sensor device;
The terminal device according to any one of claims 1 to 4, further comprising:
A message transmission processing unit for determining whether the image includes a face image of the user together with the reading object;
The terminal device according to any one of claims 1 to 4, further comprising:
A message transmission processing unit for determining whether the image includes a user's face image together with the reading object;
The dialog processing unit outputs information for prompting photographing including the face image when the face image of the user is not included with the reading object in the image. The terminal device according to item.
The terminal according to claim 10 or 11, wherein, when it is determined that the user's face image is not included in the image together with the reading object, the message transmission processing unit stops transmitting the message information. apparatus.
The terminal device according to any one of claims 10 to 12, wherein the message transmission processing unit transmits both the face image and the message information.
The terminal device according to any one of claims 10 to 13, wherein the message transmission processing unit performs authentication processing of the face image and transmits the message information when authentication is successful.
The terminal device according to any one of claims 1 to 14, wherein the message transmission processing unit transmits the message information to a predetermined specific device other than the transmission destination terminal device.
Destination specifying information for searching for a predetermined area in which the destination specifying information is described from an image of a reading object in which the destination specifying information and message information are described, and detecting the destination specifying information included in the predetermined area A detection unit;
A message information detector for detecting the message information from the image;
A message transmission processing unit that transmits the message information to the destination terminal device identified based on the destination identification information;
An information processing system comprising:
A terminal device and a server device;
The server device acquires a face image included in the image, performs authentication processing based on the face image, and receives the message information transmitted from the terminal device when authentication is successful. The information processing system described.
The server device performs a transaction process after the authentication process is successful; and
The information processing system according to claim 17.
Search the predetermined area in which the transmission destination identification information is described from the image of the reading object in which the transmission destination identification information and the message information are described, and detect the transmission destination identification information included in the predetermined area;
Detecting the message information from the image;
A processing method for transmitting the message information to a transmission destination terminal device identified based on the transmission destination identification information.
On the computer,
Search the predetermined area in which the transmission destination identification information is described from the image of the reading object in which the transmission destination identification information and the message information are described, and detect the transmission destination identification information included in the predetermined area;
Detecting the message information from the image;
Transmitting the message information to the destination terminal device identified based on the destination identification information;
A program that executes processing.