JP2015115926A - Portable terminal device, lip-reading communication method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique suitable to transmit the contents of a speech extracted based on a lip movement to a communication partner.SOLUTION: A portable terminal device includes: an image acquisition unit for acquiring the image of a lip movement; a character extraction unit for extracting characters corresponding to the lip movement on the basis of the image acquired by the image acquisition unit; a voice generation unit for generating voice by using the characters extracted by the character extraction unit; and a communication unit for transmitting the voice generated by the voice generation unit to a first communication destination.

Description

  The present invention relates to a mobile terminal device, a lip reading communication method, and a program.

  In recent years, research on techniques for reading the content of utterances from mouth movements has progressed.

  Japanese Patent Laid-Open No. 2004-133867 discloses that “an imaging unit that acquires a face image including at least a lip region, a region extraction unit that extracts a lip region from the acquired image, and a feature amount measurement unit that measures a shape feature amount from the extracted lip region A keyword DB for registering the feature amount of the keyword utterance scene measured in the registration mode, and a feature amount of the registered keyword and a feature amount measured for the utterance scene of the sentence in the recognition mode. It comprises a judging means for performing recognition processing for recognizing the utterance content of the lips by comparing and performing word spotting lip reading for recognizing a keyword from the sentence, and a display means for displaying the recognition result made by the judging means. A word spotting lip reading device is disclosed.

JP 2012-59017 A

  By the way, when carrying out a telephone conversation, it may be difficult to speak depending on the situation of the speaker. For example, this may be the case when it is difficult to speak due to the physical condition of the speaker, or when the speaker is present in a place that causes trouble to the surroundings by speaking (such as in a train). Further, when a conversation is performed in a place where noise is significant, the voice of the speaker may be mixed with the noise and not transmitted to the other party.

  In particular, when a phone call is made using a mobile phone, the surrounding situation changes, so that the situation is likely to occur.

  The technique disclosed in Patent Document 1 does not consider transmitting utterance content extracted from mouth movements to a communication partner.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a technique suitable for transmitting the utterance content extracted from mouth movements to a communication partner.

  The present application includes a plurality of means for solving the above-described problems, and examples thereof are as follows.

  In order to solve the above-described problem, a mobile terminal device according to the present invention includes an image acquisition unit that acquires an image of mouth movement, and a character corresponding to the movement of the mouth based on the image acquired by the image acquisition unit. A character extraction unit that extracts a voice, a voice generation unit that generates a voice using the characters extracted by the character extraction unit, and a communication that transmits the voice generated by the voice generation unit to a first communication destination And a section.

  Moreover, in order to solve the said subject, the portable terminal device of this invention is provided with the display part which displays the said image acquired by the said image acquisition part, and the said character extracted by the said character extraction part, It is characterized by the above-mentioned. It is good.

  Moreover, in order to solve the said subject, the portable terminal device of this invention is equipped with the input part which receives the correction of the said character displayed by the said display part, The said audio | voice production | generation part is after the correction which the said input part received The voice may be generated using the characters.

  Moreover, in order to solve the said subject, the portable terminal device of this invention is provided with the memory | storage part which memorize | stores the communication destination information which linked | related the message and the 2nd communication destination, and the said character extraction part extracted the said communication part. When the message is included in the character, the call may be made to the second communication destination associated with the message in the communication destination information.

  In order to solve the above-described problem, the mobile terminal device of the present invention includes an SOS transmission instruction reception unit that receives an instruction of SOS transmission that is transmission based on the communication destination information, and the communication unit includes the first When the SOS transmission instruction reception unit receives the SOS transmission instruction during communication with the communication destination, and the character extraction unit extracts the character including the message, the connection with the first communication destination is established. It is good also as a characteristic to cut | disconnect and to transmit to the said 2nd communication destination.

  Further, in order to solve the above-described problem, the communication unit further transmits the second communication destination when the image acquisition unit acquires the image of the mouth having a predetermined shape for a predetermined time. It is good.

  In order to solve the above problem, the display unit further displays a home screen for accepting selection of a continuous call mode or a single call mode, and the communication unit is configured to select the first call mode when the continuous call mode is selected. The sound generated based on the image acquired by the image acquisition unit after communication with the communication destination is established is transmitted to the first communication destination, and the image acquisition unit is configured to select the image when the single transmission mode is selected. The voice may be transmitted to the first communication destination with which communication has been established after acquiring.

  Moreover, in order to solve the said subject, the portable terminal device of this invention is equipped with the audio | voice acquisition part which acquires the periphery audio | voice of the said portable terminal device periphery, The said communication part adds the said audio | voice produced | generated by the said audio | voice production | generation part. In addition, the peripheral sound acquired by the sound acquisition unit may be transmitted to the first communication destination.

  In order to solve the above-mentioned problem, the communication unit may transmit the peripheral sound when the peripheral sound acquired by the sound acquisition unit is equal to or higher than a predetermined volume.

  In order to solve the above-described problem, the lip reading communication method of the present invention supports an image acquisition procedure for acquiring an image of mouth movement and the mouth movement based on the image acquired in the image acquisition procedure. Character extraction procedure for extracting characters to be generated, a voice generation procedure for generating voice using the characters extracted in the character extraction procedure, and the voice generated in the voice generation procedure are transmitted to a first communication destination And a communication procedure.

  In order to solve the above problems, a program according to the present invention is a program that causes a computer to function as a mobile terminal device, and includes an image acquisition procedure for acquiring an image of mouth movements and the image acquisition procedure acquired in the image acquisition procedure. A character extraction procedure for extracting a character corresponding to the movement of the mouth based on an image, a voice generation procedure for generating a voice using the character extracted in the character extraction procedure, and a voice generation procedure And a communication procedure for transmitting the voice to the first communication destination.

  ADVANTAGE OF THE INVENTION According to this invention, the technique suitable for telling the communicating party the utterance content extracted from the movement of the mouth can be provided.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a functional block diagram of a portable terminal device. It is a figure which shows an example of SOS setting information. It is a figure which shows the hardware structural example of the computer which implement | achieves the function of a portable terminal device. It is a flowchart which shows the flow of a continuous call process. It is a flowchart (the 1) which shows the flow of a single call process. It is a flowchart (the 2) which shows the flow of a single call process. It is a flowchart which shows the flow of the SOS communication process in continuous call mode. It is a flowchart which shows the flow of the SOS communication process in single call mode. It is a figure which shows an example of a home screen. It is a figure which shows an example of a transmission screen. It is a figure which shows an example of the lip reading screen. It is a figure which shows an example of the lip reading screen. It is a figure which shows an example of a lip reading communication setting screen. It is a figure which shows an example of a SOS setting screen. It is a figure which shows an example of the lip reading log | history display screen.

  Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram of the mobile terminal device 10. The portable terminal device 10 is a portable information processing device, and is, for example, a PC, a PDA (Personal Digital Assistant), or a smartphone. The mobile terminal device 10 has a so-called lip reading function that reads the utterance content according to the movement of the lips.

  The mobile terminal device 10 includes a control unit 110 and a storage unit 120. The control unit 110 controls communication based on the lip reading function in the mobile terminal device 10. The storage unit 120 stores information necessary for communication based on the lip reading function.

  The control unit 110 includes an image acquisition unit 111, a character extraction unit 112, a voice generation unit 113, a display unit 114, a communication unit 115, an input unit 116, an SOS transmission instruction reception unit 117, and a voice acquisition unit 118. With.

  The image acquisition unit 111 acquires an image using a camera unit described later. The acquired image is an image of the speaker's mouth, and is an image of the speaker moving his lips for utterance. The acquired image may be an image in a state where a voice is actually produced or an image obtained by moving only the mouth without producing a voice. Hereinafter, utterance refers to a state in which the mouth is moved regardless of the presence or absence of utterance.

  The character extraction unit 112 extracts characters from the uttered image. For the process of extracting characters from the movement of the lips, a known method is used, for example, by attaching coordinates to a predetermined portion of the lips and determining the character based on the amount of movement. In addition, about the technology to extract characters from lip movement, for example, “Takeshi Saito, Ryosuke Konishi,“ Lip reading by trajectory features based on lip and mouth area shape ”, Information Science and Technology Forum General Lecture Collection 6 (3), 39-40, 2007-08-22 "is used. At this time, if necessary, a basic model of the lip movement of the speaker stored in advance in an area (not shown) of the storage unit 120 may be used.

  The voice generation unit 113 generates voice data from characters. The processing for generating speech data from characters is not described in detail here because a known technique is used. The display unit 114 displays a display screen such as a home screen 140 or a lip reading screen described later on the display device. The display screen includes an image acquired by the image acquisition unit 111 and text indicating characters extracted by the character extraction unit 112.

  The communication unit 115 controls communication and a call between the mobile terminal device 10 and another telephone. Specifically, the communication unit 115 makes a call to another telephone based on the input operation. The communication unit 115 receives communications from other telephones. Further, as will be described in detail later, the communication unit 115 makes a call to the SOS communication destination stored in the SOS setting information in a predetermined case.

  The input unit 116 controls input processing using an input device such as a touch panel. For example, the input unit 116 receives an input for correcting the character extracted by the character extraction unit 112.

  The SOS transmission instruction reception unit 117 receives an instruction to transmit an SOS due to circumstances or the like that have occurred to the speaker when performing lip-reading communication. The SOS transmission instruction may be accepted during a call with the communication destination, or may be accepted at a timing when the call is not being made.

  The voice acquisition unit 118 acquires peripheral voice including a speaker's voice input via a microphone (not shown) included in the mobile terminal device 10. When the sound acquired by the sound acquisition unit 118 is equal to or higher than a predetermined volume, the communication unit 115 transmits the peripheral sound to the communication destination.

  The storage unit 120 stores SOS setting information 121. The SOS setting information 121 is information that stores an SOS communication destination to be transmitted when a character extracted by the character extraction unit 112 includes a predetermined message.

  In the present embodiment, the image acquisition unit 111 acquires an image of the holder's mouth, the character extraction unit 112 extracts characters from the mouth image, and the speech generation unit 113 generates speech from the characters. The communication unit 115 transmits the generated voice to the communication destination. Thus, even if the speaker does not utter, the character indicated by the speaker's mouth movement can be transmitted to the communication destination. In addition, even when the speaker's voice is not transmitted to the other party due to noise or the like, the voice generated by the voice generation unit 113 is transmitted to the communication destination, and the transmission of surrounding sounds is controlled, so that the speaker's voice is controlled. It is possible to convey the spoken content to the communication destination more clearly.

  FIG. 2 is a diagram illustrating an example of the SOS setting information 121. The SOS setting information 121 is information that associates the lip reading pattern with the SOS communication destination.

  The lip reading pattern is information indicating a pattern constituted by a combination of characters. The SOS communication destination is information indicating a communication destination when a lip reading pattern is uttered by a speaker. In the SOS communication destination, for example, an arbitrary call destination such as a telephone number for an emergency call to a police agency or a fire fighting agency, a telephone number of a hospital that visits a hospital, and the like are registered in advance.

  Next, a hardware configuration example of the mobile terminal device 10 will be described.

  FIG. 3 is a diagram illustrating a hardware configuration example of a computer that realizes the function of the mobile terminal device 10. The mobile terminal device 10 includes a CPU (Central Processing Unit) 130, an auxiliary storage device 131, a network I / F (Interface) 132, a memory 133, an input I / F 134, and an output I / F 135. A camera unit 137 is connected to the input I / F 134, and a touch panel 138 is connected to the output I / F 135. Each component is connected by a bus.

  The CPU 130 executes processing according to a program recorded in the memory 133 or the auxiliary storage device 131. Each unit constituting the control unit 110 is realized by the CPU 130 executing a program.

  The auxiliary storage device 131 is, for example, an HDD (Hard Disk Drive), a CD-R (Compact Disc-Recordable), a DVD-RAM (Digital Versatile Disk-Random Access Memory), etc. Device. The network I / F 132 is an interface for connecting the mobile terminal device 10 to the network.

  The memory 133 is a storage device such as a RAM (Random Access Memory) or a flash memory, and functions as a storage area from which programs and data are temporarily read. The input I / F 134 is an interface for connecting each input device to the mobile terminal device 10. The output I / F 135 is an interface for connecting each output device such as a display device to the mobile terminal device 10.

  The camera unit 137 acquires an image captured by an image sensor using a sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) as digital data and records it in the memory 133 or the auxiliary storage device 131. It is. The camera unit 137 can shoot a moving image. The touch panel 138 is a device having both an input device and an output device, and is connected to the input I / F 134 and the output I / F 135. The touch panel 138 is a device that includes, for example, a capacitive sensor and a liquid crystal display that displays a GUI (Graphical User Interface) or the like.

  The function of the storage unit 120 is realized by the memory 133 or the auxiliary storage device 131. The function of the storage unit 120 may be realized by a storage device (not shown) on the network.

  In addition, the process of each component of the portable terminal device 10 may be executed by one hardware, or may be executed by a plurality of hardware. Moreover, the process of each component of the portable terminal device 10 may be implement | achieved by one program, and may be implement | achieved by the some program.

  FIG. 9 is a diagram illustrating an example of the home screen 140. The mobile terminal device 10 has a continuous call mode and a single call mode as calls using the lip reading function. As will be described in detail later, the continuous call mode is a mode in which a conversation with a call partner is performed, and the single call mode is a mode in which a message generated by lip reading is transmitted to the other party and a response from the communication destination is not assumed.

  Home screen 140 is a screen for accepting the start of a call in continuous call mode or single call mode. The home screen 140 includes a call mode selection area 141 and a setting button 142. The call mode selection area 141 displays whether the continuous call mode or the single call mode can be selected. The setting button 142 is a button for displaying setting items for the continuous call mode and the single call mode. The setting items will be described in detail later.

  FIG. 4 is a flowchart showing the flow of continuous call processing. When selection of the continuous call mode is accepted in the call mode selection area 141 of the home screen 140, the processing according to this flowchart is started.

  First, the display unit 114 displays the transmission screen 150 on the touch panel 138 (step S11).

  FIG. 10 is a diagram illustrating an example of the transmission screen 150. The call screen 150 includes a call mode display area 151, a number selection area 152, and a call button 153.

  In the call mode display area 151, information indicating the continuous call mode is displayed. In the number selection area 152, a number is displayed so as to be selectable so that the telephone number of the communication destination can be selected. The call button 153 accepts selection when making a call to the selected telephone number.

  Returning to FIG. Next, the input unit 116 accepts input of a telephone number for the call screen 150 and selection of the call button 153 (step S12).

  Next, the communication unit 115 transmits to the communication destination of the selected telephone number (step S13).

  Next, the communication unit 115 starts a call with the communication destination (step S14). The call is started when the communication destination responds to the call and communication is established. When the communication destination does not respond, the display unit 114 displays information indicating that the communication destination does not respond on the touch panel 138, and ends the process.

  Next, the image acquisition unit 111 starts image acquisition (step S15). The image acquisition unit 111 continues to acquire images until the call with the communication destination ends.

  Next, the display unit 114 displays the lip reading screen A160 on the touch panel 138 (step S16). Note that the processing of step S15 and step S16 is not limited to the order of this flowchart.

  FIG. 11 is a diagram showing an example of the lip reading screen A160. Lip reading screen A 160 includes a call type display area 161, an image display area 162, a text display area 163, and an end button 164.

  The call type display area 161 is an area for displaying the current call mode. In the call type display area 161, information indicating the continuous call mode is displayed. In the present embodiment, the sound generated by the lip reading is transmitted to the communication destination, and the transmission of the peripheral sound to the communication destination is limited. However, when the predetermined condition is satisfied, the generated sound and the peripheral sound are Also sent together. In the call type display area 161, information indicating the type of whether the voice generated by the lip reading is transmitted to the communication destination and the transmission of the peripheral voice is restricted or the peripheral voice is transmitted together with the generated voice. May be displayed. The predetermined condition will be described in detail later.

  In FIG. 11, “continuous call mode (lip reading only)” is displayed in the call type display area 161. This indicates that the continuous call mode is a type in which the transmission of the surrounding voice is restricted and only the voice generated by the lip reading is transmitted to the communication destination.

  In the image display area 162, the image acquired by the image acquisition unit 111 in step S15 is displayed in real time.

  In the text display area 163, characters extracted by the character extraction unit 112 are displayed. The text display area 163 is displayed so as to be selectable, and accepts the selection when starting the SOS communication process described later. The end button 164 is a button for accepting selection when the call is ended.

  Returning to FIG. Next, the communication unit 115 determines whether a connection disconnection instruction has been received (step S17). When the end button 164 of the lip reading screen A160 is selected, the communication unit 115 determines that a connection disconnection instruction has been received.

  When the communication unit 115 does not determine that a connection disconnection instruction has been received (in the case of “NO” in step S17), the character extraction unit 112 refers to the image acquired by the image acquisition unit 111, and It is determined whether or not a motion has been detected (step S18).

  When the character extraction unit 112 detects the movement of the speaker's lips ("YES" in step S17), the character extraction unit 112 extracts characters from the image and generates text data (step S19).

  Next, the display unit 114 displays the text data generated by the character extraction unit 112 on the lip reading screen A160 (step S20). The text data is displayed in the text display area 163 of the lip reading screen A160.

  Next, the character extraction unit 112 stores the generated text data (step S21). The text data is stored in a text data storage area (not shown) in the storage unit 120.

  Next, the voice generation unit 113 generates a voice based on the text data generated by the character extraction unit 112 (step S22). In addition, about the process of step S20-step S22, it is not limited to the order of this flowchart.

  Next, the communication unit 115 transmits the voice generated in step S22 to the communication destination (step S23). Note that, as described above, the communication unit 115 restricts transmission of peripheral audio to a communication destination.

  Next, the character extraction unit 112 determines whether or not the movement of the lips has stopped for a predetermined time (step S24).

  If the character extraction unit 112 determines that the movement of the lips has stopped for a predetermined time or more (in the case of “YES” in step S24), the character extraction unit 112 returns the process to step S17. If the character extraction unit 112 does not determine that the movement of the lips has stopped for a predetermined time or more (in the case of “NO” in step S24), the character extraction unit 112 returns the process to step S19 and performs the character extraction process again.

  When it is determined that the communication unit 115 has received a connection disconnection instruction (“YES” in step S17), the communication unit 115 disconnects the connection (step S25). The communication unit 115 thereafter ends the process of this flowchart.

  Note that the voice acquisition unit 118 acquires ambient voice including the voice of the speaker during a call with the communication destination. When the surrounding sound volume exceeds a predetermined amount, the sound acquisition unit 118 notifies the communication unit 115. Based on the notification from the voice acquisition unit 118, the communication unit 115 transmits the peripheral voice acquired by the voice acquisition unit 118 to the communication destination in addition to the voice generated by the voice generation unit 113. Similarly, when the surrounding sound falls below a predetermined volume, the communication unit 115 restricts the transmission of the surrounding sound and transmits only the sound generated by the sound generating unit 113 to the communication destination. Thereby, the voice can be switched between transmission and non-transmission depending on the presence or absence of the utterance, and the convenience is improved.

  In other words, whether or not to limit the transmission of the surrounding voice to the communication destination may depend on the input of the speaker. For example, the input unit 116 may switch between transmission and non-transmission of peripheral voices by receiving selection of the call type display area 161 of the lip reading screen A160. Further, the selection may be accepted so that the voice generated by the lip reading is not transmitted to the communication destination, but only the peripheral voice is transmitted to the communication destination in the same manner as a normal telephone.

  In the above-described embodiment, continuous call processing is started by making a call from the mobile terminal device 10 to the communication destination, but the communication destination makes a call to the mobile terminal device 10 and receives the call at the mobile terminal device 10. The same processing is also performed when a call is started.

  In the present embodiment, the utterance content is extracted from the movement of the mouth, and is transmitted as a voice to the communication destination. Thereby, it is possible to have a conversation even when it is difficult to speak. At this time, it is possible to confirm what character has been extracted by referring to an image of the movement of his / her mouth. As a result, the speaker can recognize how the characters are extracted depending on how the mouth is moved, and the accuracy of extraction of the utterance content by lip reading is improved.

  FIG. 5 is a flowchart (part 1) showing the flow of single call processing. When the selection of the single call mode is accepted in the call mode selection area 141 of the home screen 140, the processing according to this flowchart is started.

  First, the display part 114 displays the transmission screen 150 (step S31).

  Next, the communication unit 115 accepts input of a telephone number and selection of the call button 153 (step S32). Since the processing performed in step S31 and step S32 is the same as the processing performed in step S11 and step S12, description thereof is omitted.

  Next, the image acquisition unit 111 starts acquiring an image using the camera unit 137 (step S33). The image acquisition unit 111 continues to acquire an image until selection of a text display area described later is received.

  Next, the display unit 114 displays the lip reading screen B170.

  FIG. 12 shows an example of the lip reading screen B170. Lip reading screen B170 includes a call type display area 171, an image display area 172, a text display area 173, an input end button 174, and a cancel button 175. Since the objects displayed in the call type display area 171 and the image display area 172 are the same as those in the lip reading screen A160, description thereof is omitted.

  In the text display area 173, the text extracted from the image by the character extraction unit 112 is displayed. The text display area 173 is displayed in a selectable manner. The processing of the text display area 173 differs depending on the selection method. For example, when the text display area is selected during the generation of text data by lip reading, the generation of text data is terminated. When the text display area 173 is selected with a short duration such as a tap after the generation of the text data is completed, a text correction screen (not shown) is displayed and the text can be corrected. When the selection is performed for a duration exceeding a predetermined length, such as a long press, an SOS communication process described later is started.

  Note that the end of the generation of the text data, the screen transition to the text correction screen, or the start of the SOS communication process is not limited to the above-described embodiment. For example, a button dedicated to text correction or a button dedicated to starting SOS communication processing may be provided on the lip reading screen B170.

  The input end button 174 is a button for accepting the end of character input by lip reading. The cancel button 175 is a button for accepting selection when canceling input. When the cancel button 175 is selected, the display screen changes from the lip reading screen B170 to the home screen 140.

  Returning to FIG. Next, the character extraction unit 112 determines whether or not the movement of the lips is detected based on the image acquired by the image acquisition unit 111 (step S35). When the character extraction unit 112 does not detect the lip movement (in the case of “NO” in step S35), the character extraction unit 112 returns the process to step S35 and monitors the image until the lip movement is detected.

  When the character extraction unit 112 detects the movement of the lips (if “YES” in step S35), the character extraction unit 112 generates text data from the acquired image (step S36). The processing performed in step S36 and step S37 is the same as the processing in step S19 and step S20, and thus description thereof is omitted.

  Next, the character extraction unit 112 determines whether selection of the text display area 173 included in the displayed lip reading screen B170 has been received (step S38). In the text display area 173, characters extracted from the movement of the speaker's lips are displayed. When the extraction of characters by lip reading is to be terminated, selection of the text display area 173 is accepted.

  If the character extraction unit 112 determines that the selection of the text display area 173 has been accepted (“YES” in step S38), the character extraction unit 112 ends the generation of text data (step S39).

  If the character extraction unit 112 does not determine that the selection of the text display area 173 has been received (“NO” in step S38), the character extraction unit 112 returns the process to step S35.

  Next, the character extraction unit 112 determines whether or not further selection of the text display area 173 has been received (step S40).

  If the character extraction unit 112 determines that the selection of the text display area 173 has been accepted (“YES” in step S38), the character extraction unit 112 accepts correction of text data (step S41). Specifically, when receiving the selection of the text display area 173, the character extraction unit 112 displays a text correction screen (not shown). The text correction screen is a screen for accepting correction of displayed characters, and is an input screen such as a keyboard. The character extraction unit 112 receives an input to the text correction screen.

  If the character extraction unit 112 does not determine that the selection of the text display area 173 has been accepted (“NO” in step S38), the character extraction unit 112 advances the process to step S42.

  Next, the character extraction unit 112 accepts selection of the input end button 174 displayed on the lip reading screen B170 (step S42).

  Next, the character extraction unit 112 stores the text data in the text data storage area of the storage unit 120 (step S43).

  FIG. 6 is a flowchart (part 2) showing the flow of the single call processing.

  Next, the communication part 115 transmits with respect to a communication destination (step S44). Communication unit 115 makes a call to the telephone number received in step S32.

  Next, the communication unit 115 determines whether or not there is a response from the communication destination (step S45).

  If the communication unit 115 determines that there is a response from the communication destination (“YES” in step S45), the communication unit 115 establishes communication with the communication destination and starts a call (step S46).

  Next, the voice generation unit 113 generates voice data from the text data stored in the text data storage area in step S43 (step S47). Note that the process of step S45 may be performed before the start of a call with the communication destination. Anything can be used as long as it is performed after the text data is saved in step S43.

  Next, the communication unit 115 transmits the audio data generated in step S47 to the communication destination.

  Next, the communication unit 115 disconnects the connection (step S49). Thereafter, the communication unit 115 ends the process of this flowchart.

  If the communication unit 115 does not determine that there is a response from the communication destination (“NO” in step S45), the communication unit 115 determines whether or not the number of outgoing calls has reached a predetermined number (step S50).

  When the communication unit 115 determines that the number of outgoing calls has reached the predetermined number (in the case of “YES” in step S50), the communication unit 115 ends the process of this flowchart.

  When the communication unit 115 does not determine that the number of transmissions has reached the predetermined number (in the case of “NO” in step S50), the communication unit 115 transmits again to the communication destination (step S51). Thereafter, the communication unit 115 returns the process to step S45.

  In the present embodiment, in the single call mode, the voice generated based on the lip reading is transmitted to the communication destination, and the call is terminated. At that time, the character generated based on the lip reading can be corrected. Thereby, for example, when a speaker who cannot speak is desired to transmit a message that does not require a response, communication by lip reading according to the convenience of the speaker can be realized.

  FIG. 7 is a flowchart showing the flow of SOS communication processing during the continuous call mode. The SOS communication indicates that a predetermined contact is notified when characters extracted by lip reading correspond to a message preset in the SOS setting information 121. The processing according to this flowchart is executed in a state where SOS communication is set to ON in the lip reading communication setting screen described later and a call with the communication destination is performed in the continuous call mode. In addition, when a call with the communication destination is not performed in the continuous call mode, a process similar to the SOS communication process in the single call mode described later is performed. On the touch panel 138, a lip reading screen A160 is displayed.

  First, the SOS transmission instruction receiving unit 117 receives the selection of the text display area 163 of the lip reading screen A160 (step S61).

  Next, based on the image acquired by the image acquisition unit 111, the character extraction unit 112 determines whether or not the speaker has closed his mouth for a predetermined period of time (step S62). This determination is performed to prevent an unexpected SOS communication from being transmitted if the selection of the text display area 163 is not intended by the speaker. Note that the shape of the mouth used for the determination is not limited to the closed state, and may be any predetermined shape.

  If the character extraction unit 112 does not determine that the speaker has closed his mouth for a certain period of time (“NO” in step S62), the character extraction unit 112 ends the processing of this flowchart. Specifically, when the character extraction unit 112 does not detect that the mouth is closed for a certain period of time within a predetermined time after the text display area 163 is selected, the process of this flowchart is terminated. The selection of the text display area 163 is not intended for SOS communication, but is considered to have been selected by chance, for example, by hitting the cheek. Done.

  When the character extraction unit 112 determines that the speaker has closed his mouth for a certain period of time (“YES” in step S62), the character extraction unit 112 is based on the image acquired by the image acquisition unit 111. Then, it is determined whether or not the movement of the lips has been detected (step S63). When the character extraction unit 112 does not determine that the movement of the lips has been detected (“NO” in step S63), the character extraction unit 112 performs the process again in step S63, and the image is detected until the movement of the lips is detected. To monitor.

  If the character extraction unit 112 determines that the movement of the lips has been detected (“YES” in step S63), the character extraction unit 112 creates text data from the acquired image (step S64).

  Next, the character extraction unit 112 determines whether or not a lip reading pattern in the SOS setting information 121 has been detected (step S65). Specifically, the character extraction unit 112 refers to the SOS setting information 121 using the text data created in step S64, and determines whether any of the lip reading patterns of the SOS setting information 121 is included in the text data. To do. For example, when the created text data includes a character “Takesuke”, the character extraction unit 112 determines that one of the lip reading patterns of the SOS setting information 121 illustrated in FIG. 2 is included in the text data.

  When the character extraction unit 112 does not determine that the lip reading pattern of the SOS setting information 121 has been detected (“NO” in step S65), the character extraction unit 112 ends the process of this flowchart.

  If the character extraction unit 112 determines that the lip reading pattern of the SOS setting information 121 has been detected (“YES” in step S65), the communication unit 115 disconnects the ongoing connection. (Step S66).

  Next, the communication part 115 transmits to the SOS communication destination of the SOS setting information 121 (step S67). Specifically, the communication unit 115 transmits to the SOS communication destination associated with the lip reading pattern included in the text data created in step S64 and the SOS setting information 121. In the above example, since the text data includes the characters “Takesuke”, the communication unit 115 is “000-0000”, which is the SOS communication destination associated with the lip reading data “Takesuke” in the SOS setting information 121 of FIG. -**** ".

  Next, the communication unit 115 starts a call with the calling number (step S68). Thereafter, the communication unit 115 ends the process of this flowchart. The call is started when there is a response from the communication destination, and when there is no response from the communication destination, the communication unit 115 ends the process of this flowchart without performing the call. Transmission to the SOS communication destination is not limited to one time, and a predetermined number of transmissions to the SOS communication destination may be performed.

  According to the present embodiment, when an urgent situation occurs during a conversation with a communication destination, the connection is immediately disconnected and a call is made to the SOS communication destination. At this time, since the speaker does not need to speak, the speaker can make a call to the SOS communication destination without being noticed by the surroundings.

  FIG. 8 is a flowchart showing the flow of SOS communication processing during the single call mode. The processing according to this flowchart is executed in a state where the SOS communication is set to ON on the lip reading communication setting screen 180, which will be described later, and in the single call mode, and no call is made with the communication destination. On the touch panel 138, a lip reading screen B170 is displayed. In the single call mode and when a call is made with the communication destination, the same process as the SOS communication process in the continuous call mode shown in FIG. 7 is performed.

  First, the SOS transmission instruction receiving unit 117 receives a selection of the text display area 173 (long press) of the lip reading screen B170 (step S71). The SOS transmission instruction receiving unit 117 receives an SOS transmission instruction when the text display area 173 is selected continuously for a predetermined time or more.

  Next, based on the image acquired by the image acquisition unit 111, the character extraction unit 112 determines whether or not the mouth has been closed for a certain period of time (step S72). Since the process performed between step S72 and step S75 is the same as the process performed between step S62 and step S65, description thereof is omitted.

  Next, the communication part 115 transmits to the SOS communication destination of the SOS setting information 121 (step S76). The processing performed in step S76 and step S77 is the same as the processing performed in step S67 and step S68, and thus description thereof is omitted.

  In the present embodiment, even in the single call mode, a call is made to a registered communication destination by reading a predetermined message. For example, when an emergency situation occurs during the lip reading in the single call mode, it is possible to make a call to a communication destination registered in advance, and convenience is improved.

  FIG. 13 is a diagram illustrating an example of the lip reading communication setting screen 180. When the setting button 142 is selected on the home screen 140 shown in FIG. 9, the display screen transitions to the lip reading communication setting screen 180 in this figure.

  Lip reading communication setting screen 180 includes a retry setting button 181, an SOS setting button 182, and a lip reading history display button 183 as setting items.

  The retry setting button 181 is a button for setting an upper limit of the number of times to perform a re-transmission when there is no response from the communication destination in spite of the transmission to the communication destination in the single call mode. By selecting the retry setting button 181, a retry count input screen (not shown) is displayed, and an upper limit of the retry count can be set.

  The SOS setting button 182 is a button for accepting selection when the SOS setting information 121 is generated. When the SOS setting button 182 is selected, the display screen changes to an SOS setting screen 190 described later.

  The lip reading history display button 183 is a button for displaying a history of characters extracted by the character extraction unit 112. When the lip reading history display button 183 is selected, the display screen transitions to a lip reading history display screen described later.

  FIG. 14 is a diagram illustrating an example of the SOS setting screen 190. The SOS setting screen 190 includes an ON / OFF button 191 and an SOS setting display area 192.

  The ON / OFF button 191 is a button for accepting selection as to whether or not to perform SOS communication processing. When the ON / OFF button 191 is set to OFF, the SOS communication process shown in FIGS. 7 and 8 is not started.

  The SOS setting display area 192 is an area for displaying the SOS setting information 121. On the SOS setting screen 190, the SOS setting information 121 can be generated and edited. For example, by selecting a new registration button on the SOS setting screen 190, a combination of a new lip reading pattern and an SOS communication destination can be input. By selecting the change button, the currently displayed SOS setting information 121 can be edited. The input unit 116 generates the SOS setting information 121 using newly registered or edited information.

  FIG. 15 is a diagram illustrating an example of the lip reading history display screen 200. The lip reading history display screen 200 displays character history information stored in the text data storage area of the storage unit 120. If the text data is corrected, the corrected character is displayed.

  Lip reading history display screen 200 includes an item number, a mode display area (displayed as “M” in FIG. 15), a date, a time, the number of retries, a text, and a deletion input acceptance area.

  The item number is identification information for identifying history information. In the mode display area, information indicating whether the characters input on the lip reading or text correction screen are input in the continuous call mode or the single call mode is displayed. As the date and time, the date and time when characters are input are displayed. As the number of retries, the number of times the input character is transmitted to the communication destination is displayed. For example, in the history information of item number “1” in FIG. 15, the number of retries is stored as “3/5”. This is because the upper limit of the number of retries set on the lip reading communication setting screen 180 is “5”. On the other hand, it is transmitted to the communication destination after being transmitted “3” times, or the transmission is canceled.

  The input character is displayed in the text. The deletion input reception area is an area for receiving an instruction to delete the displayed history information. For example, a check box for receiving selection of text to be deleted is displayed. When the text to be deleted is selected in the deletion input receiving area and the execution button is selected, the history information with a check is deleted.

  As described above, in the present embodiment, characters are extracted based on the utterance image, and the voice generated using the extracted characters is transmitted to the communication destination, so that the intended content can be obtained without speaking. Can tell. Further, even when an emergency situation occurs during a call, it is possible to immediately contact a pre-registered communication destination, which improves convenience.

  As mentioned above, although embodiment which concerns on this invention has been demonstrated, this invention is not limited to an example of above-described embodiment, Various modifications are included.

  For example, the mobile terminal device 10 may generate an e-mail based on characters extracted by lip reading and transmit it to a communication destination.

  In addition, for example, the character extraction unit 112 extracts characters based on the image acquired by the image acquisition unit 111. However, for example, the character extraction unit 112 further extracts characters by using the audio acquired by the audio acquisition unit 118 as an auxiliary. You may do. Thereby, the accuracy of extraction is further improved as compared with the case where characters are extracted based only on the speech image.

  Further, for example, the above-described exemplary embodiment has been described in detail for easy understanding of the present invention, and the present invention is not limited to the one having all the configurations described here. A part of the configuration of an example of an embodiment can be replaced with the configuration of another example. Moreover, it is also possible to add the structure of another example to the structure of an example of a certain embodiment. In addition, for a part of the configuration of an example of each embodiment, another configuration can be added, deleted, or replaced. Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. In addition, the control lines and information lines in the figure indicate what is considered necessary for the description, and do not necessarily indicate all of them. It can be considered that almost all configurations are connected to each other.

  In addition, the functional configuration of the mobile terminal device 10 is classified according to main processing contents in order to facilitate understanding. The present invention is not limited by the way of classification and names of the constituent elements. The configuration of the mobile terminal device 10 can also be classified into more components depending on the processing content. Moreover, it can also classify | categorize so that one component may perform more processes.

  DESCRIPTION OF SYMBOLS 10: Portable terminal device, 110: Control part, 111: Image acquisition part, 112: Character extraction part, 113: Voice generation part, 114: Display part, 115: Communication part, 116: Input part, 117: SOS transmission instruction reception , 118: voice acquisition unit, 120: storage unit, 121: SOS setting information, 130: CPU, 131: auxiliary storage device, 132: network I / F, 133: memory, 134: input I / F, 135: output I / F, 136: Touch panel, 137: Camera unit, 138: Display, 140: Home screen, 141: Call mode selection area, 142: Setting button, 150: Call screen, 151: Call mode display area, 152: Number selection Area, 153: Call button, 160: Lip reading screen A, 161: Call type display area, 162: Image display area, 163: Text table 164: End button, 170: Lip reading screen B, 171: Call type display area, 172: Image display area, 173: Text display area, 174: Input end button, 175: Cancel button, 180: Lip reading communication setting screen 181: Retry setting button 182: SOS setting button 183: Lip reading history display button 190: SOS setting screen 191: ON / OFF button 192: SOS setting display area 200: Lip reading history display screen

Claims (11)

An image acquisition unit for acquiring an image of mouth movement;
A character extraction unit that extracts characters corresponding to the movement of the mouth based on the image acquired by the image acquisition unit;
A voice generation unit that generates voice using the characters extracted by the character extraction unit;
A communication unit that transmits the voice generated by the voice generation unit to a first communication destination;
A portable terminal device comprising:
The mobile terminal device according to claim 1,
A portable terminal device comprising: a display unit that displays the image acquired by the image acquisition unit and the character extracted by the character extraction unit.
The mobile terminal device according to claim 2,
An input unit that accepts correction of the character displayed by the display unit;
The voice generation unit generates the voice using the corrected character received by the input unit.
It is a portable terminal device according to claim 2 or 3,
A storage unit for storing communication destination information in which a message and a second communication destination are associated;
The communication unit makes a call to the second communication destination associated with the message in the communication destination information when the message is included in the character extracted by the character extraction unit. Terminal device.
The mobile terminal device according to claim 4,
An SOS transmission instruction receiving unit that receives an instruction of SOS transmission that is transmission to the second communication destination;
The communication unit receives the SOS transmission instruction during the communication with the first communication destination, and when the character extraction unit extracts the character including the message, A mobile terminal device that disconnects communication with a first communication destination and makes a call to the second communication destination.
The mobile terminal device according to claim 5,
The communication unit further transmits a call to the second communication destination when the image acquisition unit acquires the image of the mouth having a predetermined shape for a predetermined time.
The mobile terminal device according to any one of claims 2 to 6,
The display unit further displays a home screen for accepting selection of a continuous call mode or a single call mode,
The communication unit transmits the audio generated based on the image acquired by the image acquisition unit to the first communication destination after communication with the first communication destination is established when the continuous call mode is selected. And when the said single transmission mode is selected, the said audio | voice is transmitted to the said 1st communication destination where communication was established after the said image acquisition part acquired the said image, The portable terminal device characterized by the above-mentioned.
A portable terminal device according to any one of claims 2 to 7,
A voice acquisition unit for acquiring peripheral voice around the mobile terminal device;
The communication unit transmits the peripheral sound acquired by the sound acquisition unit to the first communication destination in addition to the sound generated by the sound generation unit.
The mobile terminal device according to claim 8,
The said communication part transmits the said periphery sound, when the said periphery sound acquired by the said sound acquisition part is more than predetermined | prescribed volume, The portable terminal device characterized by the above-mentioned.
A lipreading communication method using a mobile terminal device,
An image acquisition procedure for acquiring images of mouth movements;
A character extraction procedure for extracting a character corresponding to the movement of the mouth based on the image acquired in the image acquisition procedure;
A speech generation procedure for generating speech using the characters extracted in the character extraction procedure;
A communication procedure for transmitting the voice generated in the voice generation procedure to a first communication destination;
A lipreading communication method characterized by comprising:
A program that causes a computer to function as a mobile terminal device,
An image acquisition procedure for acquiring images of mouth movements;
A character extraction procedure for extracting a character corresponding to the movement of the mouth based on the image acquired in the image acquisition procedure;
A speech generation procedure for generating speech using the characters extracted in the character extraction procedure;
A communication procedure for transmitting the voice generated in the voice generation procedure to a first communication destination;
A program that causes a computer to execute.

Images