JP4600643B2 - Videophone device having character display function and voice character conversion display method in videophone device - Google Patents

Description

  The present invention relates to a videophone device having a character display function and a method of converting and displaying characters on the basis of received voice data in the videophone device, and more particularly to a mobile phone, a PHS phone, a portable information terminal, and the like equipped with a videophone function. The present invention relates to a videophone device having a character display function suitable for application to a mobile communication terminal or the like, and a method for displaying characters by converting them into characters based on received voice data.

  Recently, mobile phones having a camera and a monitor and having a videophone function have become widespread. In order to make a call using the video function with this type of phone, you can shoot yourself with the camera and make a call while watching the monitor, so when using a phone that does not have a video phone function, It is difficult to talk while holding the earpiece close to the ear on the side of the user's head.

  For this reason, a hands-free function is used in which a microphone with an earphone is used to receive a received sound with the earphone, or a voice is amplified from the speaker using a microphone and a speaker. The former is mainly used in an environment where there is a problem if sound leaks to the surroundings, such as inconvenience to others when the received sound leaks to the surroundings, or when it is inconvenient if heard by others. It is often used in an environment where there is no problem with leakage, for example, at home or in a private office room.

  Normally, an incoming videophone call occurs regardless of the location of the mobile terminal, but there are few users who always wear a microphone with an earphone on the mobile terminal, and a microphone with an earphone after receiving a videophone call. If it is attached, it takes time to start a videophone, which is not convenient.

  Since the hands-free function is implemented as a function in the telephone itself, a TV telephone can be used only by operating a button, not a relationship between the telephone and another device to be attached, such as a microphone with an earphone. However, in order to make a call using the hands-free function, the other party's voice that is loudened from the speaker spreads to the surroundings, which may cause inconvenience to the surroundings or hear the contents of the call, which is inconvenient. There is a case. For example, when there is an incoming call to a mobile phone with a videophone function during a conference or when there is an incoming call on a train.

  Conventionally, a mobile phone capable of displaying text on a display unit instead of voice or together with voice has been proposed. For example, Japanese Patent Laid-Open No. 2003-18278 (Patent Document 1) receives voice data, outputs text data by voice recognition, and displays it on the screen. This text display is a mobile phone for a user who has a handicap in hearing.

  Japanese Patent Laying-Open No. 2003-188948 (Patent Document 2) discloses a mobile phone for a user with a handicap capable of displaying text together with voice output.

  These conventional technologies display text on a mobile phone intended for a specific user, and do not display text on a video phone or a phone having a video phone function intended for general users.

Japanese Patent Laid-Open No. 2003-18278 Japanese Patent Laid-Open No. 2003-188948

  SUMMARY OF THE INVENTION An object of the present invention is to provide a videophone or a mobile communication terminal having a videophone function capable of converting received voice data into text and displaying the text in a videophone device or a mobile communication terminal having a videophone function. is there.

  Another object of the present invention is to provide a method for text-converting voice data and a method for displaying text-converted characters in a videophone device or a mobile communication terminal having a videophone function.

  According to the present invention, means for extracting audio data by referring to received video data from received audio data, means for converting to text data based on the extracted audio data, and means for displaying a character string based on the text data A mobile communication terminal with a videophone function is obtained.

  Preferably, the extraction means outputs the extracted voice data corresponding to the presence of the movement of the other party's mouth included in the received video data from the received voice data.

  Preferably, the extraction means does not output the extracted voice data in a portion corresponding to the absence of movement of the other party's mouth included in the received video.

  Further, preferably, the extraction means does not output the extracted voice data when the voice level of the received data is equal to or lower than a predetermined level even if there is a movement of the other party's mouth included in the received video.

  In a desirable mode, the mobile communication terminal with a videophone function has a text display mode for activating the character string display means when a handsfree function mode in which a call voice is emitted from a speaker is not set. Operates in the text display mode.

  Furthermore, in another aspect, the mobile communication terminal with a videophone function includes storage means for storing text data supplied from the text conversion means.

  In another aspect, the mobile communication terminal with a videophone function is set to a storage mode for starting the storage of the text data when the handsfree function mode in which a call voice is emitted from a speaker is not set. If so, it operates in the save mode.

  According to the present invention, there is also provided a text display method of audio data in a mobile mobile communication terminal having a videophone function, wherein audio data is extracted from received audio data with reference to received video data, and the extracted audio data is Based on the text data, a text data display method for voice data in a mobile communication terminal with a videophone function for displaying a character string based on the text data is obtained.

  Preferably, the voice data is extracted from the received voice data in accordance with the presence of the movement of the other party's mouth included in the received video data. Further, preferably, the voice data is not extracted in a portion corresponding to the absence of movement of the other party's mouth included in the received video.

  Furthermore, according to the present invention, there is provided a text data conversion method for audio data used for a mobile communication terminal with a videophone function, wherein audio data is extracted from received audio data with reference to received video data, and the extracted audio data is Based on this, a text data conversion method for voice data used for a mobile communication terminal with a videophone function for converting text data can be obtained.

  Preferably, in the voice data text conversion method used for the mobile communication terminal with a videophone function, the voice data is extracted from the received voice data in accordance with the presence of movement of the other party's mouth included in the received video data. Done. Further, the voice data is not extracted in a portion corresponding to the absence of movement of the other party's mouth included in the received video.

  Further, according to the present invention, based on the text data, data analysis means for outputting the extracted voice data with reference to the received video data from the received voice data, the means for converting to the text data based on the extracted voice data, A videophone device including means for displaying a character string is obtained.

  According to the present invention, a text display mode can be set as necessary in a videophone device or mobile communication with a videophone function, so that an image can be sent to the call partner while the caller is imaged, and the image of the call partner can be used as a monitor. Since it is possible to project and display the content of the other party's speech as a text message, videophone calls can be made even in an environment where a call using the hands-free function may cause trouble to the surroundings or when it is inconvenient to hear the content of the call be able to.

  The text conversion of the audio data in the embodiment of the present invention is to convert the audio data when the audio data is present and there is movement in the mouth of the video data, so that the transmitting side TV phone is used. Audio data that includes a component picked up by a microphone in the surrounding environment is extracted and converted to text when there is movement in the mouth of the video. Text conversion can be suppressed.

  In addition, text data is also stored by extracting voice data when there is movement in the mouth from voice data including a component in which ambient sounds are picked up by a microphone in an environment where the transmitting-side TV phone is used. Since the text is converted and saved, misidentified text data or unnecessary text data can be excluded from the saved text data.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram when the videophone function according to the embodiment of the present invention is realized by a mobile phone (referred to as a TV phone). In the figure, a TV telephone 100 has its control unit 101 controlling each function of the TV telephone. The control unit 101 includes a CPU 102 and performs control according to a program. 103 is a radio unit that modulates an image signal or a sound signal via an antenna and converts the signal into a high frequency in the wireless band and transmits it, or it demodulates the image signal or the sound signal from the high frequency in the wireless band to communicate with the outside. I do. Reference numeral 104 denotes an LCD display unit, 105 denotes an operation unit for performing input operations such as various keys, and 106 denotes a memory unit for storing dictionary data, a phone book, an address book, and the like. The display unit 104 displays a video received from the other party's TV phone via the wireless unit 101, a character string obtained by text-converting the other party's audio data, and an image captured by the caller's own camera. The operation unit 105 performs operations such as a call start operation, a TV phone call operation, a hands-free setting, a text display setting, etc. when a call is received. 107 is a mouthpiece / microphone, 108 is a speaker, and 109 is a camera unit. The mouthpiece / microphone 107 inputs a caller's voice during a telephone call or a videophone call, and converts it into an electrical signal. The speaker 108 outputs a ring tone of a telephone / TV phone or a voice of a call partner when hands-free is set. The camera 109 shoots itself when using the videophone function. Further, the TV phone includes a data analysis unit 110 and a text conversion unit 111. The data analysis unit 110 analyzes the video signal sent from the call partner and the voice data of the call partner and outputs the sound data. The text conversion unit converts the voice data analyzed by the data analysis unit 111 into text data.

FIG. 2 is a plan view showing the external appearance of the TV phone when it is opened and unfolded, and the same reference numerals are assigned to the external parts corresponding to the functions of the blocks described in FIG.
In the figure, the upper panel 200 is provided with an earpiece 21, a camera 109, and a display unit 104 for receiving a received voice by ear. The video 23 is displayed. On the lower panel, an operation unit 105 in which a mouthpiece / microphone 107, a speaker 108, and buttons for various functions are arranged is arranged. When a videophone call is received, operations such as starting a videophone call, setting a hands-free function, and setting a text display function are performed using the buttons on the operation unit 105.

  FIG. 3 shows the screen display on the display unit when the text display mode is set and the screen display state when the hands-free mode is set in the TV phone. The former is shown in FIG. b).

  Referring to FIG. 6B, in the hands-free setting mode, in addition to the other party's video and own video, a display 'sound' indicating that the hands-free operation is being performed is displayed on the display unit. The voice from the other party is emitted from the speaker 108 as a loud sound, not from the earpiece 21. When there is an incoming call when the hands-free setting is set and the start button or TV button is pressed, the other party usually starts a conversation and suddenly hears loud sound and can be heard without bringing the earpiece close to the ear or wearing an earphone. Although it is very convenient, it is often inconvenient in public places.

  Screen when the text display mode is set so that the voice data of the other party can be converted into text information on the receiving side and displayed on the display screen when there is an incoming videophone call in such an inconvenient environment FIG. 3A shows the display state and the sound output state. In the figure, when the text display mode is set by the operation of the operation unit 105, “text” indicating that the text display mode is set is displayed on the display surface 25 of the display unit 104. When the text display mode is selected for the incoming call and the start button is pressed, the voice signal from the other party's caller is converted to text, and the text display portion of the display section shows the text string 24 as “・ ”Is displayed. The sound flows from the earpiece 21, but there is no radiation from the speaker. The conversion to text is performed based on the audio signal and the video signal for displaying the video 22 of the other party, details of which will be described later.

  Therefore, the TV telephone according to the present invention can make a call with a video by switching between the hands-free mode and the text display mode according to the environment where the TV telephone is located. This switching can be appropriately switched by operating the operation unit 105 during a call. The voice of the user of the TV phone is picked up by the mouthpiece / microphone 107. However, since the strength of the voice is a normal voice that is emitted when the normal telephone is used, the normal telephone can be used. In an environment, even in public places, the volume is much lower than the speaker sound during a hands-free call.

  Next, conversion from voice data to text will be described with reference to FIG. In this embodiment, the text is created from the received voice data, but the data analysis unit 110 analyzes the received video and voice data of the other party under the control of the control unit 101. In the analysis of the voice data, whether the other party is speaking is analyzed from the presence / absence of the voice data and the other party's video (for example, mouth movement) data. The patterns are classified into the following four patterns according to the presence / absence of audio data and the image of the other party.

  In pattern 1, there is no audio data, and there is no movement in the image of the other party, and no audio data is extracted in this state. Therefore, text conversion and character string display are not performed. FIG. 4A shows the state of the upper panel in the pattern 1 state. The image of the other party is displayed on the image display surface of the display unit 104, and the mouth 41 is closed by the image.

  In pattern 2, there is audio data and there is no movement in the image of the other party. In this state, the audio data is regarded as noise, and no audio data is extracted. Therefore, text conversion and character string display are not performed. FIG. 4B shows a pattern 2 in which a sound 42 is emitted from the earpiece 21 but the mouth 41 is closed, and the sound in this state is regarded as noise.

  In pattern 3, there is no audio data and there is movement in the video of the other party. In this state, it is assumed that the other party is moving differently from the utterance (for example, a mouth movement occurs but there is no voice output), and no voice data is extracted. Therefore, text conversion and character string display are not performed. (C) of FIG. 4 is the pattern 3, and the mouth 41 is open, but there is no reception sound.

  In pattern 4, there is audio data and there is movement in the video of the other party. In this state, it is assumed that the other party is speaking, and voice data is extracted. Based on this extracted data, text conversion and character string display are performed. In FIG. 4D, the voice 42 is emitted from the earpiece 21, the mouth 41 is opened, and a character string obtained by converting the received voice data into text is displayed in the lower part 24 of the image of the other party.

  For detecting the movement of the mouth, for example, a known technique such as feature point extraction can be used.

  FIG. 5 is a flowchart for explaining the operation of the TV telephone of the present invention. The operation will be described with reference to FIG. 1, FIG. 1, FIG. 2, FIG. 3 and FIG.

  When the video phone A makes a call to the video phone B (step 501), the video phone A arrives at the video phone B via the communication line (step 502). A videophone call is started by operating the start button of the operation unit 104 or the TV button (step 503). The control unit 101 ticks whether the hands-free mode is set (step 504). In the case of the hands-free mode, the videophone is continued in that mode (step 512). Therefore, the voice from the other party is emitted from the speaker 108 and a videophone call is performed in a situation where the other party's image is displayed on the display screen.

  If the setting is not the hands-free mode, the control unit checks whether the text display mode is set (step 505). If the text display mode is not set, the videophone call is continued (step 512). In the TV phone in this case, since the voice is emitted from the earpiece, it is difficult to hear the call while looking at the screen.

  If the text display mode is set, the data analysis unit 110 checks whether there is audio data in the received signal (step 506). If there is no audio data, it is checked whether or not the videophone is being used (step 511). If the audio phone is being used, it is checked whether there is audio data (step 506). If the videophone is not being continued in step 511, communication disconnection processing is performed (step 513).

  If there is audio data in step 506, the data analysis unit 110 checks whether the mouth is moving in the image of the incoming call partner (step 507). The process goes to step 511 for checking and if it is continuing, the process of checking the presence of the audio data and the process of the loop of whether the mouth of the video of the other party is moving is executed.

  In step 507, if the mouth is moving in the video of the incoming call partner, the data analysis unit 110 extracts voice data (step 508). The extraction result of the voice data, that is, the voice data uttered by the other party is supplied to the text conversion unit. Based on the extraction result, the text conversion unit 111 recognizes voice and converts it into characters (step 509), and displays the text on the display unit (step 510). When the text display process is performed, it is checked whether the videophone is being continued (step 511). If the videophone is being continued, the presence of the next voice data and whether there is movement in the mouth in the incoming video are checked. The operation of displaying text one after another is performed. Eventually, if it is determined in step 511 that the TV phone is not being used, communication disconnection processing is performed (step 513), and the process ends.

  Note that the text display on the display unit 104 may be a multi-line display as shown in FIG. 6B, even if the text string shown in FIG. Also good.

  Next, a second embodiment of the present invention will be described. In the first embodiment, voice data is converted into text and a character string is displayed on the screen. However, in this embodiment, text information can be stored in a memory inside or outside the TV phone. At this time, instead of simply converting the voice data into text data and storing it, the voice data related to the operation of the other party's mouth is converted into text data and stored.

  FIG. 7 schematically shows a case in which the image is stored in an external storage device 61. The screen of the display unit 104 of the TV phone includes the image 22 of the other party of the call and the image 23 of the user captured by his camera. In the mode display area 25, “memory” indicating the storage mode is displayed. When this mode is set, the voice data is converted to text and the text is stored in the storage device.

  Also in the second embodiment, conversion of voice data into text data is performed in the same manner as in the first embodiment, so FIGS. 1, 2, 3, and 3 used in the first embodiment. The operation of the second embodiment will be described with reference to the flowchart of FIG.

  When the video phone A makes a call to the video phone B (step 801), the video phone A arrives at the video phone B via the communication line (step 802). A videophone call is started by operating the start button or the TV button of the operation unit 104 (step 803). The control unit 101 ticks whether the hands-free mode is set (step 804). In the case of the hands-free mode, the videophone call is continued in that mode (step 812). Therefore, the voice from the other party is emitted from the speaker 108 and a videophone call is performed in a situation where the other party's image is displayed on the display screen.

  If the setting is not the hands-free mode, the control unit checks whether there is a memory storage mode setting (step 805). If the memory storage mode is not set, the videophone call is continued (step 812). In the TV phone in this case, since the voice is emitted from the earpiece, it is difficult to hear the call while looking at the screen.

  If the memory storage mode is set, the data analysis unit 110 checks whether there is audio data in the received signal (step 806). If there is no audio data, it is checked whether or not the videophone is being used (step 811), and if it is continuing, it is checked whether there is audio data (step 806) and loop processing is executed. If the videophone is not being continued in step 811, communication disconnection processing is performed (step 813).

  If the voice data exists in step 806, the data analysis unit 110 checks whether the mouth is moving in the video of the incoming call partner (step 807). If the mouth is not moving, is the video phone continued? Step 711 for checking is performed, and if it is continued, the processing in a loop of steps for checking the presence of the voice data and checking whether the mouth of the video of the called party is moving is executed.

  In step 807, when the mouth is moving in the video of the incoming call partner, the voice data analysis unit 110 extracts voice data (step 808). The extraction result of the voice data, that is, the voice data uttered by the other party is supplied to the text conversion unit. The text conversion unit 111 converts it into a character string (step 809) and stores the text information in the external memory 61 (step 810). Next, it is checked whether or not the videophone is ongoing (step 811). If the videophone is ongoing, the presence of the next voice data and whether there is movement in the mouth in the incoming video are checked, and the text is converted one after another To save to memory. Therefore, the character string is stored in the memory. Eventually, when it is determined in step 811 that the TV phone is not being used, communication disconnection processing is performed (step 813), and the process ends.

  If the call message is stored as text on the TV phone as in the second embodiment, the amount of data can be smaller than the voice data itself. Therefore, more messages can be stored. In addition, if the text data is saved, the message can be reused for a document or e-mail.

  In the case of the present embodiment, the audio data is not simply converted into text data and stored, but the audio data to be converted into text is extracted in association with the video. In other words, since the voice data when the voice data exists and the movement in the mouth of the video data exists is converted into text, the voice data sent from the transmission side is used in an environment where the TV phone on the transmission side is used. Including the component picked up by the microphone in the surrounding sound, but the voice data when there is movement of the mouth in relation to the movement of the mouth in the video is extracted and converted to text, so misidentified voice data or unnecessary The text conversion of the voice data is suppressed. Therefore, misidentified text data or unnecessary text data can be excluded from the stored text data.

It is a block diagram of a mobile phone showing an embodiment of the present invention. It is the top view which opened the upper panel of the mobile telephone which shows embodiment of this invention. It is a top view which shows the state of the text display mode and hands-free mode which are used with the mobile telephone of embodiment of this invention. It is a schematic diagram which shows the relationship pattern of the audio | voice and image | video used with the mobile telephone of embodiment of this invention. It is a flowchart which shows the operation example of the mobile telephone of this invention. The text display used in the embodiment of the present invention is also a schematic diagram showing a screen display example. It is a figure which shows the 2nd Embodiment of this invention typically. It is a flowchart which shows the operation example of the mobile telephone of the 2nd Embodiment of this invention.

Explanation of symbols

21 Earpiece 22 Call partner image 23
24 text display area 25 mode display area 61 external memory 100 mobile phone with videophone function 101 control unit 103 radio unit 104 display unit 105 operation unit 108 speaker 109 camera 110 data analysis unit 111 text conversion unit

Claims (10)

A mobile communication terminal with a videophone function that can be switched between a text display mode and a voice output mode, with reference to the received video data, the absence of movement of the other party's mouth included in the received video from the received voice data Means for extracting voice data in response to the presence of movement of the other party's mouth included in the received video data without extracting voice data in the corresponding part ; means for converting into text data based on the extracted voice data; Means for displaying a character string based on the text data, and the character string is displayed together with the received video when the videophone function is operating in the text display mode. 2. The videophone function according to claim 1, wherein the extraction means does not extract voice data when the voice level of the received data is equal to or lower than a predetermined level even if there is movement of the other party's mouth included in the received video. Mobile communication terminal. 2. When the hands-free function mode in which call voice is radiated from a speaker is not set and the text display mode for starting the character string display means is set, the text display mode operates. Or the mobile communication terminal with a video telephone function of 2 description. 4. The mobile communication terminal with a videophone function according to claim 1, further comprising storage means for storing text data supplied by the text conversion means. 5. The operation according to claim 4, wherein when a hands-free function mode in which a call voice is emitted from a speaker is not set and a saving mode for starting saving of the text data is set, the operation is performed in the saving mode. Mobile communication terminal with videophone function. This is a text data display method for audio data in a mobile communication terminal with a videophone function that can be switched between text display mode and voice output mode, and text is displayed along with received video when the videophone function is operating in text display mode. In response to the absence of movement of the other party's mouth included in the received video from the received audio data with reference to the received video data, the audio data is not extracted from the received audio data and is included in the received video data. In a mobile communication terminal with a videophone function that extracts voice data in response to the presence of movement of the other party's mouth, converts the extracted voice data into text data, and displays a character string based on the text data Text display method for audio data. The mobile communication with a videophone function according to claim 6, wherein the voice data is not extracted when the voice level of the received data is equal to or lower than a predetermined level even if there is movement of the other party's mouth included in the received video. Text display method of voice data on the terminal. Text data conversion method for audio data used for mobile communication terminals with videophone function that can be switched between text display mode and voice output mode, and text is displayed along with received video when videophone function is operating in text display mode In the portion corresponding to the absence of movement of the other party's mouth included in the received video from the received audio data with reference to the received video data, the call included in the received video data without extracting the audio data A voice data text conversion method used for a mobile communication terminal with a videophone function, which extracts the voice data corresponding to the presence of movement of the other party's mouth and converts the voice data into text data based on the extracted voice data. 9. Mobile communication with a videophone function according to claim 8, wherein the voice data is not extracted when the voice level of the received data is equal to or lower than a predetermined level even if there is movement of the other party's mouth included in the received video. Text data conversion method for audio data used for terminals. A videophone that can switch between text display mode and audio output mode, and refers to the received video data, and the audio corresponding to the absence of movement of the other party's mouth included in the received video from the received audio data Means for extracting voice data corresponding to the presence of movement of the other party's mouth included in the received video data without extracting data; means for converting into text data based on the extracted voice data; And a means for displaying a character string based on the character string, the character string being displayed together with an image when the videophone is operating in a text display mode.

Images