KR101932097B1 - Method and apparatus for providing message service using voice of user - Google Patents

Abstract

The present invention relates to a method and apparatus for providing a message service using a voice capable of voice-converting the contents of a received message in a user device and outputting the same to a user or creating a reply message for a received message by voice input by a user, A method of supporting a message service of a user device includes the steps of: outputting a pop-up window informing reception of a message when a message is received; Receiving a menu selection of one of a voice listening and a voice reply menu through a pop-up window; If the selected menu is a voice listening menu, converting the contents of the message into voice; If the selected menu is a voice reply menu, converting the voice inputted from the user into text; Calling a communication application corresponding to the type of the received message; Inputting the converted text into a message content input window of the communication application; And transmitting the input text in response to a user's message transmission.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for using a voice message,

The present invention relates to a method and an apparatus for providing a message service of a user device, and more particularly, to a message service method using a voice capable of converting a received message into voice and outputting the converted voice as a reply message, And apparatus.

2. Description of the Related Art [0002] With the recent development of digital technology, various portable terminals capable of mobile communication and personal information processing such as a mobile communication terminal, a PDA (Personal Digital Assistant), an electronic notebook, a smart phone, a tablet PC, Such a mobile terminal has reached the stage of mobile convergence that does not stay in the conventional inherent domain but covers the area of other terminals. Typically, the mobile terminal is capable of transmitting a text message such as voice call, video call, SMS (Short Message Service) / MMS (Multimedia Message Service), electronic notebook, photographing, e- And various functions such as music playback, schedule management, social networking service (SNS), friend finding service, instant messenger, dictionary, game, and wireless LAN (WLAN) link formation.

In recent years, a wireless Internet service that can use the Internet via a wireless communication network even while a user is moving without restrictions on places is emerging. Therefore, not only can the users communicate with each other while freely moving anytime and anywhere using the voice call service, they can access various contents (e.g., games, chat, securities, financial transactions, news , Weather, sports, traffic information, etc.) in various forms such as text, voice, and images.

In particular, a text / multimedia message service capable of sending and receiving messages in real time between a plurality of users, and an instant message service capable of chatting among a plurality of users and exchanging various data are attracting attention. The instant message service includes a chat service for chatting with a plurality of users via text, voice, and the like, a data transfer service for exchanging various data such as a still image, a moving image, and music with the other user, various information (for example, Weather, music, E-mail, etc.). The user can use the chat service, the data transmission service, the information providing service, and the SMS / MMS message transmission service without being restricted by mobility by using the message service.

However, in a situation where the user device can be used, the message service can be used without restriction. However, in a situation where the user can not operate the user device smoothly, such as when the user is driving or traveling at a busy distance, You often encounter difficult situations in entering replies to a message. Therefore, there is a need for a new message service considering the safety and convenience of the user in providing the message service.

It is an object of the present invention to provide a method and apparatus for using a voice message to perform confirmation and reply of a received message according to a voice command.

It is another object of the present invention to provide a message service method and apparatus using a voice capable of converting various messages into voice, converting the inputted voice data into text, and transmitting the converted voice as a reply message.

Another object of the present invention is to allow a user to easily check and reply to a received message regardless of the message type by allowing the IME to process the TTS or STT function according to the user selection menu in conjunction with the speech engine when the message is received And a method and apparatus for providing a message using a voice.

It is another object of the present invention to provide a method and apparatus for using a voice message service that can improve user convenience and usability of a user device by implementing an optimal environment for supporting message confirmation and creation in a user device.

According to another aspect of the present invention, there is provided a method of supporting a message service of a user device, the method comprising: outputting a pop-up window informing reception of a message when a message is received; Receiving a menu selection of one of a voice listening and a voice reply menu through a pop-up window; If the selected menu is a voice listening menu, converting the contents of the message into voice; If the selected menu is a voice reply menu, converting the voice inputted from the user into text; Calling a communication application corresponding to the type of the received message; Inputting the converted text into a message content input window of the communication application; And transmitting the input text in response to a user's message transmission.

According to an embodiment of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for causing a processor to execute the method.

According to an aspect of the present invention, there is provided an apparatus including: a storage unit storing at least one program; And a control unit for executing the at least one program to support a message service using voice, the at least one program comprising the steps of: outputting a pop-up window informing reception of a message when a message is received; Receiving a menu selection of one of a voice listening and a voice reply menu through a pop-up window; If the selected menu is a voice listening menu, converting the contents of the message into voice; If the selected menu is a voice reply menu, converting the voice inputted from the user into text; Calling a communication application corresponding to the type of the received message; Inputting the converted text into a message content input window of the communication application; And transmitting the input text in response to the user's message transmission.

The storage unit according to an exemplary embodiment of the present invention includes an IME (Input Method Editor) for controlling a text input portion when a user inputs a voice or a character, and the IME is connected to a communication application through an input connection And is activated in a user device and resides in a demon form to detect an event that a message is received. When a voice listening menu is selected through a pop-up window, the voice engine is activated to convert the contents of the message into voice And when the voice reply menu is selected through the pop-up window, the speech engine is operated to convert the voice inputted from the user into text and transmit the text to the communication application.

The IME according to the embodiment of the present invention maps a communication application corresponding to the type of a received message among the communication applications connected through an input connection when outputting the converted voice to record a log according to message reception, And records a message reception and a log according to the message transmission by mapping the communication application to the communication application that generated the event among the communication applications connected through the input connection.

According to an aspect of the present invention, there is provided a recording medium for outputting a pop-up window informing reception of a message when a message is received, When the voice reply menu is selected, executes a communication application corresponding to the type of the received message, inputs a text converted from the voice inputted by the user into the message content input window of the communication application, and records the program And a computer readable recording medium.

The foregoing is a somewhat broad summary of features and technical advantages of the present invention in order that those skilled in the art will be better able to understand it from the following detailed description of the present invention. Additional features and advantages of the present invention, which form the subject matter of the claims of the invention, in addition to those features and advantages, will be better understood from the following detailed description of the invention.

As described above, according to the method and apparatus for providing a message using the voice proposed in the present invention, the text message received through the IME is converted into voice, the inputted voice data is converted into a text message and transmitted, It is possible to easily confirm or reply to a message even in an environment in which user device operation is difficult, such as during moving.

According to the present invention, when a message is received, the IME allows the speech engine to process the TTS and STT functions. Therefore, according to the present invention, it is possible to easily perform the confirmation of the various messages by voice conversion of the text regardless of the type of the message and the writing of the reply message by the text conversion of the voice, while recording the message transmission / reception history corresponding to each message type And manageability.

According to the present invention, it is possible to easily and safely confirm and reply to a message through voice input through the IME even when the user is not easy to use the user device when using the message service most frequently used in the user device. Therefore, according to the present invention, by providing an optimal environment for supporting a message service, the convenience of the user can be improved and the usability, convenience, and competitiveness of the user device can be improved. The present invention can be easily implemented in all types of user devices and various devices corresponding thereto.

FIG. 1 is a view schematically showing a configuration of a user device according to an embodiment of the present invention.
FIG. 2 is a schematic diagram illustrating a platform structure for processing a message using voice according to an embodiment of the present invention. Referring to FIG.
3 is a flowchart illustrating a process in which a user device processes a message using voice in accordance with an embodiment of the present invention.
FIGS. 4 to 7 illustrate operation screens of a user device according to an exemplary embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that, in the drawings, the same components are denoted by the same reference symbols as possible. Further, the detailed description of well-known functions and constructions that may obscure the gist of the present invention will be omitted. In other words, it should be noted that only parts necessary for understanding the operation according to the embodiment of the present invention will be described, and descriptions of other parts will be omitted so as not to disturb the gist of the present invention.

The present invention relates to a method and apparatus for providing a message service using a voice capable of voice-converting the content of a received message in a user device and outputting the same to a user or creating a reply message for a received message by voice input by a user. According to the embodiment of the present invention, when a message is received in a situation in which the user device is difficult to operate, such as during operation or while moving, an application, IME (Input Method Editor) (Text to Speech) or Speech to Text (STT) functions through interworking between the user and the user, so that the user can easily confirm and reply to the received message.

According to an embodiment of the present invention, messages received in various forms in the IME can be processed. Also, according to the embodiment of the present invention, the IME can automatically input the contents input from the user in cooperation with the speech engine, into the message content input window of the corresponding communication application that generated the event. This can be implemented by a unique feature of the IME that can have an input connection.

In the embodiment of the present invention, the message includes various types of messages such as an SMS message, an MMS message, an instant message, a mail message, For convenience, they are collectively called messages. In the embodiment of the present invention, the communication application refers to an application supporting message transmission / reception with another user, and may be a text / multimedia message application, a messenger (or chat) application, a mail application, a social networking service And the like are collectively referred to as a communication application hereinafter for convenience of explanation.

Hereinafter, a configuration of a user device and an operation control method according to an embodiment of the present invention will be described with reference to the following drawings. It should be noted that the configuration of the user device and the operation control method thereof according to the embodiment of the present invention are not limited to or limited to the following description, and thus can be applied to various embodiments based on the following embodiments .

FIG. 1 is a view schematically showing a configuration of a user device according to an embodiment of the present invention.

1, the user device of the present invention includes a wireless communication unit 110, a user input unit 120, a display unit 130, an audio processing unit 140, a storage unit 150, an interface unit 160, a controller 170, and a power supply unit 180. The user device of the present invention is not necessarily the components shown in FIG. 1, but may be implemented with more or fewer components.

The wireless communication unit 110 may include one or more modules for enabling wireless communication between a portable terminal and a wireless communication system or between a user device and a network in which a user device is located. For example, the wireless communication unit 110 may include a mobile communication module 111, a wireless local area network (WLAN) module 113, a short range communication module 115, a position calculation module 117, (119), and the like.

The mobile communication module 111 transmits and receives a radio signal to at least one of a base station, an external device, and a server on a mobile communication network. The wireless signal may include various types of data for transmitting and receiving a voice call signal, a video call signal or a text / multimedia message, various types of data according to an instant message transmission / reception, and feed information.

The wireless LAN module 113 represents a wireless Internet connection and a module for forming a wireless LAN link with other user devices, and may be built in or enclosed in a user device. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies. The wireless LAN module 113 may transmit or receive a message defined in the present invention over the wireless Internet.

The short-range communication module 115 represents a module for short-range communication. Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IRDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC) Can be used.

The position calculation module 115 is a module for acquiring the position of the user device, and a representative example thereof is a GPS (Global Position System) module. The position calculation module 115 calculates distance information and accurate time information from three or more base stations and then calculates the current position information of three dimensions according to latitude, longitude, and altitude by applying trigonometry to the calculated information can do. Or the position calculation module 115 can calculate the position information by continuously receiving the current position of the user device from three or more satellites in real time. The location information of the user device can be obtained by various methods.

The broadcast receiving module 119 receives a broadcast signal (e.g., a TV broadcast signal, a radio broadcast signal, a data broadcast signal, etc.) from an external broadcast management server through a broadcast channel (e.g., a satellite channel or a terrestrial channel) (E.g., information related to a broadcast channel, a broadcast program, or a broadcast service provider).

The user input unit 120 generates input data for a user to control operation of the user device. The user input unit 120 may include a key pad, a dome switch, a touch pad (static pressure / static electricity), a jog wheel, a jog switch, and the like .

The display unit 130 displays (outputs) information processed by the user device. For example, when the user device is in the call mode, a user interface (UI) associated with the call or a graphical user interface (GUI) is displayed. Also, the display unit 130 displays a captured image and / or a received image or a UI and a GUI when the user device is in a video communication mode or a shooting mode. The display unit 130 displays a UI or a GUI associated with a message collected from the inside or the outside. In particular, the display unit 130 can display a message transmitted and received through the communication application on the execution screen of the communication application. A screen configuration of the display unit 130 operated in the present invention and an operation example thereof will be described later.

The display unit 130 may be a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), a light emitting diode (LED), an organic light emitting diode ), An active matrix OLED (AMOLED), a flexible display, a bended display, and a 3D display. Some of these displays may be implemented as transparent displays that are configured as transparent or light transmissive so as to view the exterior.

According to the present invention, when the display unit 130 and the touch panel that detects the touch operation form a layer structure (hereinafter, referred to as 'touch screen'), It can also be used as an input device. The touch panel may be configured to convert a change in a pressure applied to a specific portion of the display unit 130 or a capacitance generated in a specific portion of the display unit 130 into an electrical input signal. The touch panel can be configured to detect not only the position and area to be touched but also the pressure at the time of touch. If there is a touch input to the touch panel, the corresponding signal (s) is sent to a touch controller (not shown). The touch controller (not shown) processes the signal (s) and then transmits the corresponding data to the controller 170. Thus, the control unit 170 can know which area of the display unit 130 is touched or the like.

The audio processing unit 140 transmits the audio signal received from the control unit 170 to the speaker 141 and transmits the audio signal such as voice received from the microphone 143 to the control unit 170. The audio processing unit 140 converts the audio / sound data into audible sound through the speaker 141 under the control of the control unit 170, and converts the audio signal, such as voice, received from the microphone 143 into a digital signal, (170).

The speaker 141 may receive audio data from the wireless communication unit 110 or store audio data stored in the storage unit 150 in a communication mode, a recording mode, a voice recognition mode, a broadcast receiving mode, or the like. The speaker 141 may output an acoustic signal related to a function performed in the user device (e.g., a call signal reception tone, a message reception tone, a music content reproduction, etc.).

The microphone 143 receives an external sound signal in a communication mode, a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 111 when the voice data is in the call mode, and output. The microphone 143 may be implemented with various noise reduction algorithms for eliminating noise generated in the process of receiving an external sound signal.

The storage unit 150 may store a program for processing and controlling the controller 170 and may store data input / output (e.g., a phone book, a message, an audio, a still image, Log information, and the like). The storage unit 150 may store the frequency of use of each of the data (for example, each telephone number, each message, frequency of use for each multimedia, and the like) and importance. In addition, the storage unit 150 may store data related to vibration and sound of various patterns output upon touch input on the touch screen. In particular, the storage unit 150 may map and store the messages transmitted or received through the communication application and the log information related to the transmission or reception of the messages to the executed communication application. The storage unit 170 may also store the platform of FIG. 2, which will be described later.

The storage unit 150 may include a memory such as a flash memory type, a hard disk type, a micro type, and a card type memory (e.g., SD or XD memory) (Random Access Memory), SRAM (Static RAM), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable ROM), PROM (Programmable ROM), MRAM, ), And an optical disk type of memory. The portable terminal may operate in association with a web storage that performs a storage function of the storage unit 150 on the Internet.

The interface unit 160 serves as a pathway to all external devices connected to the user device. The interface unit 160 receives data from an external device, receives power from the external device, transfers the data to each component in the user device, or allows data in the user device to be transmitted to the external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video I / O port, an earphone port, and the like may be included in the interface unit 160.

The control unit 170 controls the overall operation of the user device. For example, for voice calls, data communications, video calls, and the like. The control unit 170 may include a multimedia module (not shown) for multimedia playback. The multimedia module may be implemented in the controller 170 or separately from the controller 170. [ Particularly, the control unit 170 receives the message of the present invention and receives a menu selection for listening to a voice or a voice reply to the received message, and when the voice listening menu is selected, When the voice reply menu is selected, a communication application according to the type of the received message is executed, and the voice of the user is text-converted and input to the message input window of the communication application. That is, the control unit 170 controls overall control related to the message input and output function using the voice of the present invention. The detailed control operation of the controller 170 will be described in the operation example of the user device and the control method thereof with reference to the drawings described later.

The power supply unit 180 receives external power and internal power under the control of the controller 170 and supplies power necessary for operation of the respective components.

Meanwhile, the various embodiments described in the present invention can be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. According to a hardware implementation, the embodiments described in the present invention can be applied to a wide range of applications including application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) , Microprocessors, microprocessors, and other electronic units for performing other functions, as will be apparent to those skilled in the art. In some cases, the embodiments described herein may be implemented by the controller 170 itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein.

The user device of the present invention shown in FIG. 1 may include an application processor (AP), a graphics processing unit (GPU), a central processing unit (CPU), and the like, as well as all information communication devices, multimedia devices, Unit). ≪ / RTI > For example, the user device may include a mobile communication terminal operating according to communication protocols corresponding to various communication systems, a tablet PC (personal computer), a smart phone, A digital camera, a portable multimedia player (PMP), a media player, a portable game terminal, a laptop computer, and a PDA (Personal Digital Assistant). In addition, the function control method of the present invention can be applied to various display devices such as a digital TV, a digital signage (DS), and a large format display (LFD).

FIG. 2 is a schematic diagram illustrating a platform structure for processing a message using voice according to an embodiment of the present invention. Referring to FIG.

2, the platform of the user device according to the embodiment of the present invention may include OS (Operating System) -based software for performing various operations related to input and output of messages using the voice . 7, a user device according to an exemplary embodiment of the present invention includes a framework 210, an IME 220, a voice engine 230, and an application 240 .

The framework 210 may include a program that is the basis of an application within the application 240 layer. The framework 210 is compatible with any application and may be capable of reusing, moving, or exchanging components. The framework 210 may include a support program, a program for connecting other software components, and the like. For example, a resource manager, an activity manager associated with an application activity, a notification manager, a content provider that summarizes shared information between applications, a message manager, a message manager, and the like. In the present invention, the framework 210 exemplifies a framework for managing text / multimedia messages, a framework for messenger management, and a framework for mail management.

The IME 220 represents a module for controlling a text input portion at the time of inputting a voice of a user or inputting a character. That is, the IME 220 controls the input of text corresponding to a user inputting text using a user device or inputting a voice through a voice interface (e.g., microphone 143). The IME 220 may be assigned and executed corresponding to each application in the application 240 layer.

The speech engine 230 is an engine for converting an input signal into text or voice. The speech engine 230 is composed of a TTS engine that converts the text of a message into speech, and a STT engine that converts speech into text of a message . In the present invention, the speech engine 230 can perform the voice conversion of the text of the received message in conjunction with the IME 220, or text-converted the voice inputted for the reply message.

The application 240 layer includes various programs that can be displayed and driven within the user device. For example, a UI application related to various menus and the like in the user device, an application downloaded and stored through an external device or a network, and an application that can be installed or deleted by a user can be included. (VOD) service, a web album service, an SNS, a location-based service (LBS), a map service, a web search service, an application search service, a web search service, A text / multimedia message service, a mail service, and the like can be performed. In addition, various functions such as game and schedule management can be performed. In the present invention, it is exemplified that the application 240 is a communication application such as a text / multimedia message application, a messenger application, and a mail application.

In addition, the platform of the present invention may further include an OS kernel (not shown) and middleware (not shown). The OS kernel is the core of the OS. It provides a hardware driver for operating a user device, security of hardware and a processor in a user device, efficient management of system resources, memory management, hardware abstraction, Process, and schedule management according to a multi-process. The hardware driver in the OS kernel (not shown) may include at least one of a display driver, a Wi-Fi driver, a Bluetooth driver, a USB driver, an audio driver, a Power management, a Binder driver, . The middleware (not shown) is located between the OS kernel and the application 240 and can serve as intermediary between other hardware or software to exchange data. Thus, a standardized interface can be provided, various environment support, and system can be interworked with other tasks.

Meanwhile, in the embodiment of the present invention, when a message such as a text / multimedia message, an instant message, or a mail message is received, the IME 220 can easily confirm and reply to a message received through the TTS or STT, .

For example, in the platform of the present invention, when the IME 220 receives a message received through the wireless communication unit 110 through the framework 210, the IME 220 transmits caller information (e.g., telephone number, , E-mail address, etc.) and the content of the message. When the IME 220 receives the new message, the IME 220 may feedback the user through the pop-up window 400 as shown in FIG. That is, the IME 220 can receive a message reception event through the wireless communication unit 110, and the reason why the IME 220 processes a message according to a reception event is as follows. Can be defined.

In particular, due to the characteristics of the message, it is necessary to maintain a chat history of conversation according to transmission / reception of the message according to transmission / reception of messages with various other users, and to maintain various chat messages (text / multimedia message, message, Since the IME 220 can support the above-described functions by the following features. That is, the IME 220 can connect an input connection (IC, input connection) to input text into a text input window such as a text / multimedia message, a messenger application, a mail application, In the form of a daemon that is always executed in the user device in the form of a service, that is, in the form of a system program that resides in the user device and automatically provides various services when the application or system becomes a specific state. And the like.

Specifically, in order for the characters input from the keypad of the user device to be input to the input window of the application, a connection called an input connection (IC) must be created between the keypad and the input window of the application. That is, the input connection is an area unique to only the IME 220, and it is possible to input text (text) into an input window of an application connected to an input connection (IC).

In order to know that a message such as a character / multimedia message, an instant messenger message, and a mail message is inputted, it should be a service type process which always operates in a daemon form. Thus, rather than creating and maintaining a new daemon service, it may be more efficient to use an application that is always resident and running in the form of a daemon, such as the IME 220 implemented in the user device.

The reason for inputting text into an input window of a communication application such as a text / multimedia message application, a message application, and a mail application in an application connected with an input connection, that is, in the present invention, . For example, in a user device, a text message or the like leaves a log in the form of a conversation with a specific conversation partner as a concept of a chat screen. Therefore, if the reply message is not transmitted through the input window of the corresponding application connected through the input connection IC of the IME 220, there is a problem that the IME 2200 can not leave a trace of the reply in the conversation history. Message management can solve this problem.

And since the keypad of each communication application can input without input connection (IC), it can manage the message history according to sending and receiving of message. However, in order to implement TTS and STT for listening to a voice and responding to a voice of a received message as in the present invention, the corresponding communication application must have both a configuration for implementing TTS and STT, There is a limit to the type of content that can be processed. Accordingly, if the IME 220 processes itself, it is possible to process various types of messages received from various messages (text / multimedia messages, message messages, mail messages, etc.) supported by the user device in one module .

Hereinafter, an example of performing an interaction between modules in the platform of the present invention as shown in FIG. 2 will be described.

Referring to FIG. 2, when a WAP (Wireless Application Protocol) push is used, a push message is transmitted from a push application server (or a web server) of a WAP gateway And transmitted to the user device. Here, the push application server (or web server) may include a server such as a news, a competition score, a weather, a customer service billing system (CSBS) of a mobile communication company, and the like.

The push application server (or web server) can generate a message in a Protocol Data Unit (PDU) format and transmit it to the user device. Then, the message is received from the user device push application server (or web server) through the wireless communication unit 110, and the received message is transmitted to a framework 210 (specifically, a message framework) Lt; RTI ID = 0.0 > 240 < / RTI > For example, when a text message is received, it may be delivered to the text messaging application via a framework associated with the text message.

In accordance with an embodiment of the present invention, the received message is also delivered to the IME 220 via the framework 21. That is, the IME 220 is pushed on the received message. At this time, a pop-up window 400 for the received message is output through the display unit 130.

Then, the user can select one of the menu items "listen to voice" or "reply to voice" through the pop-up window 400 and execute the corresponding function. At this time, the IME 220 responds to the menu selection of the user through the speech engine 230 connected to the IME 220 through the STT text conversion according to the TTS voice conversion or the "voice reply" Or a reply message to the received message. Here, the text input through the STT through the speech engine 230 connected to the IME 220 from the user may be delivered to a corresponding application (e.g., a text messaging application) and delivered to the final counterpart.

Meanwhile, the platform as described above can be used universally in various electronic apparatuses as well as user devices of the present invention. The platform of the present invention may be stored or loaded in at least one of the storage unit 150 and the control unit 170 as described above or in a separate processor (not shown). In addition, a separate application processor (not shown) for executing an application may be further provided.

3 is a flowchart illustrating a process in which a user device processes a message using voice in accordance with an embodiment of the present invention.

Referring to FIG. 3, the controller 170 may control the output of a pop-up window to notify that the message is received (operation 301). At this time, the controller 170 estimates the message type and manages the message framework and the communication application corresponding to the message type. In the present invention, the message includes a text message, a multimedia message, an instant message, and a mail message, and the framework includes a text message framework, a multimedia message framework, an instant message framework, , The communication application may include a text / multimedia messaging application, a messenger (or chat) application, and a mail application.

The controller 170 may check whether there is a menu selection through a pop-up window (step 307). That is, the control unit 170 may wait for input reception of selecting any one of the <Listening Voice> and <Voice Reply> menus provided through the pop-up window.

If there is no menu selection (NO in step 307), the control unit 170 can control the execution of the corresponding operation (step 305). For example, the controller 170 counts the time corresponding to the pop-up window output, and turns off the display unit 130 if no menu is selected for a preset time. The control unit 170 may control the output of an alarm (for example, vibration, sound, or at least one of the screens) for the received message corresponding to the set period and the number of times when the display unit 130 is turned off. Alternatively, the control unit 170 may execute the communication application corresponding to the message according to the user input, and may control the content display of the received message through the executed communication application.

If there is a menu selection (YES in step 307), the control unit 170 can determine the type of the selected menu (step 309). That is, the control unit 170 can determine whether the menu is selected from the < voice listening > menu or the &quot; voice reply &quot;

If the controller 170 determines that the selected menu is the &quot; Listen &quot; menu (NO in 309), the controller 170 parses the message (step 311). For example, the control unit 170 may extract the contents (e.g., text) of the received message when the &quot; Listen to voice &quot; menu is selected. At this time, the controller 170 may extract sender information (e.g., a telephone number, a calling user ID, a chat conversation room, a mail address, etc.) of the message.

The control unit 170 performs voice conversion of the parsed content through the voice engine 230 and then controls output of the converted voice through the speaker 141 (step 313). At this time, the controller 170 transmits the parsed information to the IME 220 through the framework 210, and the IME 220 activates the speech engine 230 so that the voice conversion of the text and the output thereof are processed .

When the converted voice is output, the control unit 170 may record a log corresponding to the message reception (Step 315). In particular, when the converted voice is outputted through the speaker 141 under the control of the control unit 170, the IME 220 can record a message reception log by mapping to a communication application corresponding to the message type. At this time, the controller 170 may maintain the pop-up window display state.

The controller 170 can confirm whether there is a selection of the &quot; voice reply &quot; menu after voice output and log recording of the received message (Step 317). When the input for selecting the <voice reply> menu is received (YES in step 317), the controller 170 proceeds to step 321 and controls the following operations. In step 319, the control unit 170 may control the execution of the corresponding operation when no input for selecting the <voice reply> menu is received (NO in step 317). For example, the controller 170 counts the time corresponding to the pop-up window output, and turns off the display unit 130 if there is no menu selection for a preset time.

If the controller 170 determines that the selected menu is the &quot; voice reply &quot; menu (YES in step 309), the controller 170 receives the voice through the voice engine 230 in step 321 and parses the input voice in step 323 Text can be generated (step 325). That is, when the user's voice is inputted through the microphone 143, the control unit 170 can convert the inputted voice through the voice engine 230. At this time, the IME 220 may operate the speech engine 230 so that the text conversion of the speech and its generation are processed.

The controller 170 checks the message type (step 327) and can call the communication application corresponding to the message type (step 329). For example, when the type of the message is a text message, the control unit 170 calls the text message application. When the type of the message is an instant message, the control unit 170 calls the messenger (or chat application) You can call the application. In particular, the control unit 17 may invoke a communication application corresponding to the message type through the IME connected through the input connection with the communication application. At this time, the controller 170 can confirm the message type estimated in the message receiving step and push the execution of the confirmed communication application through the framework 210 corresponding to the message type.

The control unit 170 may generate a reply message using the converted text through the called communication application (step 331). For example, the control unit 170 may input the converted text into the message content input window of the message application, and input the extracted caller information into the recipient input window to create a message. Alternatively, the control unit 170 can input the converted text into the message content input window of the messenger application, and display the extracted chat chat screen. At this time, the IME 220 can transmit the converted text through the speech engine 230 to the communication application to be executed, and can display the converted text in the message content input window of the communication application.

The control unit 170 can transmit the created reply message (step 334). For example, the control unit 170 may receive a message transmission button input by the user, and may transmit the created reply message to the corresponding destination in response to the transmission button input. Alternatively, the control unit 170 may receive a user's voice command input, and may transmit the created reply message to the corresponding destination in response to the voice command.

The control unit 170 may record a log according to the transmission / reception of the message when the reply message is transmitted (step 335). In particular, when the reply message is transmitted through the wireless communication unit 110 under the control of the control unit 170, the IME 220 may map the received communication application to record a message reception and transmission log.

FIGS. 4 to 7 illustrate operation screens of a user device according to an exemplary embodiment of the present invention.

4 to 7, FIG. 4 illustrates an example of a screen of a user device when a mesh is received from an external specific message server. That is, the message may be received from the message server, and information about the received message may be displayed through the pop-up window 400. 4, the pop-up window 400 may display pop-up information (e.g., message reception notification information, message sender information, message content, etc.) in a message information area 410, And provides a listening menu button 430 and a voice reply menu button 450.

Then, the user can select one of the menu buttons 430 and 450 in the pop-up window 400. For example, the user may select a button to perform either "listen to voice" and "reply voice" in popup window 400. [ When the user selects the voice listening menu button 430, the text of the message is voice-converted according to the procedure described above, and the converted voice can be output through the speaker 141. [ At this time, the screen as shown in FIG. 4 can be maintained at the converted audio output. On the other hand, if the user selects the voice reply menu button 450, the screen of the user device can be switched as shown in FIG.

5 shows an example of a screen of the user device when the "voice reply menu button 450 &quot; for composing a reply message through the voice reply function is selected in the pop-up window 400. That is, Up window 400 may be switched to a guide window 500 for guiding voice input and may be displayed when the menu button 450 is selected. And a guide voice for guiding voice input can be output together through the speaker 141. [

Then, the user can input the contents of the message to be created according to the guide through the guide window 500 by voice. For example, the user can input the voice of "say yes". The user device displays the communication application screen as shown in FIG. 6 according to the voice input of the user. That is, the screen of the user device shown in FIG. 5 is switched as shown in FIG. 6 when the voice input of the user is completed.

6 shows an example of a screen of a user device when a user inputs a voice in accordance with the guide of the guide window 500. In FIG. As shown in FIG. 6, it is assumed that the communication application is the messenger application in the embodiment of FIG. That is, the type of the received message is a message associated with the messenger application, so that the messenger application can be automatically executed in a communication application for composing a reply message. Therefore, the converted text of the user's input voice "Tell me" can be automatically input into the message content input window 600 of the messenger application.

When the text corresponding to the user's voice input is displayed in the message content input window 600, the user can transmit the inputted message content (text). For example, the user can input a button (for example, a transmission button next to the message content input window 600) for transmitting the contents of the created message, or transmit the contents of the message created using a voice command (for example, transmission) . Then, the reply message can be transmitted to the other user device by the transmission input of the user.

7 shows an example screen of the user device when the user device transmits a text-converted reply message in response to the transmission of the user's reply message to the opponent user device via the messenger application. That is, FIG. 7 illustrates an example of a screen of the user device in which the content of the reply message is input to the messenger chat screen when the user device transmits a reply message through the messenger application.

As described above, according to the embodiment of the present invention, when a message is received, a "listen to voice" function and a "voice reply" function are provided through a pop-up window 400, Function can be used to confirm received messages or to create reply messages through STT.

That is, when the user selects "Listen to Voice" in the pop-up window 400 for an incoming message, the IME 220 interlocks with the speech engine 230 and sends the text of the message received via the speech engine 230 TTS voice conversion, and reads the voice corresponding to the text. Therefore, the user can replace the inconvenience of visually confirming the received message even in a situation where the user device is difficult to operate, such as during operation or while moving.

In addition, when the user selects "voice reply" in the pop-up window 400 for the received message, the IME 220 interlocks with the voice engine 230 and transmits the user voice input via the voice engine 230 to the STT text And generates and inserts the contents of the reply message in the text corresponding to the voice. That is, the calling user can automatically trace and execute the communication application to which the message is transmitted, and automatically input and provide text corresponding to the user's voice. Thus, the user can easily transmit the reply message to the receiver without the operation of the user device.

The embodiments of the present invention as described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, and the like, alone or in combination. The program instructions recorded on the recording medium may be those specially designed and constructed for the present invention or may be those known and used by those skilled in the computer software.

The computer-readable recording medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc) A magneto-optical medium such as a floppy disk, and a program command such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, Lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; The program instructions also include high-level language code that can be executed by a computer using an interpreter or the like, as well as machine code as produced by the compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. Accordingly, the scope of the present invention should be construed as being included in the scope of the present invention, all changes or modifications derived from the technical idea of the present invention.

110: wireless communication unit 120: user input unit
130: Display section 140: Audio processing section
150: storage unit 160: interface unit
170: control unit 180: power supply unit
210: Framework 220: IME
230: speech engine 240: application
400: Popup window 500: Guide window
600: Enter message content window

Claims (20)

A method of supporting a message service of a user device,
Displaying a pop-up window including a voice listening menu and a voice reply menu to receive information related to the message and a function related to the message when the message is received from the other user device;
Converting the contents of the message into speech based on a speech engine connected to an input method editor (IME) when a voice listening menu is selected in the pop-up window being displayed;
Converting a voice input from a user through a microphone into text based on a speech engine connected to the IME when a voice reply menu is selected in the pop-up window being displayed;
A communication application related to the message is executed through a framework related to the type of the message, a screen including the pop-up window is switched to a communication application screen including a message content input window, Process;
Transmitting the converted text to the communication application through the IME, inputting the converted text in a message content input window of the communication application, and displaying the converted text; And
Receiving a message transmission input of the user for the converted text displayed through a message content input window of the communication application and transmitting the text input in the message content input window in response to the transmission input to a reply message And transmitting the message service request message to the counterpart user device. The method as claimed in claim 1,
And recording a log according to the message reception when outputting the converted voice. 3. The method of claim 2,
And the IME connected to the communication application through an input connection maps the communication application corresponding to the type of the message to record a log corresponding to the message reception. . The method of claim 2, wherein the outputting through the speaker comprises:
And maintaining the display state of the pop-up window while outputting the converted voice. 5. The method of claim 4,
Further comprising the step of receiving the voice reply menu selection based on the pop-up window displayed on the display unit after outputting the converted voice. 2. The method of claim 1,
When the voice reply menu is selected, activating the voice engine connected to the IME to convert the voice input from the user into text, and the IME transmitting the converted text to the communication application The method comprising the steps of: The method of claim 1, wherein the step of inputting and displaying the converted text comprises:
And generating the reply message using the converted text through the communication application. 8. The method of claim 7, wherein the transmitting comprises:
And transmitting the reply message in response to a user's message transmission button input or a user's voice command input. 9. The method of claim 8,
And recording a message according to the message reception and message transmission when the reply message is transmitted. 10. The method of claim 9,
Wherein the IME connected to the communication application via an input connection maps the message to the communication application and records a log corresponding to the message reception and message transmission. A display section;
A wireless communication unit;
A storage unit in which at least one program is stored; And
And a control unit for executing the at least one program to control a voice message service,
A pop-up window including a voice listening menu and a voice reply menu for receiving information related to the message and a function related to the message when the message is received from the other user device through the wireless communication unit,
When the voice listening menu is selected in the pop-up window being displayed, the contents of the message are converted on the basis of the speech engine connected to the IME (Input Method Editor)
If the voice reply menu is selected in the pop-up window being displayed, the voice inputted from the user through the microphone is converted into text based on the speech engine connected to the IME,
A communication application related to the message is executed through a framework related to the type of the message, a screen including the pop-up window is switched to a communication application screen including a message content input window, and,
Transmitting the converted text to the communication application through the IME, inputting and displaying the converted text in a message content input window of the communication application, and
Receiving a message transmission input of the user for the converted text displayed through a message content input window of the communication application and transmitting the text input in the message content input window in response to the transmission input to a reply message To the other user device. 12. The apparatus according to claim 11,
Controls the output of the voice conversion of the message based on the voice engine connected to the IME and controls creation of a reply message corresponding to the message through input by text conversion of the input voice. . 13. The apparatus according to claim 12,
And the IME for controlling a text input portion when a user inputs a voice or when a character is input. 14. The method of claim 13,
Detecting an event that the message is received,
When the voice listening menu is selected through the pop-up window, the voice engine connected to the IME is activated to convert the contents of the message into voice,
When the voice reply menu is selected through the pop-up window, activates the voice engine connected to the IME to convert the voice input from the user into text and transmit the text to the communication application. 15. The method of claim 14,
And displays the converted text in a message content input window of the communication application in cooperation with a corresponding communication application that has generated the event. 16. The method of claim 15,
Wherein the IME is connected to the communication application via an Input Connection. 17. The method of claim 16,
And when the converted voice is output, maps the communication application corresponding to the type of the message among the communication applications connected through the input connection, and records a log according to the message reception. 17. The method of claim 16,
And when the reply message is transmitted, mapping the message to the communication application that has generated the event among the communication applications connected through the input connection, and records the log according to the message reception and message transmission. 15. The method of claim 14,
Wherein the user device is running in the user device and resides in a daemon form. A pop-up window including a voice listening menu and a voice reply menu for receiving information related to the message and a function related to the message is displayed on the display unit, and the voice listening menu is selected through the pop-up window Converts the contents of the message to speech through a speaker based on a speech engine connected to an input method editor (IME), and if the voice reply menu is selected through the pop-up window, Translating the text into a text based on a speech engine connected to the IME, confirming the type of the message, executing a communication application related to the message through a framework related to the type of the message, Switch to the communication application screen containing the content input window and display And transmitting the converted text to the communication application through the IME, inputting and displaying the converted text in a message content input window of the communication application, displaying the converted text on the message content input window of the communication application, Readable recording medium having recorded thereon a program for receiving a user's message transmission input for a text and transmitting the text entered in the message content input window in response to the transmission input as a reply message corresponding to the message, .

Images