KR20130135567A - Method and apparatus for providing message service using voice of user - Google Patents

Abstract

The present invention relates to a method and a device for providing a message service using voice which converts a received message into voice by a user device and outputs the voice to a user or creates a reply message to the received message using user voice input. The message service supporting method for a user device according to the present invention comprises the following steps: outputting a popup window for indicating message reception if message input is received; inputting a menu selection which is either voice listening or voice replying through the popup window; converting the received message into voice and outputting the voice if voice listening is selected; converting voice inputted by the user into text if voice replying is selected; calling the communication application corresponding to the the type of the received message; inputting the converted text onto the message input window of the communication application; and transmitting the inputted text in response to the message transmission of the user. [Reference numerals] (301) Message reception;(303) Message reception popup;(305,319) Perform a relevant operation;(307) Menu selection?;(309,317) Voice reply?;(311) Message phishing;(313) Output voice through a voice engine;(315,335) Record a log;(321) Input voice through a voice engine;(323) Voice phishing;(325) Generate text;(327) Check a message type;(329) Call an message application;(331) Write a replay message;(334) Transmit;(AA) Start;(BB,CC) End

Description

METHOD AND APPARATUS FOR VOICE MESSAGE {METHOD AND APPARATUS FOR PROVIDING MESSAGE SERVICE USING VOICE OF USER}

The present invention relates to a method and apparatus for providing a message service of a user device, and more particularly, to a message service method using a voice capable of converting and outputting a received message into a voice, and converting a voice input from the user into a text to be transmitted as a reply message. And to an apparatus.

Recently, with the development of digital technology, a variety of portable terminals capable of communication and personal information processing, such as mobile communication terminals, PDAs (Personal Digital Assistants), electronic notebooks, smart phones, and tablet PCs, have been released. Such mobile terminals are approaching a mobile convergence stage that covers not only their own unique areas but also other devices. Representatively, the mobile terminal transmits a text message such as a voice call, a video call, a Short Message Service (SMS) / MMS (Multimedia Message Service), an electronic notebook, photographing, an e-mail transmission, a broadcast play, a video play, the Internet, an e-commerce, Various functions may be provided, such as music playback, schedule management, social networking service (SNS), friend search service, messenger, dictionary, game, and wireless local area network (WLAN) link formation.

Moreover, in recent years, wireless Internet services that can use the Internet via a wireless communication network have emerged even while the user is moving without restriction of a place. Therefore, users can freely move and talk with the other party anytime and anywhere using the voice call service, and access various types of contents (eg, games, chats, securities, financial transactions, news, etc.) after accessing the Internet using the wireless Internet service. , Weather, sports, traffic information, etc.) can be provided in various forms such as text, voice, and image.

In particular, a text / multimedia message service capable of sending and receiving messages in real time among a plurality of users, and an instant message service capable of chatting and transmitting various data among a plurality of users are in the spotlight. The instant message service is a chat service for chatting with a plurality of users through text, voice, etc., a data transmission service for exchanging various data such as still images, videos, and music with the counterpart user, and various kinds of information (for example, news stocks, Software that provides information service that provides weather, music, e-mail, etc.). By using such a message service, a user is able to use a chat service, a data transmission service, an information service service, an SMS / MMS message transmission service, and the like without being restricted in mobility.

However, in the situation where the user device can be used, the message service can be used without any restriction, but in the situation where the user device cannot be operated smoothly, such as when the user is driving or moving on a busy street, the received message is received and received. You are often faced with a difficult situation in entering a reply message for a given message. Therefore, there is a need for a new message service considering the safety and convenience of the user in providing a message service.

An object of the present invention is to provide a message service method and apparatus using voice that can perform confirmation and reply on a received message according to a voice command.

Another object of the present invention is to provide a message service method and apparatus using a voice that converts various messages into voice and converts input voice data into text and transmits the reply message.

Still another object of the present invention is to allow a user to easily check and reply to an incoming message regardless of the message type by allowing the IME to process a TTS or STT function in accordance with a user selection menu when the message is received. The present invention provides a message service method and apparatus using voice.

Another object of the present invention is to provide a message service method and apparatus using voice that can implement an optimal environment for supporting message checking and writing in a user device, thereby improving user convenience and usability of the user device.

According to an aspect of the present disclosure, there is provided a method of supporting a message service of a user device, the method including: outputting a pop-up window notifying a message reception when a message reception is input; Receiving a menu selection of a voice listening and voice reply menu through a pop-up window; If the selected menu is a voice listening menu, converting a message content and outputting a voice; Converting a voice input from a user into text if the selected menu is a voice reply menu; Calling a communication application corresponding to the type of the received message; Inputting the converted text into a message content input window of the communication application; And transmitting the input text in response to the user's message transmission.

According to an embodiment of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for causing a processor to execute the method.

An apparatus according to an embodiment of the present invention for solving the above problems, the storage unit for storing at least one program; And a controller configured to execute the at least one program to control a message service using voice, wherein the at least one program comprises: outputting a pop-up window notifying the reception of a message when a reception of a message is input; Receiving a menu selection of a voice listening and voice reply menu through a pop-up window; If the selected menu is a voice listening menu, converting a message content and outputting a voice; Converting a voice input from a user into text if the selected menu is a voice reply menu; Calling a communication application corresponding to the type of the received message; Inputting the converted text into a message content input window of the communication application; And instructions for executing a process of transmitting the input text in response to the user's message transmission.

The storage unit according to an embodiment of the present invention includes an input method editor (IME) for controlling a text input portion when a user inputs a voice or a character, and the IME is connected to a communication application through an input connection. It is executed on the user device and resides in the form of a daemon to detect an event where a message is received, and when a voice listening menu is selected through a pop-up window, a voice engine is started to convert the contents of the message into voice and output the voice. When the voice reply menu is selected through the pop-up window, the voice engine is operated to convert the voice input from the user into text and transmit the text to the communication application.

When the IME outputs the converted voice, the IME maps to a communication application corresponding to a type of a received message among communication applications connected through an input connection, records a log according to message reception, and transmits the input text. During the communication connection connected via the input connection to the communication application that generated the event, characterized in that for recording the log according to the message reception and message transmission.

According to an embodiment of the present invention, a recording medium outputs a pop-up window for notifying a message reception when a message reception is input, and voice-transforms the message contents when a voice listening menu is selected through the pop-up window. When the voice reply menu is selected, the communication application corresponding to the type of the received message is executed, and the program for inputting and transmitting the text converted from the voice input from the user into the message content input window of the communication application is recorded. Computer-readable recording media.

The foregoing is a somewhat broad summary of features and technical advantages of the present invention in order that those skilled in the art will be better able to understand it from the following detailed description of the present invention. Additional features and advantages of the present invention, which form the subject matter of the claims of the invention, in addition to those features and advantages, will be better understood from the following detailed description of the invention.

As described above, according to the message service method and apparatus using the voice proposed by the present invention, by converting a text message received through the IME into a voice and converts the input voice data into a text message, the user is driving In addition, there is an effect that can easily check or reply to the message even in the environment difficult to operate the user device, such as moving.

According to the present invention, when a message is received, the IME processes the TTS and STT functions via a speech engine. Therefore, according to the present invention, the message transmission and reception history is recorded corresponding to each message type while easily performing confirmation by voice conversion of text and writing a reply message by text conversion of voice, regardless of the message type. And manageable effects.

According to the present invention, when using a message service most frequently used in a user device, even when the user device is not easy to use, the user can easily and safely check and reply to a message through a voice input through an IME. Therefore, according to the present invention, by implementing an optimal environment for supporting a message service, it is possible to contribute to improving the convenience of the user and improving the usability, convenience and competitiveness of the user device. The present invention can be easily implemented in all types of user devices and various devices corresponding thereto.

1 is a diagram schematically illustrating a configuration of a user device according to an exemplary embodiment of the present invention.
2 is a diagram schematically illustrating a platform structure for processing a message using voice according to an embodiment of the present invention.
3 is a flowchart illustrating a process in which a user device processes a message using voice according to an embodiment of the present invention.
4 to 7 illustrate examples of an operation screen of a user device according to an exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that, in the drawings, the same components are denoted by the same reference symbols as possible. Further, the detailed description of well-known functions and constructions that may obscure the gist of the present invention will be omitted. In other words, it should be noted that only parts necessary for understanding the operation according to the embodiment of the present invention will be described, and descriptions of other parts will be omitted so as not to disturb the gist of the present invention.

The present invention proposes a method and apparatus for providing a message service using a voice that can convert a content of a received message in a user device and output the voice message to a user or write a reply message for the received message by a user's voice input. According to an embodiment of the present invention, when a message is received in a situation in which the user device is difficult to operate, such as while driving or moving, an application according to a message type, an input method editor (IME), and a voice engine By interworking with each other to support the text to speech (TTS, Text to Speech) or speech to text (STT, Speech to Text) function, it is possible to support the user can easily check and reply to the received message.

According to an embodiment of the present invention, a message received in various forms in an IME may be processed. In addition, according to an embodiment of the present invention, the input from the user in conjunction with the voice engine in the IME may be automatically input into the message content input window of the corresponding communication application that generated the event. This can be implemented by an IME-specific feature that can have an input connection.

In an embodiment of the present invention, the message includes various types of messages, such as a text message (SMS message), a multimedia message (MMS message), an instant message, an e-mail message, and the like. For convenience, it is generically defined as a message. In addition, in the embodiment of the present invention, the communication application indicates an application supporting message transmission and reception with another user, and a text / multimedia message application, a messenger (or chat) application, a mail application, and a social network service (SNS) application. Etc., hereinafter, collectively referred to as a communication application for convenience of description.

Hereinafter, a configuration of a user device and an operation control method according to an embodiment of the present invention will be described with reference to the following drawings. It is to be noted that the configuration of the user device and the operation control method thereof according to an embodiment of the present invention are not limited or limited to the contents described below, and thus may be applied to various embodiments based on the following embodiments. .

1 is a diagram schematically illustrating a configuration of a user device according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the user device of the present invention includes a wireless communication unit 110, a user input unit 120, a display unit 130, an audio processor 140, a storage unit 150, an interface unit 160, and a controller ( controller), and a power supply unit 180. The user device of the present invention is not necessarily the components shown in FIG. 1, but may be implemented with more or fewer components.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the mobile terminal and the wireless communication system or between a user device and a network in which the user device is located. For example, the wireless communication unit 110 may include a mobile communication module 111, a wireless local area network (WLAN) module 113, a short range communication module 115, a position calculation module 117, (119), and the like.

The mobile communication module 111 transmits and receives a wireless signal with at least one of a base station, an external device, and a server on a mobile communication network. The wireless signal may include a voice call signal, a video call signal or various types of data according to transmission / reception of text / multimedia message, various types of data according to instant message transmission and reception, and feed information.

The wireless LAN module 113 represents a module for establishing a wireless Internet link and a wireless LAN link with another user device, and may be embedded or external to the user device. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies. The WLAN module 113 may transmit or receive a message defined in the present invention through the wireless Internet.

The short-range communication module 115 represents a module for short-range communication. Short range communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). This can be used.

The position calculation module 115 is a module for obtaining a position of a user device, and a representative example thereof is a GPS (Global Position System) module. The position calculation module 115 calculates distance information and accurate time information from three or more base stations and then calculates the current position information of three dimensions according to latitude, longitude, and altitude by applying trigonometry to the calculated information can do. Alternatively, the location calculation module 115 may calculate location information by continuously receiving a current location of the user device in real time from three or more satellites. The location information of the user device can be obtained by various methods.

The broadcast receiving module 119 receives a broadcast signal (e.g., a TV broadcast signal, a radio broadcast signal, a data broadcast signal, etc.) from an external broadcast management server through a broadcast channel (e.g., a satellite channel or a terrestrial channel) (E.g., information related to a broadcast channel, a broadcast program, or a broadcast service provider).

The user input unit 120 generates input data for a user to control operation of the user device. The user input unit 120 may include a keypad, a dome switch, a touch pad (static pressure / capacitance), a jog wheel, a jog switch, and the like. .

The display unit 130 displays (outputs) information processed in the user device. For example, when the user device is in a call mode, a user interface (UI) or a graphic user interface (GUI) related to a call is displayed. In addition, when the user device is in a video call mode or a photographing mode, the display unit 130 displays a photographed and / or received image, a UI, or a GUI. The display unit 130 displays a UI or GUI related to messages collected internally or externally. In particular, the display unit 130 may display a message transmitted and received through the communication application on the execution screen of the communication application. A screen configuration of the display unit 130 and an operation example thereof used in the present invention will be described later.

The display unit 130 includes a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), a light emitting diode (LED), an organic light emitting diode (Organic LED, OLED) ), At least one of an active matrix OLED (AMOLED), a flexible display, a bent display, and a 3D display. Some of these displays may be implemented as a transparent display configured to be transparent or light transmissive so that the outside can see.

According to the present invention, when the display unit 130 and the touch panel sensing a touch operation form a mutual layer structure (hereinafter, referred to as a touch screen), the display unit 130 may be used in addition to the output device. Can also be used as an input device. The touch panel may be configured to convert a change in a pressure applied to a specific portion of the display unit 130 or a capacitance generated in a specific portion of the display unit 130 into an electrical input signal. The touch panel can be configured to detect not only the position and area to be touched but also the pressure at the time of touch. When there is a touch input to the touch panel, the corresponding signal (s) is sent to a touch controller (not shown). The touch controller (not shown) processes the signal (s) and then transmits the corresponding data to the controller 170. As a result, the controller 170 can determine which area of the display unit 130 has been touched.

The audio processing unit 140 transmits the audio signal received from the control unit 170 to the speaker 141 and transmits the audio signal such as voice received from the microphone 143 to the control unit 170. The audio processing unit 140 converts the audio / sound data into audible sound through the speaker 141 under the control of the control unit 170, and converts the audio signal, such as voice, received from the microphone 143 into a digital signal, (170).

The speaker 141 may output audio data received from the wireless communication unit 110 or stored in the storage unit 150 in a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, and the like. The speaker 141 may output a sound signal related to a function (eg, a call signal reception sound, a message reception sound, music content playback, etc.) performed by a user device.

The microphone 143 receives an external sound signal in a call mode, a recording mode, a voice recognition mode, etc., and processes the external sound signal into electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 111 when the voice data is in the call mode, and output. The microphone 143 may be implemented with various noise reduction algorithms for eliminating noise generated in the process of receiving an external sound signal.

The storage unit 150 may store a program for processing and controlling the controller 170, and input / output data (for example, a phone book, a message, audio, a still image, an e-book, a video, feed information, It can also perform a function for temporary storage of log information). The storage 150 may also store the frequency of use of each of the data (eg, each telephone number, each message, frequency of use of each multimedia, etc.) and importance. In addition, the storage unit 150 may store data regarding vibration and sound of various patterns output when a touch input on the touch screen is performed. In particular, the storage unit 150 may map and store a message transmitted or received through the communication application and log information according to the transmission or reception of the message to the executed communication application. In addition, the storage unit 170 may store the platform of FIG. 2 to be described later.

The storage unit 150 may include a flash memory type, a hard disk type, a micro type, and a card type memory (eg, SD or XD memory) and RAM. , Random Access Memory (SRAM), Static RAM (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable ROM (EEPROM), Programmable ROM (PROM), Magnetic Memory (MRAM, Magnetic RAM), Magnetic Disk (magnetic disk) And a storage medium of at least one type of optical disk type memory. The portable terminal may operate in association with a web storage that performs a storage function of the storage unit 150 on the Internet.

The interface unit 160 serves as a pathway to all external devices connected to the user device. The interface unit 160 receives data from an external device, receives power from the external device, transfers the data to each component in the user device, or allows data in the user device to be transmitted to the external device. For example, wired / wireless headset ports, external charger ports, wired / wireless data ports, memory card ports, ports for connecting devices with identification modules, audio input / output (I / O) ports, The video I / O port, the earphone port, and the like may be included in the interface unit 160.

The control unit 170 controls the overall operation of the user device. For example, for voice calls, data communications, video calls, and the like. The controller 170 may include a multimedia module (not shown) for multimedia playback. The multimedia module may be implemented in the controller 170 or may be implemented separately from the controller 170. In particular, the controller 170 receives a message of the present invention, receives a menu selection for voice listening or a voice reply for the received message, and converts the text of the received message into a speaker 141 when the voice listening menu is selected. When the voice reply menu is selected, the communication application may be executed according to the type of the received message, and the voice of the user may be converted into text and input into the message input window of the communication application. That is, the controller 170 is responsible for the overall control related to the message input and output function using the voice of the present invention. The detailed control operation of the controller 170 will be described in the operation example of the user device and the control method thereof with reference to the drawings described later.

The power supply unit 180 receives an external power source and an internal power source under the control of the controller 170 to supply power for operation of each component.

Meanwhile, the various embodiments described in the present invention can be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. According to a hardware implementation, the embodiments described in the present invention may include Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs). ), Processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions. In some cases, the embodiments described herein may be implemented by the controller 170 itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein.

The user device of the present invention shown in FIG. 1 is the same as any information communication device, a multimedia device, and an application device supporting the function of the present invention, an AP (Application Processor), a GPU (Graphic Processing Unit), and a CPU (Central Processing). It can include any device that uses a unit. For example, the user device may include a tablet PC, a smart phone, a mobile communication terminal operating based on communication protocols corresponding to various communication systems, and the like. It may include a device such as a digital camera, a portable multimedia player (PMP), a media player, a portable game terminal, a laptop computer, and a personal digital assistant (PDA). In addition, the function control method of the present invention can be applied to various display devices such as a digital TV, a digital signage (DS), and a large format display (LFD).

2 is a diagram schematically illustrating a platform structure for processing a message using voice according to an embodiment of the present invention.

Referring to FIG. 2, a platform of a user device according to an exemplary embodiment of the present disclosure may include software based on an operating system (OS) to perform various operations related to message input and output using the above-described voice. . As shown in FIG. 7, a user device according to an embodiment of the present invention is designed to include a framework 210, an IME 220, a speech engine 230, and an application 240. Can be.

The framework 210 may include a program that is a base of an application in the application 240 layer. The framework 210 may be compatible with any application, and may reuse, move, or exchange components. The framework 210 may include a support program, a program that connects other software components, and the like. For example, a resource manager, an activity manager related to an application's activity, a notification manager, a content provider that summarizes shared information between applications, and a message manager. (message manager) and the like. In the present invention, for example, the framework 210 is a framework for text / multimedia message management, a framework for messenger management, a framework for mail management, and the like.

The IME 220 represents a module for controlling a text input portion when a user inputs a voice or a character. That is, the IME 220 controls the input of text corresponding to the user's input of text using a user device or a voice through a voice interface (eg, the microphone 143, etc.). The IME 220 may be allocated and executed corresponding to each application in the application 240 layer.

The voice engine 230 is an engine for converting an input signal into text or voice, and includes a TTS engine for converting text of a message into voice, and a program including an STT engine for converting voice into text of a message. Can be. In the present invention, the voice engine 230 may interwork with the IME 220 and output the text of the received message by voice conversion, or may convert the voice input for the reply message to text.

The application 240 layer includes various programs that can be driven and displayed in the user device. For example, the application may include UI applications related to various menus in a user device, an application downloaded and stored through an external device or a network, and installed or deleted by a user. Through the applications in the application 240 layer, Internet phone service, video on demand (VOD) service, web album service, SNS, location based service (LBS), map service, web search service, application search service, Text / multimedia message service, mail service, etc. may be performed. In addition, various functions such as game and schedule management can be performed. In the present invention, an example of the application 240 layer is a communication application such as a text / multimedia message application, a messenger application, a mail application, and the like.

In addition, the platform of the present invention may further include an OS kernel (not shown) and middleware (not shown). The OS kernel is the core of the OS, which drives a hardware driver when the user device is driven, the security of the hardware and the processor in the user device, the efficient management of system resources, the memory management, the interface to the hardware by the hardware abstraction, and the multi At least one of a process and schedule management according to a multi-process may be performed. The hardware driver in the OS kernel (not shown) may include at least one of a display driver, a WiFi driver, a Bluetooth driver, a USB driver, an audio driver, a power manager, a binder driver, a memory driver, and the like. . The middleware (not shown) may be located between the OS kernel and the application 240, and may serve as an intermediary to exchange data between different hardware or software. As a result, a standardized interface can be provided, and various environment support and systems can be interconnected with other tasks.

Meanwhile, in an embodiment of the present invention, when a message such as a text / multimedia message, an instant message, or a mail message is received, the IME 220 easily processes the confirmation and reply to the message received through the TTS or STT, which is a voice recognition function. Do it.

For example, in the platform of the present invention, when the IME 220 delivers a message received through the wireless communication unit 110 through the framework 210, caller information (eg, a phone number, a calling user ID, a chat chat room). , Mail addresses, etc.) and the contents of the message can be parsed and obtained. When the new message is received, the IME 220 may feed back to the user through the pop-up window 400 as shown in FIG. 4 to be described later. That is, the IME 220 may receive an incoming event of the message through the wireless communication unit 110. In the present invention, the reason for processing the message according to the received event using the IME 220 is as follows. Can be defined.

In particular, due to the characteristics of the message, as the message is sent and received with several other users, a chat history in the form of conversations according to the transmission and reception of messages should be maintained, and various messages (text / multimedia message, message message, mail message, etc.) It should be able to support, because the following features of the IME 220 can support the above-described function. That is, the IME 220 is capable of connecting an input connection (IC, Input connection), it is characterized in that the text can be input to the text input window, such as text / multimedia messages, messenger applications, mail applications, IME 220 Explicit operation and invocation procedures in the form of daemons that are always running on the user device in the form of services, that is, system programs that reside on the user device and automatically provide various services when the application or system is in a specific state. It is characterized by unnecessary.

In detail, first, in order for characters input from a keypad of a user device to be input into an input window of an application, a connection called an input connection (IC) must be generated between the keypad and the input window of the application. That is, the input connection is an area unique to the IME 220 and may input text (characters) to the input window of an application connected to the input connection (IC).

In order to know that a message such as a text / multimedia message, a messenger message, or a mail message is incoming, the process must be a service type that always operates in the form of a daemon. Therefore, rather than creating and maintaining a new daemon service, it may be more efficient to use an application that always resides and runs in daemon form, such as the IME 220 implemented in the user device.

In addition, in the present invention, the reason for inputting text into an input window of a communication application such as a text / multimedia message application, a message application, and a mail application is obvious. I can tell. For example, a text message or the like in a user device has a chat screen as a concept, and a conversation with a specific conversation partner leaves a log in a conversation form. Therefore, if the reply message is not transmitted through the input window of the corresponding application connected through the input connection (IC) of the IME 220, there is a problem in that no trace of the reply can be left in the conversation history. This can be solved through message management.

And since the keypad of each communication application can be input without an input connection (IC), it is possible to manage the message history according to the transmission and reception of messages. However, in order to implement TTS and STT for voice listening and voice reply to the received message as in the present invention, the corresponding communication application must have both a configuration for implementing TTS and STT, or each communication application has a message related to self. There is a restriction that can only handle the contents of the type. Therefore, if this is handled by the IME 220 itself, it is possible to process various types of messages received from various messages (text / multimedia message, message message, mail message, etc.) supported by the user device with one module. .

Hereinafter, an example of performing an interaction between each module in the platform of the present invention as shown in FIG. 2 will be described.

First, referring to FIG. 2, when a WAP (Wireless Application Protocol) push is used, a push message is transmitted from a push application server (or a web server) of a WAP gateway. A case where it is generated and transmitted to the user device will be exemplified. Here, the push application server (or web server) may include a server, such as news, game scores, weather, the customer service system (CSBS, Customer Service Billing System) of the mobile carrier.

The push application server (or web server) may generate a message in a protocol data unit (PDU) format and transmit it to the user device. Then, the message is received from the user device push application server (or web server) through the wireless communication unit 110, and the received message is a framework corresponding to the message type of the framework 210 (particularly, the message framework). It is delivered to the application 240 layer through. For example, when a text message is received, it may be delivered to the text message application through a framework related to the text message.

And according to an embodiment of the present invention, the received message is also delivered to the IME 220 through the framework 21. That is, the IME 220 is pushed in the received message. At this time, the pop-up window 400 for the received message is output through the display unit 130.

Then, the user may select one of the menu items "Listen to voice" or "Reply to voice" through the pop-up window 400 to execute the corresponding function. At this time, the IME 220 receives the received message through TTS speech conversion based on "voice listening" or STT text conversion based on "voice reply" through the speech engine 230 connected to the IME 220 in response to the user's menu selection. You will write an acknowledgment message or reply message for an incoming message. Here, the text input through the STT from the user via the voice engine 230 connected to the IME 220 may be delivered to the corresponding application (eg, a text message application) and delivered to the final counterpart.

On the other hand, the platform as described above can be used in a variety of other electronic devices, as well as the user device of the present invention. In addition, the platform of the present invention may be stored or loaded in at least one or a separate processor (not shown) of the storage unit 150 and the controller 170 as described above. In addition, a separate application processor (not shown) for executing an application may be further provided.

3 is a flowchart illustrating a process in which a user device processes a message using voice according to an embodiment of the present invention.

Referring to FIG. 3, when a message reception is input (step 301), the controller 170 may control a pop-up window output informing of the message reception (step 303). In this case, the controller 170 may estimate a message type and manage a message framework and a communication application corresponding to the message type. In the present invention, the message includes a text message, a multimedia message, an instant message, and a mail message. The framework includes a text message framework, a multimedia message framework, an instant message framework, and a mail framework. The communication application may include a text / multimedia message application, a messenger (or chat) application, and a mail application.

The controller 170 may check whether there is a menu selection through a pop-up window (step 307). That is, the controller 170 may wait to receive an input for selecting any one of a menu of <voice listening> and <voice reply> provided through the pop-up window.

If there is no menu selection (NO in step 307), the controller 170 may control to perform a corresponding operation (step 305). For example, the controller 170 may count the time according to the output of the pop-up window, and if there is no menu selection for a preset time, the controller 170 may control the display 130 to be turned off. In addition, the controller 170 may control an output of an alarm (eg, at least one of a vibration, a sound, and a screen) for a received message in response to a cycle and a number set when the display 130 is turned off. Alternatively, the controller 170 may execute a communication application corresponding to a message according to a user input, and control the display of contents of a received message through the executed communication application.

If there is a menu selection (YES in step 307), the controller 170 may determine the type of the selected menu (step 309). That is, the controller 170 may determine whether the <voice listening> menu is selected or the <voice reply> menu.

If the controller 170 determines that the selected menu is a <voice listening> menu (NO in step 309), the controller 170 may parse the message (step 311). For example, the controller 170 may extract the content (eg, text) of the received message when the <voice listening> menu is selected. In this case, the controller 170 may extract sender information (eg, a phone number, a calling user ID, a chat chat room, a mail address, etc.) of the message.

The controller 170 may convert the parsed content through the speech engine 230 and then control the output of the converted speech through the speaker 141 (step 313). At this time, the controller 170 transmits the parsed information to the IME 220 through the framework 210, and the IME 220 operates the speech engine 230 to interwork so that the voice conversion of text and its output are processed. Can be.

When the converted voice is output, the controller 170 may record a log according to the reception of the message (step 315). In particular, when the converted voice is output through the speaker 141 under the control of the controller 170, the IME 220 may map the communication application corresponding to the message type to record a log according to the reception of the message. In this case, the controller 170 may maintain the pop-up window display state.

The controller 170 may check whether there is a selection of the <voice reply> menu after voice output and log recording of the received message (step 317). In addition, when an input for selecting a <voice reply> menu is received (YES in step 317), the controller 170 may proceed to step 321 to control the following operation. In addition, when the input for selecting the <voice reply> menu is not received (NO in step 317), the controller 170 may control to perform a corresponding operation (step 319). For example, the controller 170 may count the time according to the output of the pop-up window, and turn off the display 130 if there is no menu selection for a preset time.

If the controller 170 determines that the selected menu is a <voice reply> menu (YES in step 309), the voice is input through the voice engine 230 (step 321), and parsing of the input voice (step 323). The text may be generated (step 325). That is, when the user's voice is input through the microphone 143, the controller 170 may convert the input voice through the voice engine 230. In this case, the IME 220 may operate the speech engine 230 to be linked to process the text conversion and generation thereof.

The controller 170 may check the message type (step 327) and call a communication application corresponding to the message type (step 329). For example, the controller 170 calls a text message application when the message type is a text message, calls a messenger (or chat) application when the message type is an instant message, and mails when the message type is a mail. Invoke the application. In particular, the controller 17 may call a communication application corresponding to a message type through an IME connected through the input connection with the communication application. In this case, the controller 170 may check the message type estimated in the message receiving step and push the execution of the identified communication application through the framework 210 corresponding to the message type.

The controller 170 may create a reply message using the converted text through the called communication application (step 331). For example, the controller 170 may input the converted text into the message content input window of the message application, and input the extracted caller information into the receiver input window to compose a message. Alternatively, the controller 170 may input the converted text into a message content input window of a messenger application and display the chat chat room screen extracted earlier. In this case, the IME 220 may transfer the converted text to the communication application to be executed through the voice engine 230 and input the converted text into the message content input window of the communication application.

The controller 170 may transmit the written reply message in step 334. For example, the controller 170 may receive a user's message transmission button input and transmit the written reply message to the corresponding destination in response to the transmission button input. Alternatively, the controller 170 may receive a user's voice command input and transmit the written reply message to the corresponding destination in response to the voice command.

When the reply message is transmitted, the controller 170 may record a log according to message transmission and reception (step 335). In particular, when the reply message is transmitted through the wireless communication unit 110 under the control of the controller 170, the IME 220 may map to the executed communication application and record a log according to the reception and transmission of the message.

4 to 7 illustrate examples of an operation screen of a user device according to an exemplary embodiment of the present invention.

4 to 7, FIG. 4 illustrates an example of a screen of a user device when a mesh is received from an external specific message server. That is, the message may be received from the message server, and the information on the received message may be displayed through the pop-up window 400. The pop-up window 400 displays pop-up information (eg, message reception notification information, message sender information, message contents, etc.) in the message information area 410 as shown in FIG. 4, and performs a voice listening or voice reply function. A listening menu button 430 and a voice reply menu button 450 are provided.

In addition, the user may select one of the menu buttons 430 and 450 in the pop-up window 400. For example, the user may select a button for performing any one of “listen to voice” and “voice reply” in the popup window 400. When the user selects the voice listening menu button 430, the text of the message may be voice converted and the converted voice may be output through the speaker 141 according to the above procedure. In this case, the screen as shown in FIG. 4 may be maintained when the converted voice is output. On the other hand, when the user selects the voice reply menu button 450, the screen of the user device may be switched as shown in FIG. 5.

5 illustrates an example of a screen of a user device when the "voice reply menu button 450 is selected for creating a reply message through the voice reply function in the pop-up window 400. That is, FIG. An example of a screen of a user device when the menu button 450 is selected, the pop-up window 400 may be converted to a guide window 500 for guiding a voice input, and the guide window 500 may be displayed. In addition, the guide voice for guiding the voice input may be output together through the speaker 141.

The user may input the contents of a message to be written according to the guide through the guide window 500 using voice. For example, the user may enter a voice of "Please speak." The user device displays a communication application screen as shown in FIG. 6 according to a voice input of the user. That is, the screen of the user device shown in FIG. 5 is switched as shown in FIG. 6 when the user's voice input is completed.

6 illustrates an example of a screen of a user device when a user inputs a voice according to a guide of the guide window 500. As shown in FIG. 6, in the embodiment of FIG. 6, it is assumed that the communication application is a messenger application. That is, the type of the received message is a message associated with a messenger application, and accordingly, the messenger application may be automatically executed as a communication application for creating a reply message. Therefore, the text converted by the user's input voice "Please speak" may be automatically displayed in the message content input window 600 of the messenger application.

When the text corresponding to the user's voice input is input to the message content input window 600, the user may transmit the input message content (text). For example, a user may input a button for transmitting the content of a written message (eg, a send button next to the message content input window 600), or transmit a written message content using a voice command (eg, a transfer). Can be. Then, the reply message may be transmitted to the counterpart user device by the user's transmission input.

FIG. 7 illustrates an example of a screen of the user device when the user device transmits a text-converted reply message to the counterpart user device through a messenger application in response to the user's reply message transmission. That is, FIG. 7 illustrates an example of a screen of the user device in which the content of the reply message is input to the messenger chat screen when the user device transmits a reply message through the messenger application.

As described above with reference to FIGS. 4 to 7, according to an embodiment of the present invention, when a message is received, a "voice listening" and a "voice reply" function are provided through a pop-up window 400 to allow a user to make a TTS. It supports to check incoming message through function or to write reply message through STT.

That is, when the user selects "voice listening" in the pop-up window 400 for the received message, the IME 220 interworks with the voice engine 230 and displays the text of the message received through the voice engine 230. Converts TTS voice and reads voice corresponding to text. As a result, the user may replace the inconvenience of visually checking the received message even in a situation in which the manipulation of the user device is difficult, such as while driving or moving.

In addition, when the user selects "voice reply" in the pop-up window 400 for the received message, the IME 220 is interworked with the voice engine 230, and the user's voice input through the voice engine 230 is STT text. The content of the reply message is generated and inserted into the text corresponding to the voice. That is, the caller may automatically track and execute a communication application that transmits a message to automatically input and provide text corresponding to the user's voice. This makes it possible for the user to simply send a reply message to the receiver without manipulating the user device.

The embodiments of the present invention as described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, and the like, alone or in combination. The program instructions recorded on the recording medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in computer software.

The computer-readable recording medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc) A magneto-optical medium such as a floppy disk, and a program command such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, Lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; The program instructions also include high-level language code that can be executed by a computer using an interpreter or the like, as well as machine code as produced by the compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. Accordingly, the scope of the present invention should be construed as being included in the scope of the present invention, all changes or modifications derived from the technical idea of the present invention.

110: wireless communication unit 120: user input unit
130: Display section 140: Audio processing section
150: storage unit 160: interface unit
170: control unit 180: power supply unit
210: framework 220: IME
230: speech engine 240: application
400: pop-up window 500: guide window
600: message content input window

Claims (20)

In the message service support method of the user device,
Outputting a pop-up window notifying the reception of the message when the reception of the message is input;
Receiving a menu selection of a voice listening and voice reply menu through a pop-up window;
If the selected menu is a voice listening menu, converting a message content and outputting a voice;
Converting a voice input from a user into text if the selected menu is a voice reply menu;
Calling a communication application corresponding to the type of the received message;
Inputting the converted text into a message content input window of the communication application; And
And transmitting the input text in response to the user's message transmission. The method of claim 1, wherein the outputting process
And recording a log according to the reception of the message when outputting the converted voice. 3. The method of claim 2,
An input method editor (IME) connected to the communication application through an input connection maps to a communication application corresponding to the type of the received message and records a log according to the reception of the message. How message services are supported. 3. The method of claim 2,
And maintaining the display state of the pop-up window when the converted voice is output. The method of claim 4, wherein the outputting process
And receiving a voice reply menu selection by the pop-up window after outputting the voice and recording the log. The method of claim 1, wherein the converting of the text to text is performed.
And when the voice reply menu is selected, an IME activates a voice engine to convert a voice input from the user into text, and transmit the converted text to the communication application. How message services are supported. The method of claim 1, wherein the inputting of the converted text is performed.
The message service supporting method of the user device, characterized in that the process of generating a reply message using the converted text through the communication application. The method of claim 7, wherein the transmitting process
And transmitting the reply message in response to a user's message transmission button input or a user voice voice command input. 9. The method of claim 8,
And recording a log according to the message reception and the message transmission when the reply message is transmitted. 10. The method of claim 9,
And an IME connected through the input connection with the communication application to the communication application to record a log according to the message reception and message transmission. A storage unit in which at least one program is stored; And
And a controller configured to execute the at least one program to control a message service using voice.
Wherein the at least one program comprises:
Outputting a pop-up window notifying the reception of the message when the reception of the message is input;
Receiving a menu selection of a voice listening and voice reply menu through a pop-up window;
If the selected menu is a voice listening menu, converting a message content and outputting a voice;
Converting a voice input from a user into text if the selected menu is a voice reply menu;
Calling a communication application corresponding to the type of the received message;
Inputting the converted text into a message content input window of the communication application; And
And instructions for executing a process of transmitting the input text in response to a user's message transmission. 12. The apparatus of claim 11, wherein the control unit
And controlling the output of the received message by voice conversion and controlling the creation of a reply message corresponding to the received message through input by text conversion of the input voice. The method of claim 12, wherein the storage unit
And an input method editor (IME) for controlling a text input portion when a user inputs a voice or a character. The method of claim 13, wherein the IME
Detect an event where the message is received,
When the voice listening menu is selected through the pop-up window, the voice engine is operated to convert the contents of the message into voice and output the voice.
And when the voice reply menu is selected through the pop-up window, activates a voice engine, converts a voice input from the user into a text, and transmits the voice to the communication application. The method of claim 14, wherein the IME
And converting the converted text into a message content input window of the communication application in association with the corresponding communication application that generated the event. 16. The method of claim 15,
And the IME is connected to a communication application through an input connection. The method of claim 16, wherein the IME
And outputting the converted voice to a communication application corresponding to the type of the received message among the communication applications connected through the input connection, and recording a log according to the reception of the message. The method of claim 16, wherein the IME
And a log corresponding to the message reception and the message transmission by mapping to the communication application that generated the event among the communication applications connected through the input connection when the input text is transmitted. The method of claim 14, wherein the IME
And run in the user device and reside in a daemon form. If a message reception is input, a pop-up window for notifying the reception of the message is output. If the voice listening menu is selected through the pop-up window, the message contents are converted to voice and output. If the voice reply menu is selected, the communication corresponding to the received message type is output. A computer-readable recording medium having recorded thereon a program that executes an application and inputs text converted from a voice input by a user into a message content input window of the communication application.

Images