KR101469286B1 - Method for multimodal messaging service - Google Patents

Abstract

The present invention relates to a multimodal messaging service method, in which multimodal information and a message type to be transmitted to a called terminal are checked and analyzed in a message transmitted from a calling terminal, a symbol is inquired, a symbol is linked to multimodal information, To provide a multimodal messaging service method for transmitting a message generated according to a message type to be transmitted to a destination terminal to a destination terminal and then responding to a request for confirmation of multimodal information linked to the symbol by the destination terminal do.

To this end, the present invention provides a multi-modal messaging service method, comprising: checking multi-modal information and a 'second message type to be transmitted to a destination terminal' in a first message transmitted from a calling terminal; And linking the inquired symbol to the multimodal information and transmitting a second message generated according to the 'second message type to be transmitted to the called terminal' to the called terminal.

Multi-modal messaging service, symbol, multi-modal information, symbol message type, raw message, one-sided, link

Description

[0001] METHOD FOR MULTIMODAL MESSAGING SERVICE [0002]

The present invention relates to a multimodal messaging service method, and more particularly, to a multimodal messaging service method in which multimodal information and a message type to be transmitted to a called terminal are checked and analyzed in a message transmitted from a calling terminal, Modal messaging service to link a message to a destination terminal and to send a message generated according to the message type to be transmitted to the destination terminal to the destination terminal and to respond to a request for confirmation of the multimodal information linked to the symbol by the destination terminal ≪ / RTI >

The conventional messaging service is a character-based messaging method for creating a message using a keypad of a terminal, such as a Short Message Service (SMS) or a Multimedia Message Service (MMS) As the usage of mobile phones grows, especially among teenagers who are fast and familiar, they are becoming important revenue sources in the mobile communication business. In addition, the conventional messaging service is excellent in the ability to utilize the keypad of the terminal, and if the message can be quickly generated, the message can be quickly exchanged with the other party.

However, since the conventional messaging service requires a keypad operation of the terminal, there is a limitation that the keypad operation of the terminal is a communication means that can alienate old aged people, children, and some people with disabilities. For this, conventionally, a messaging method using voice recognition as an alternative means of inputting characters has been proposed, but it has been difficult to transmit a clear message to the other party when the user is in a noisy environment because the voice recognition rate is lowered by using only voice as an alternative means. That is, when the user is in a noisy environment, the user relinquishes an attempt to transmit a message created through speech recognition to the other party, and finally transmits his / her message through a voice call with the other party.

Therefore, a conventional messaging service needs a messaging method that allows a user to transmit a message conveniently in any desired manner to anyone, anytime, anywhere, regardless of whether they are male or female through various input / output methods. That is, the conventional messaging service has to evolve in a direction that can absorb a hierarchy that has been left out of the conventional method and provide a messaging service through an interface using various input / output, because the keypad operation of the terminal is difficult.

Accordingly, there is a problem that the conventional technology as described above has a problem that the keypad operation of the terminal alienates those who are uncomfortable, and it is an object of the present invention to solve such a problem.

Therefore, according to the present invention, the multi-modal information and the message type to be transmitted to the destination terminal are analyzed and analyzed in the message transmitted from the originating terminal, the symbol is inquired, the symbol is linked to the multi-modal information, Modal messaging service method for transmitting a message generated according to a multi-modal messaging service to a destination terminal so that a destination terminal can respond to a request for confirmation of multi-modal information linked to a symbol.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be readily apparent that the objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

According to an aspect of the present invention, there is provided a multimodal messaging service method comprising: checking multimodal information and a 'second message type to be transmitted to a called terminal' in a first message transmitted from a calling terminal, ; And linking the inquired symbol to the multimodal information and transmitting a second message generated according to the 'second message type to be transmitted to the called terminal' to the called terminal.

Further, the present invention further includes the step of, after the transmitting step, responding to the confirmation request of the multimodal information linked to the inquired symbol by the destination terminal.

Also, according to the present invention, there is provided a multimodal messaging service method comprising: checking multimodal information in a first message transmitted from a calling terminal to determine whether a multimodal type set by a caller and a 'second message type to be transmitted to a called terminal' ; And a transmission step of processing and transmitting the first message according to the determination result.

In addition, the present invention provides a multimodal messaging service method comprising the steps of: checking a multimodal type and a message type in a 'message transmitted to a called terminal' and judging whether the message corresponds to a multimodal type and a message type preset by the called party; ; And a transmission step of processing and transmitting the 'message transmitted to the called terminal' according to the determination result.

The present invention has the effect of enabling a user to conveniently transmit a message to any person, at any time, anywhere, regardless of sex, age, and sex through various input / output methods.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, It can be easily carried out. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a multimodal messaging service network according to an embodiment of the present invention.

1, the multimodal messaging service network according to the present invention includes an originating terminal 110, a multimodal messaging service apparatus 120, and a terminating terminal 130. As shown in FIG. Hereinafter, a message including the multimodal information and the text content and transmitted from the originating terminal 110 to the multimodal messaging service apparatus 120 will be referred to as a "raw message ", a " symbol corresponding to the multimodal information " And a message transmitted from the multimodal messaging service apparatus 120 to the destination terminal 130 will be described below as a "symbol message". In the present invention, primitive messages and symbol messages are a kind of multi-modal messages. That is, the multi-modal messaging service apparatus 120 analyzes the original message transmitted from the originating terminal 110, generates a symbol message, and transmits the symbol message to the terminating terminal 130. The terminating terminal 130 may retrieve the source corresponding to the symbol from the multimodal messaging service apparatus 120 using the symbol included in the symbol message.

The originating terminal 110 and the terminating terminal 130 are terminals having an input / output function of the same multimodal information (i.e., a camera for inputting an image, a microphone for inputting sound, a touch pad for inputting text, etc.) , But does not perform the functions limited to the calling party and the called party, but will be described separately for the convenience of explanation.

In particular, the originating terminal 110 and the terminating terminal 130 should be equipped with a program and software (hereinafter referred to as "messaging application") for utilizing the multimodal messaging service of the present invention. That is, the originating terminal 110 generates and transmits a raw message after driving the message application, and when the receiving terminal 130 receives the symbol message, the messaging application is driven according to the user setting (i.e., Can be confirmed automatically or through the approval process).

As described above, the originating terminal 110 and the terminating terminal 130 operate based on the message application settings to utilize the multimodal messaging service. This is because a source message is not transmitted from the originating terminal 110 to the terminating terminal 130 due to the originating / incoming phone number of the original message, To the terminal (130).

Meanwhile, the caller generates an original message using the message application installed in the calling terminal 110. [ First, the sender selects a multimodal type (i.e., image, sound, text, etc.) and a symbol message type (i.e., link type, recognizable type).

Here, the multimodal type indicates which multimodal information should be included in the original message. The symbol message type indicates how to generate a symbol message after analyzing the original message in the multimodal messaging service apparatus 120. The link type is a symbol indicating a 'multimodal type' (that is, , And the recognition type is a type that links 'the symbol corresponding to the recognition result of the multimodal information' to the multimodal information.

In addition, the symbol message type is automatically set to a default type when either the link type or the recognizable type is set to default (defalut) and there is no selection by the user, Can be determined through selection by the user. Therefore, it is desirable to have a menu for selecting 'type not set to default' in the message application.

However, in order to facilitate a clear understanding of the selection of a symbol message type, the present invention will describe a process in which a multimodal messaging service is performed when the user is determined as either a link type or a recognition type 2a to 4b).

Thereafter, the sender enters the multimodal information and creates the text content. At this time, the originating terminal 110 generates a raw message by combining the multimodal information input by the sender and the contents of the text created by the sender.

In addition, when the multimodal type button is selected by the caller, the calling terminal 110 activates a function of receiving the multimodal information corresponding to the multimodal type, and when the multimodal type button is selected again by the calling party The corresponding function is terminated. That is, the originating terminal 110 enters multimodal information (where the image is captured once) until the multimodal type button is selected and reselected by the originator.

In addition, the called party confirms the symbol message using the message application installed in the receiving terminal 130. [ That is, the called party confirms the symbol and the contents of the text in the symbol message. Here, the symbols are either linked to the element for the multimodal information, or replaced with a specific word.

Hereinafter, the multimodal messaging service apparatus 120 will be described. The multimodal messaging service apparatus 120 includes a raw message input unit 121, a symbol message generation unit 122, a symbol message output unit 123 and a storage unit 124.

The original message input unit 121 transmits the original message transmitted from the originating terminal 110 to the symbol message generating unit 122.

The symbol message generator 122 extracts the 'multimodal information and the symbol message type' from the primitive message to grasp the multimodal information and the symbol message type, searches the corresponding symbol according to the multimodal information and the symbol message type, . In particular, the symbol message generation unit 122 interprets the corresponding source message if the symbol message type is a recognition type.

Specifically, the case where the multimodal type is 'image' will be described.

First, when the symbol message type is 'link type' (see FIG. 2A to be described later), the symbol message generation unit 122 generates a 'symbol representing an image' in the corresponding image file, which is multimodal information, To generate a symbol message.

In addition, when the symbol message type is 'recognition type' (see FIG. 2B to be described later), since the symbol message type is a recognition type, the symbol message generation unit 122 generates a symbol corresponding to the recognition result of the image file, Through the storage unit 124, and generates a symbol message by linking the 'retrieved symbol' through the storage unit 124 after recognizing the corresponding image in the image file. Here, the symbol message generating unit 122 uses a normal image recognition processing technique. For example, a pattern recognition algorithm that recognizes an image by detecting pixel information and a pattern of an image object in a state of collecting and analyzing images in advance Can be applied.

In addition, the image file may be an image captured through a camera function, or an image captured or photographed in advance.

Next, the case where the multimodal type is 'sound' will be described.

First, when the symbol message type is 'link type' (refer to FIG. 3A to be described later), since the symbol message type is a link type, the symbol message generation unit 122 generates a 'symbol representing a sound' To generate a symbol message.

When the multi-modal type is 'part of music' and the symbol message type is 'recognition type' (see FIG. 3B to be described later), since the symbol message type is the recognition type, Searches the 'music part' through the storage unit 124 using the extracted lyric information and generates a symbol message by linking 'symbol representing sound' to all parts of the music.

In addition, when the multi-modal type is 'voice' and the symbol message type is 'recognition type' (see FIG. 3C to be described later), the symbol message generation unit 122 generates a multi- A message is generated by converting the content into text using a speech recognition technology, and a symbol determined as the emotional state of the sender is retrieved through the storage unit 124 using the emotion recognition technology, And generates a symbol message by linking the symbol 'determined as the emotional state of the sender after recognizing the emotion of the voice.

Next, the case where the multimodal type is 'text' will be described. When the symbol message type is 'link type' (see FIG. 4A to be described later), since the symbol message type is a link type, the symbol message generation unit 122 generates 'symbol representing text' To generate a symbol message. In addition, when the symbol message type is 'recognition type' (see FIG. 4B to be described later), the symbol message generation unit 122 recognizes the text file, which is the multimodal information, A symbol is retrieved through the storage unit 124, and a symbol is generated by converting a specific word in the text into a 'symbol corresponding to a specific word'.

Since the symbol message also has a basic function of transmitting and receiving with a text message, if the text is not specifically linked, the text is transmitted as it is.

When the symbol message is generated through the symbol message generating unit 122, the symbol message output unit 123 transmits the symbol message to the terminating terminal 130 using the telephone number of the terminating terminal 130. At this time, when there is a request for confirmation of 'one-sided for multi-modal information linked to a symbol' by the receiver, the symbol message output unit 123 stores 'one-sided for multi-modal information linked to the symbol' And provides it to the receiving terminal 130. [

2A and 2B are views illustrating an embodiment of a multimodal messaging service to which the present invention is applied.

2A shows a case where the sender selects the 'image' for the multimodal type and the 'link type' for the symbol message type by driving the message application of the originating terminal 110, And creates a raw message by creating a body content (that is, it wants to eat), and the called party confirms the 'symbol representing image 201' and the body content 202 in the symbol message. At this time, if the recipient selects the 'symbol representing the image' 201, the recipient can confirm the 'source of multimodal information' (ie, apple photo) input by the sender.

2B shows a case where the sender selects the 'image' for the multimodal type and the 'recognition type' for the symbol message type by driving the message application of the originating terminal 110, (I.e., an apple-shaped symbol) through the storage unit 124 after recognizing the image file in the symbol message, and generates the original message (i.e., (203) and the text content (204). At this time, when the recipient recognizes the image file and selects the retrieved symbol '203' through the storage unit 124, the recipient can confirm 'one element of multimodal information' (that is, an apple photo inputted by the sender) .

FIG. 3A and FIG. 3D are views illustrating another embodiment of a multimodal messaging service to which the present invention is applied.

3A shows a case where the sender selects the 'sound (music or voice)' for the multimodal type and the 'link type' for the symbol message type by driving the message application of the originating terminal 110, (I.e., music or voice) and creates a textual content (i.e., listen) to generate a raw message, and the called party identifies the 'symbol representing the sound' 301 and the textual content 302 in the symbol message. At this time, if the called party selects the 'symbol representing the sound' (301), it can confirm 'the element of multimodal information (that is, the music or voice inputted by the sender)'.

3B shows a case where the sender selects the 'sound (part of music)' for the multimodal type and the 'recognition type' for the symbol message type by driving the message application of the originating terminal 110, , A piece of music) and creates a textual content (i.e., listen) to generate a raw message and the called party identifies the 'symbol representing the sound' 303 and the textual content 304 in the symbol message. At this time, if the called party selects the 'symbol representing the sound' (303), 'one element of the recognition result of the multimodal information (that is, the entire music part)' can be confirmed.

3C shows a case where the sender selects the 'sound' for the multimodal type and the 'recognizable type' for the symbol message type by driving the message application of the originating terminal 110, 305) and a textual content (306) indicating the result of emotion recognition of the multi-modal information are displayed in the symbol message. Check. At this time, when the recipient selects 'Symbol' 305 indicating the result of emotion recognition of the multi-modal information, 'the origin of multi-modal information' (that is, the voice inputted by the caller) can be confirmed.

4A and 4B illustrate another embodiment of a multimodal messaging service to which the present invention is applied.

4A shows a case where the sender selects the 'text' for the multimodal type and the 'link type' for the symbol message type by driving the message application of the originating terminal 110. The sender inputs the text, which is the multimodal information, (Read)) to generate a raw message, and the called party confirms the 'symbol representing the text' 401 and the textual content 402 in the symbol message. At this time, if the called party selects the 'symbol representing the text' 401, it can confirm 'the element of multimodal information (that is, the text entered by the sender)'.

4B shows a case where the sender operates the message application of the originating terminal 110 to select 'text' for the multimodal type and 'recognition type' for the symbol message type, and the sender inputs the text, which is the multimodal information, And the called party confirms 'symbols corresponding to a specific word' (403, 404) in the symbol message. Here, the body content of the text, which is the multi-modal information input by the caller, is "Have you listened to the story about the sun and the moon?", And "Sea" and "Moon" .

5 is a flowchart of an embodiment of a method of a multimodal messaging service according to the present invention. Here, the originating terminal 110 and the terminating terminal 130 operate based on a pre-installed message application to use the multimodal messaging service.

First, when a multi-modal type and a symbol message type are selected by the caller (S501), the originating terminal 110 inputs multi-modal information for the multi-modal messaging service by the caller and the content of the message is created by the caller (S502 ). In this case, the originating terminal 110 generates and transmits a raw message using the multimodal information and the text content (S503, S504).

Then, the multimodal messaging service apparatus 120 extracts the multimodal information and the content of the text while grasping the multimodal type and the symbol message type in the original message transmitted from the originating terminal 110 (S505). At this time, the multimodal messaging service apparatus 120 inquires a symbol according to the multimodal information and the symbol message type (S506), and generates and transmits a symbol message in which the symbol and the text content are combined to the terminating terminal 130 (S507 , S508).

Meanwhile, when the symbol of the symbol message is selected by the called party, the terminating terminal 130 calls the one-sided symbol linked to the symbol from the multimodal messaging service apparatus 120 and provides it to the called party (S509).

As an alternative embodiment of the present invention, the multimodal messaging service device 120 may send a message according to message preferences (i.e., preferred multimodal type, preferred symbol message type) by the foot / You can make / receive calls.

First, the caller may configure the multimodal messaging service apparatus 120 to use only a predetermined multimodal messaging service. At this time, when the originator generates and transmits a raw message, the multimodal messaging service apparatus 120 transmits the original message including any multimodal information without using the message application to set the multimodal type and the symbol message type, And transmits the generated symbol message to the terminating terminal 130 according to a predetermined message configuration. This means that the sender can transmit a message to the called party according to the message characteristics desired by the terminal according to the terminal specification and the message preference.

Specifically, the multimodal messaging service apparatus 120 checks multimodal information in the original message transmitted from the originating terminal, and determines whether the multimodal messaging service apparatus 120 corresponds to the multimodal type and the symbol message type set by the originator. Thereafter, the multimodal messaging service apparatus 120 processes the original message according to the determination result, and transmits the original message to the called terminal. At this time, if the multimodal messaging service apparatus 120 corresponds to the multimodal type and the symbol message type set by the caller, the multimodal messaging service apparatus 120 checks the multimodal information and the symbol message type to inquire the symbol, Generates a symbol message according to the symbol message type, and transmits the symbol message to the terminating terminal 130.

Likewise, the called party may be configured to use only a predetermined multimodal messaging service through the multimodal messaging service device 120. At this time, when receiving a symbol message, the called party can receive only a desired type of symbol message according to a predetermined message environment setting in the multimodal messaging service apparatus 120. [ For example, the called party may remove the link of the symbol message and receive the message. This means that the called party can receive the message according to the desired message characteristic according to the terminal specification and the message preference.

Specifically, the multi-modal message service apparatus 120 determines whether the multi-modal type and the message type correspond to the multi-modal type and the message type predetermined by the called party by checking the multi-modal type and the message type in the symbol message. Thereafter, the multi-modal message service device 120 processes and transmits the symbol message. At this time, if the multimodal message service apparatus 120 corresponds to the multimodal type and the message type set by the called party, the multimodal message service apparatus 120 transmits the symbol message to the called terminal 120 as it is. On the other hand, if the multimodal message service apparatus 120 does not correspond to the multimodal type and the message type set by the called party, the symbol message is removed from the symbol message and transmitted.

Meanwhile, the method of the present invention as described above can be written in a computer program. And the code and code segments constituting the program can be easily deduced by a computer programmer in the field. In addition, the created program is stored in a computer-readable recording medium (information storage medium), and is read and executed by a computer to implement the method of the present invention. And the recording medium includes all types of recording media readable by a computer.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. The present invention is not limited to the drawings.

1 is a configuration diagram of an embodiment of a multimodal messaging service network according to the present invention;

FIGS. 2A and 2B are diagrams illustrating an embodiment of a multimodal messaging service to which the present invention is applied;

FIGS. 3A and 3D illustrate another embodiment of a multimodal messaging service to which the present invention is applied;

4A and 4B are diagrams illustrating another embodiment of a multimodal messaging service to which the present invention is applied;

5 is a flowchart of an embodiment of a method of a multimodal messaging service according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS

110: originating terminal

120: Multimodal messaging service device

130:

Claims (11)

A multimodal messaging service method comprising: An inquiry step of checking the multi-modal information and the 'second message type to be transmitted to the destination terminal' in the first message transmitted from the calling terminal and inquiring the symbol; And Linking the inquired symbol to the multimodal information and transmitting a second message generated according to the 'second message type to be transmitted to the called terminal' to the called terminal Wherein the multi-modal messaging service method comprises: The method according to claim 1, After the transmitting step, Responding to the confirmation request of the multi-modal information linked to the inquired symbol by the terminating terminal Further comprising the steps of: The method according to claim 1, The originating terminal, A method of multimodal messaging service using an application for a multimodal message service, the method comprising: generating the first message including the multimodal information and a second message type to be transmitted to the called terminal. 3. The method of claim 2, The called terminal, A method for multimodal messaging service using an application for a multimodal message service, said multimodal messaging service requesting said multimodal message service to confirm said multimodal information linked to said queried symbol in said second message. 5. The method according to any one of claims 1 to 4, The 'second message type to be transmitted to the called terminal' Modal information is a second type for linking a 'symbol indicating a multimodal type' or a 'second symbol for linking a recognition result of multimodal information' to the multimodal information. Modal messaging service method. 6. The method of claim 5, The 'second message type to be transmitted to the called terminal' The method of any of the preceding claims, wherein the first and second types are determined by a default (defalut) and are determined by a user selection. 6. The method of claim 5, The multi- Image, sound, and text. ≪ Desc / Clms Page number 21 > A multimodal messaging service method comprising: Modal information in the first message transmitted from the originating terminal and determining whether the multimodal type is determined by the originator and the second message type to be transmitted to the called terminal; And a transmission step of processing and transmitting the first message according to the determination result Wherein the multi-modal messaging service method comprises: 9. The method of claim 8, Wherein, If the result of the determination is that the multimodal type set by the caller and the 'second message type to be transmitted to the called terminal', the multi-modal information and the 'second message type to be transmitted to the called terminal' step; And Linking the inquired symbol to the multimodal information and transmitting a second message generated according to the 'second message type to be transmitted to the called terminal' to the called terminal Wherein the multi-modal messaging service method comprises: A multimodal messaging service method comprising: Determining a multi-modal type and a message type in a 'message transmitted to a receiver terminal' and determining whether the multi-modal type and the message type correspond to a predetermined multi-modal type and a message type; And A transmission step of processing and transmitting the 'message transmitted to the called terminal' according to the determination result Wherein the multi-modal messaging service method comprises: 11. The method of claim 10, Wherein, Transmitting the 'message transmitted to the called terminal' as it is, if the result of the determination corresponds to the multimodal type and the message type preset by the called party; And If the result of the determination does not correspond to the multimodal type and the message type preset by the called party, removing the link of the symbol of the 'message transmitted to the called terminal' Wherein the multi-modal messaging service method comprises:

Images