KR20190062145A - Method for providing messaging service and computer readable medium for storing program for executing the method for providing messaging service - Google Patents

Abstract

According to the present invention, provided is a method for providing a messaging service, which comprises the steps of: analyzing a real image obtained in real time by a user terminal to identify a first image region matching a predetermined first criterion; displaying a pre-stored messenger image visually above the identified first image region while displaying the obtained real image through a display provided on the user terminal; and generating a conversation message including a voice obtained by the user terminal and a messenger image displayed through the display while the real image and the messenger image are displayed together.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a messaging service providing method and a messaging service providing method, and more particularly,

The present invention relates to a method of providing a messaging service and a computer-readable medium storing a program for performing the messaging service providing method.

As mobile terminals that can be portable to most users are widely used, a method of exchanging messages between user terminals in various ways has been proposed. In particular, due to the development of the network, the burden of increasing the size of the data constituting the message is less than in the conventional art. Meanwhile, since various input and output means are provided in the user terminal, It is now possible to create.

The present invention provides a method of providing a messaging service for generating and transmitting an interactive message using an input means such as an imaging device or a microphone provided in a portable user terminal and an output means such as a display or a speaker and a method of providing such a messaging service And provides a medium that can be read by a computer storing a program to be used.

The present invention provides a messaging service for analyzing a real image obtained from a user terminal and generating a conversation message by outputting a virtual object (for example, a character) considered as a messenger for delivering a message together with a real image through a user terminal And a computer-readable medium storing a program for performing the messaging service providing method.

The present invention generates a conversation message together with a voice acquired from a user terminal while displaying a real image and a virtual object to the user terminal, or generates a conversation message including contents that can be purchased online together with a virtual object A messaging service providing method, and a computer readable medium storing a program for performing the messaging service providing method.

The present invention provides a method of providing a messaging service for outputting a received conversation message, the method comprising: analyzing a real image obtained from a user terminal and outputting a conversation message when an image matching the pre- And a computer readable medium storing a program for performing the method.

A method for providing a messaging service according to the present invention comprises the steps of: analyzing a real image obtained in real time by a user terminal to identify a first image area corresponding to a predetermined first reference; Displaying the pre-stored messenger image visually at the top of the identified first image area while displaying the obtained real image, and displaying the pre-stored messenger image captured by the user terminal while the real image and the messenger image are displayed together And generating a conversation message, the conversation message including a voice and a messenger image displayed via the display.

In one embodiment, generating the conversation message includes generating the conversation message including the characteristic information of the messenger image, which is displayed on the display during the time the voice is acquired by the user terminal can do.

In one embodiment, the displaying of the messenger image may include rendering the messenger image in three dimensions based on characteristics information of the messenger image.

In one embodiment, the step of displaying the messenger image may include adjusting the size of the pre-stored messenger image corresponding to the size of the identified first image area, and displaying the adjusted size of the messenger image.

In one embodiment, generating the conversation message may include modulating the obtained speech.

In one embodiment, the generating of the conversation message may include setting a conversation message output condition including a reality image reference to which the messenger image included in the conversation message is to be output.

The method of providing a messaging service according to the present invention includes the steps of: analyzing a real image obtained in real time by the user terminal when receiving the conversation message to identify a second image region matching a predetermined second criterion; Displaying the messenger image on the visual upper part of the second image area on the basis of the messenger image characteristic information included in the chat message in the identified second image area while displaying the obtained reality image through the display And outputting the voice included in the conversation message together with the messenger image.

For example, the predetermined second criterion may include a match rate with a predetermined shape that is equal to or greater than a reference value.

In one embodiment, the generating of the conversation message may include generating the conversation message further including content purchased by connecting to the content server.

In one embodiment, generating the conversation message includes generating the purchase content identifier based on recipient or sender information of the conversation message, and providing the purchase content identifier to the user terminal can do.

The method of providing a messaging service according to the present invention further includes a step of authenticating a purchaser through authentication information provided through the user terminal, and requesting settlement of contents in accordance with the payment information associated with the purchaser can do.

The method of providing a messaging service according to the present invention is a method for providing a messaging service in which when a message output condition including a reality image reference in which the messenger image is displayed is set for the received chat message, The method comprising the steps of:

A method for providing a messaging service according to the present invention includes the steps of: analyzing a real image obtained in real time by a first user terminal to identify a first image region matching a predetermined first reference; Displaying a messenger image visually above the identified first image area together with displaying the obtained real image through a display, acquiring a voice through the first user terminal while the messenger image is being displayed Setting a chat message output condition for outputting a chat message including the messenger image and voice, setting the chat message output condition including the characteristic information for the messenger image, the obtained voice, and the chat message output condition Generating a conversation message, receiving a conversation message from a second user terminal, It is determined whether or not the second user terminal meets the dialog message output condition and a step of outputting the message dialog.

In one embodiment, the step of receiving the conversation message by the second user terminal may include receiving the conversation message output condition first in the conversation message.

According to an exemplary embodiment of the present invention, a method for providing a messaging service according to the present invention may further include the step of outputting information indicating the conversation message output condition through the display by the second user terminal.

In one embodiment, the step of receiving the conversation message by the second user terminal may include requesting and receiving the entire conversation message when it is determined that the conversation message output condition is satisfied.

The present invention includes a program stored on a medium for executing in a computer a method according to an embodiment of the present invention.

The present invention includes a computer-readable recording medium storing a program for causing a computer to execute a method according to an embodiment of the present invention.

A method of providing a messaging service according to the present invention and a computer-readable medium storing a program for performing the messaging service providing method include input means such as an image pickup element or a microphone provided in a portable user terminal, And can transmit and receive a conversation message by using an output means such as the Internet.

A method of providing a messaging service according to the present invention and a computer-readable medium storing a program for performing the messaging service providing method include analyzing a real image acquired from a user terminal and transmitting the message together with a real image It is possible to generate a conversation message by outputting a virtual object such as a character to be considered through the user terminal.

A method of providing a messaging service according to the present invention and a computer-readable medium storing a program for carrying out the method of providing a messaging service, includes the steps of: receiving, from a user terminal, The user can generate a conversation message together with content that can be purchased online together with the virtual object.

A method of providing a messaging service according to the present invention and a computer-readable medium storing a program for performing the messaging service providing method include the steps of analyzing a real image obtained from a user terminal in outputting a received conversation message, It is possible to output a conversation message when an image matching the preset criteria is identified.

1 is a diagram schematically illustrating a configuration of a system for providing a messaging service according to an embodiment of the present invention.
2 is a block diagram showing a specific configuration of a user terminal providing a messaging service and a server.
FIG. 3 is a sequence diagram for explaining a conversation message generation process included in the messaging service providing method according to an embodiment of the present invention. FIGS. 4A and 4B are diagrams illustrating a process of generating a conversation message through the user terminal 10 Are views showing exemplary screens.
FIGS. 4A to 4D are views illustrating screens provided when a virtual reality providing method according to various embodiments of the present invention is performed in a user terminal.
5A and 5B are sequence diagrams for explaining a step of purchasing contents and generating a conversation message. FIGS. 6A to 6C show an exemplary screen output through a display of a user terminal in a process of purchasing contents and generating a conversation message Respectively.
FIG. 7 is a sequence diagram illustrating a process of outputting a conversation message, and FIGS. 8A and 8B are diagrams illustrating an exemplary screen in which a conversation message is output through a display of a user terminal.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to clarify the technical idea of the present invention. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of a computer system according to an embodiment of the present invention; Fig. For convenience of explanation, the apparatus and method are described together when necessary.

1 is a diagram schematically illustrating a configuration of a system for providing a messaging service according to an embodiment of the present invention.

Referring to FIG. 1, a messaging service providing system 1 may include a user terminal 10 and a server 20 connected through a network 30.

The user terminal 10 may be provided with an application for performing a messaging service providing method according to the present invention. Through the application, the user terminal 10 can generate a conversation message and deliver the generated message to the server 20. [ The conversation message delivered to the server 20 can be transmitted to the user terminal of the designated recipient of the conversation message. The user terminal 10 of the recipient receiving the conversation message can execute the application for performing the messaging service providing method and output the received conversation message.

In the present invention, an application installed in the user terminal 10 may be recorded in a separate storage medium provided in the user terminal 10. The application installed in the user terminal can be automatically executed in the user terminal 100 as the conversation message is received in the user terminal 10. [ In another embodiment, the information that enables the application to run with the conversation message is delivered, and the application may be executed in response to the application execution information being selected through the user terminal that received the conversation message.

The user terminal 10 may be provided with imaging means capable of obtaining a real image. The conversation message according to the present invention can be generated in the process of displaying the real image obtained through the image sensing means through the display provided on the user terminal 10 as well. In the present invention, a chat message transmitted and received may include a virtual messenger image displayed together with a real image on a real image, a user's voice or purchased contents. According to the embodiment, a condition for the conversation message to be output to the recipient's user terminal 10 may be set for the conversation message, and according to the embodiment, such a condition may be included in the conversation message as the conversation message output condition.

In the process of outputting the conversation message at the user terminal 10 receiving the conversation message, the real image and the conversation message obtained through the imaging means provided in the user terminal 10, exactly as in the case where the conversation message is generated, The virtual IM image included in the conversation message and the voice or the content may be displayed together. Therefore, the messaging service according to the present invention can be understood as a service for transmitting and receiving a virtual reality conversation message including an augmented reality. The user terminal 10 may correspond to the virtual reality providing apparatus since the conversation message to be transmitted and received may be provided as a virtual reality.

The generation and output of the conversation message will be described in detail with reference to FIG.

The server 20 can construct a server-client system with the user terminal 10. The server 20 according to the present invention may be implemented as a single server device or a plurality of server devices connected to each other, and may be implemented as a cloud server according to an embodiment.

The system 1 for performing the messaging service providing method according to an embodiment of the present invention may further include a content server 40. [ The content server 40 may be a server that sells content-type content. The content server 40 stores content information and provides content information to the user terminal 10. [ When the user confirms the content information provided to the user terminal 10 and transmits a content purchase request to the content server 40, the content server 40 can request the user terminal 10 for authentication. According to the embodiment, the content server 40 can perform settlement of the content fee according to the settlement information linked to the authentication information based on the authentication of the user. The content sold by the content server 40 may include various digital content. The content sold by the content server 40 includes not only intangible products but also digital exchange coupons that can be exchanged into tangible goods. If the content can be provided to the user terminal 10 through the network 30 And may be included in the content sold by the content server 40.

The operating entity of the server 20 and the content server 40 may be the same or different. The content server 40 may be physically implemented or separately implemented with the server 20 according to the embodiment.

In another embodiment, the server 20 and the content server 40 may share the user information provided from the user terminal 10. When the user terminal 10 is connected to the server 20 or the application is activated in the user terminal 10 and the user identifier is delivered to the server 20 as the user information is shared, The user information of the user terminal 10 may be transmitted to the content server 40. [ Therefore, when purchasing content or checking content information through the user terminal 10, it is not necessary to separately provide user information (e.g., a user identifier) to the content server 40 through the user terminal 10. [

2 is a block diagram showing a specific configuration of a user terminal providing a messaging service and a server.

2, the user terminal 10 includes an input unit 110, an image processing unit 120, a control unit 130, an output unit 140, a conversation message generation unit 150, a storage unit 160, 170, and an interfacing unit 180.

The input unit 110 may include various input means included in the user terminal 10. The input unit 110 may include a visual information receiving unit for receiving visual information such as an image sensing unit such as a camera and an infrared sensor and an auditory information receiving unit for receiving auditory information such as a microphone. According to an embodiment, the input unit 110 may include a position information receiving unit for receiving position information such as GPS.

The time information receiving unit included in the input unit 110 scans real time information in two or three dimensions to display a real image through an output unit 140 such as a display provided in the user terminal 10 . At the same time, the visual information receiving unit provides the real image to the image processing unit 120 to analyze the characteristics of the real image and identify the image region or the object meeting the predetermined criteria. In addition, the image processing unit 120 may determine the output condition of the conversation message by identifying the specific image region or object of the real image obtained through the time information receiving unit.

The audible information receiving unit included in the input unit 110 receives the voice information input from the user and includes the voice information in the conversation message or authenticates the true user of the user terminal 10 through voice.

In addition, in the method of providing a messaging service according to an exemplary embodiment of the present invention, a conversation message output condition may be set to output a conversation message when a specific condition is determined to be satisfied through a user's voice. Therefore, the conversation message output condition can be determined through the voice collected through the auditory information receiving unit.

The image processing unit 120 can analyze real images collected through the time information receiving unit of the input unit 110 in real time. The image processing unit 120 may extract three-dimensional feature points on a frame-by-frame basis according to predetermined conditions and analyze feature points during a predetermined frame to identify whether the image regions conform to predetermined conditions. According to the embodiment, the image processing unit 120 sets a region of interest (ROI) of a part of the real images obtained through the input unit 110 and analyzes the ROI to analyze the image processing speed Can be improved.

The image processing unit 120 may identify a conversation message generating area suitable for displaying a character image, that is, a messenger image, on the real image in the conversation message generating step. In order for the messenger image to be displayed through the display, it may be required that a space of a certain area or more having a uniform color is secured so that the character image overlaps with the real image to stably display the messenger image. Accordingly, the image processing unit 120 can analyze the real image in real time and identify the image as a message generation area suitable for displaying the messenger image when it is determined that the floor or the planar space is equal to or larger than a predetermined area. Of course, the condition for identifying the conversation message generation area may be variously set according to the embodiment.

The image processing unit 120 may transmit the identification information of the dialog message generation area to the control unit 130. [ The reference for identifying the dialog message generation region by the image processing unit 120 may include a case where the detected plane is greater than or equal to a predetermined region as the three-dimensional feature points of the real image are extracted and the plane is detected based on the stable feature points .

In the outputting step of outputting the dialog message, that is, in the step of receiving and confirming the received dialog message, the image processing unit 120 identifies whether the dialog message output area suitable for outputting the received dialog message is on the real image. The image processing unit 120 may identify the dialogue message output area on the same basis as the dialogue message creation area, or may identify the feature point on a frame-by-frame basis and use a part of the body of a person or animal such as a hand, palm, A specific object such as a plate, a cup, and the like can be recognized to identify a dialog message output area.

At this time, the image processing unit 120 stores shape information about the object to be identified. When the matching ratio between the shape information of the previously stored object and the real image is equal to or greater than the reference value, . For example, the image processing unit 120 stores information on a feature point distribution formed by the shape of the palm, and analyzes real images obtained through the input unit 110 in real time. When the minutia distribution matching rate is 85% or more through the minutiae analysis of the obtained real image, the image processing unit 120 can identify the corresponding area as a dialog message output area.

The image processing unit 120 can identify the conversation message generation area and the conversation message output area according to various criteria and can provide the identified area information to the control unit 130 in real time. According to the embodiment, the image processing unit 120 may identify different types of objects according to the conversation message output conditions included in the conversation message when the conversation message output area is identified.

The control unit 130 may control the operation of a plurality of components constituting the user terminal 10. [ The controller 130 may correspond to a processor and may include a computing device such as a central processing unit (CPU) or an application processor (AP). In one embodiment, the control unit 130 may further include a GPU (Graphical Processing Unit). The GPU can control the output screen through the display of the output unit 140. [ Also, in one embodiment, the controller 130 may be implemented as an integrated chip (IC) such as a system on chip (SoC) integrated with various computing devices. The integrated chip may include at least a part of the communication unit 170, for example, in addition to the control unit 130. [

The controller 130 receives the information indicating that the conversation message generating region or the conversation message output region has been identified from the image processing unit 120. The control unit 130 then displays the messenger image on the corresponding identification area of the display of the user terminal 10 via the output unit 140, Can be displayed. The messenger image displayed through the image processing unit 120 according to the embodiment may be rendered in two or three dimensions by deriving various resources stored in the storage unit 160 according to the messenger image characteristic information. For example, if only a variety of characteristic information such as vector information of a plurality of feature points constituting a messenger image, size, color, effect, motion and the like of the messenger image are provided to the image processing unit 120, A resource can be derived based on the messenger image characteristic information from the server 160 and a messenger image outputted through the display can be implemented. The resources stored in the storage unit 160 may be provided from the server 20 or purchased from the content server 40.

Accordingly, the output unit 140 displays the messenger image on the region identified in the real image together with the real image according to the control of the control unit 130.

In one embodiment, a conversation message may be output only if a particular region of the real image is identified and additional external conditions are satisfied. These additional external conditions may be set as default or as conversation message output conditions in the process of generating the conversation message. The controller 130 may control the output of the conversation message by determining the output condition of the conversation message based on the external information received from the input unit 110 or the interfacing unit 180, Information indicating a dialog message output condition may be displayed to the unit 140 to guide the user to form an external condition for confirming the conversation message.

The output unit 140 may include a visual information output unit such as a display and an audible information output unit such as a speaker. The visual information output unit may output a real image obtained from the input unit 110 in real time. In the present invention, the visual information output unit includes an output unit for visually expressing a real image, a messenger image or a conversation image, Various types of display devices such as an LCD (Liquid Crystal Display), an LED (Light Emitting Diode), an OLED (Organic LED), an AMOLED (Active Matrix OLED), and a PDP (Plasma Display Panel)

In one embodiment, the time information output unit of the output unit 140 includes sensing means for recognizing various inputs occurring in a region for displaying an image, an input by a user's touch input, an electronic pen such as a stylus It is possible.

The output unit 140 may display the real image and the messenger image through the time information output unit, and may output the audio information through the audio information output unit. The auditory information output unit of the output unit 140 may reproduce the voice inputted in the process of generating the conversation message by the user and provide it to the user as feedback or output the voice information included in the conversation message to the user who received the conversation message can do.

In the present invention, the output unit 140 outputs the modulated voice rather than outputting the voice of the user, thereby increasing the interest of the user in the conversation message. The operation of modulating the voice of the user may be performed in the control unit 130, the conversation message generation unit 150, or the output unit 140 according to the embodiment.

The conversation message generation unit 150 may generate the conversation message by combining the messenger image characteristic information for implementing the messenger image or the messenger image itself and the voice information. When the user further purchases the contents from the contents server 40 according to the embodiment, the conversation message generator 150 may generate a conversation message including the contents purchased by the user. In another embodiment, the conversation message generation unit 150 may generate a conversation message including a conversation message output condition for setting various conditions under which the conversation message can be output.

The storage unit 160 may store an application running on the user terminal 10 or may store user information. In addition, the storage unit 160 may store a resource of a messenger image displayed when the conversation message is generated, and may store the conversation message generated by the user and the received conversation message. The storage unit 160 may store identification reference information such as a minutia template for identifying an image area when generating or outputting a conversation message, and a voice modulation program for modulating voice.

The communication unit 170 can transmit the conversation message generated in the user terminal 10 to the server 20 through the network 30 (see FIG. 1) or receive various data necessary for generating the conversation message from the server 20 have. For example, the user terminal 10 may receive resources, messenger image characteristic information for constructing various kinds of instant messenger images from the server 20, and receive various voice modulation programs for modulating voice. Also, the user terminal 10 may receive the conversation message transmitted from another user through the server 20.

According to the embodiment, only the acquisition of the real image is performed in the user terminal 10, and the analysis and processing of the real image and the modulation of the audio information can be performed in the server 20. In this case, the real image and voice obtained through the user terminal 10 can be provided to the server 20 in real time. In addition, since the conversation image generated by the user terminal 10 includes voice and moving image information, the capacity may be large. Therefore, the communication unit 170 may compress the image and the voice including the image to the server 20 in consideration of the resolution and the network state of the user terminal 10. For example, the image processing unit 120 of the user terminal 10 can compress an image using h.264 or vp8 / 9, and voice using a codec such as Opus, iSAC, or iLBC. The user terminal 10 can send meta information including network information of the user terminal 10 and header and image data of the user terminal 10 to another user's terminal or server 20 for network communication to prepare for network communication have.

After the communication is established once, the communication unit 170 can manage the network session and the packet so that the established communication is continued. For this, the communication unit 170 may use a module specialized in real time video / audio communication such as WebRTC, which is implemented in advance.

The communication unit 170 communicates with the communication unit 210 of the server 20 through a cellular communication network such as 2G / 3G, LTE (Long Term Evolution), and LTE-A (LTE-Advance) Lt; / RTI > The communication unit 170 may include an RF (Radio Frequency) chip or a circuit that supports various communication methods. According to the embodiment, the communication unit 170 may directly communicate with the communication unit of another user terminal without going through the server 20. In this case, a short distance communication method may be used.

The interfacing unit 180 may provide various interface objects to the user terminal 10 and obtain information received therefrom. The interfacing unit 180 may receive the user's command from the user terminal 10 by providing the interface object through the time information output unit. The generation of the conversation message can be started and completed based on the user command received through the interfacing unit 180 and the operations such as transmission of the conversation message and output of the conversation message can be performed.

The user terminal 10 may further include an authentication unit 190 according to an embodiment. The authentication unit 190 stores authentication information for a user who is a true owner of the user terminal 10 and may perform user authentication by comparing the previously stored authentication information with the inputted authentication information when user authentication is requested. The authentication unit 190 may store various authentication information such as iris, fingerprint, hand shape, voiceprint, password, payment information, and signature information of the user.

According to an embodiment, the authentication unit 190 may perform authentication of an external authentication server without internally storing the authentication information. For example, the authentication unit 190 may provide the authentication information obtained from the user terminal 10 to the external authentication server in response to the authentication request, and may transmit the authentication completion information received from the authentication server.

The server 20 may include a communication unit 210, a user management unit 220, and a storage unit 230. Various steps for performing the messaging service providing method according to the present invention may be performed dispersedly between the user terminal 10 and the server 20 according to the embodiment. Accordingly, the server 20 can be configured to selectively perform functions performed in the user terminal 10. [

The communication unit 210 may receive the conversation message from the sender user terminal 10 that generated the conversation message and may transmit the conversation message to the recipient user terminal 10. [ The server 20 may receive the completed conversation message from the user terminal 10 but according to the embodiment the server 20 receives the messenger image characteristic information and the respective information of the voice from the user terminal 10, Output conditions, and the like, and may process the received information to finally generate a conversation message.

The server 20 may also mediate communication between the content server 40 and the user terminal 10. [ The server 20 transmits a purchase request to the content server 40 and provides the user information of the user terminal 10 to the content server 40 when the content purchase request is received from the user terminal 10, May be implemented so as to be recommended. In another embodiment, the server 20 communicates information about the user terminal 10 to the content server 40 and allows the content server 40 and the user terminal 10 to communicate directly.

The user management unit 220 manages the user information received from the user terminal 10. For example, the user management unit 220 may manage a conversation message transmission / reception history, user identification information, user content purchase information, user friend information, and messenger image information selected by the user with respect to the user identifier. If necessary, the user management unit 220 can derive specific information from the storage unit 230 based on the user information.

The storage unit 230 may manage a conversation message, a resource for generating a messenger image, and an audio modulation program resource.

The messaging service providing method according to the present invention can be divided into a step of generating and outputting a conversation message. The steps of generating a conversation message will be described first with reference to Figs. 3 to 6C.

FIG. 3 is a sequence diagram illustrating a process of generating a conversation message included in a method of providing a messaging service according to an embodiment of the present invention. FIGS. 4A and 4B illustrate an exemplary screen FIG.

Referring to FIG. 3, a user may subscribe to a messaging service by executing an application at the user terminal 10 and transmitting information necessary for service subscription to the server 20 (step S110). The server 20 receives the terminal identification information of the user terminal 10, personal information input by the user, and registers the user. When the application is executed after the service registration and the connection is requested from the user terminal 10, the server 20 can identify the user.

An interface is provided through a visual information output unit such as a display of the user terminal 10, and the user can input a command to start the generation of the conversation message via the provided interface. The time information receiving unit included in the input unit 110 of the user terminal 10 can be activated in response to the user's start command to obtain a real image (step S120).

The real image obtained from the input unit 110 may be provided to the image processing unit 120 and analyzed in real time (step S130). The image processing unit 120 may perform the feature point analysis by analyzing the real image obtained from the input unit 110 on a frame-by-frame basis.

Depending on the embodiment, the real-time image analysis may be performed at the server 20 as it is not performed at the user terminal 10, but rather conveys or compresses the real image to the server 20 at the user terminal 10 . The server 20 may analyze the image received from the user terminal 10 and transmit the analysis result to the user terminal 10. [

4A is a diagram illustrating an exemplary screen displayed through a display of the user terminal 10 in the process of performing minutia analysis. Referring to FIG. 4A, a real image 410 is displayed on a display, and the real image is analyzed to confirm that a certain area is identified as a conversation message generating area 420. According to the embodiment, the control unit 130 can help the user to move the user terminal 10 to easily secure the conversation message generating area by highlighting and expressing the conversation message generating area with respect to the output unit 140 .

Referring to FIG. 4A, various messenger image thumbnails 430 may be displayed on the display. The user may select one of the IM image thumbnails 430 to select the IM image to generate the IM message. The messenger image characteristic information stored in the storage unit 160 is transmitted to the image processing unit 120 as the user selects the IM image thumbnail 430 and the image processing unit 120 transmits the messenger image characteristic information stored in the storage unit 160 160 and implements the messenger image through the output unit 140. [

3, when the image processing unit 120 identifies the conversation message generating area (step S140, YES), the user terminal 10 displays the real image obtained in real time, while displaying the messenger image in the conversation message generating area (Step S150). The image processing unit 120 of the user terminal 10 may display the messenger image in a specific one of the real images by overlaying the messenger image in the real image messaging area.

According to the embodiment, the resources necessary for the implementation of the messenger image and the message image characteristic information may be provided from the server 20 (step S210). However, the timing with which the server 20 provides the information (characteristic information, resources, etc.) necessary for implementing the messenger image is not limited to the order shown in FIG. 3. However, as shown in FIG. 4A, The messenger image information may be requested from the server 20 to the user terminal 10 when the user terminal 430 is selected. If the messenger image is already implemented through the user terminal 10, information necessary for implementing the messenger image may be provided to the user terminal 10 at an arbitrary point in time, or may be provided to the storage unit 160 of the user terminal 10 In the case where information necessary for implementing a messenger image is already stored, the messenger image may not be provided from the server 20 during the conversation message generation process.

However, for at least some time before the conversation message generation area is identified and the messenger image is displayed, information necessary for the implementation of the messenger image may be stored in the storage unit 160 of the user terminal 10 and output through the display .

Referring to FIG. 4B, it can be seen that the messenger image 440 is displayed in the conversation message creation area 421 together with the real image 410. [ The messenger image 440 may be displayed in at least a part of the conversation message creation area 421. [ After the messenger image 440 is displayed, the real image 410 acquired by the user terminal 10 can be changed according to the movement of the user terminal 10, and the conversation message generation area 421 ) May vary in position and size. The image processing unit 120 may continuously monitor the chat message generation area 421 by analyzing the reality image even after the messenger image 440 is displayed. Accordingly, the image processing unit 120 may adjust the size of the messenger image 440 by comparing the size of the conversation message generating area 421 and the size of the messenger image 440. [

For example, since the display size of the user terminal 10 is limited, the maximum size at which the IM image 440 can be displayed can be limited. The maximum size of the messenger image 440 may be determined to be less than a predetermined size regardless of specifications of the user terminal 10 or may be determined in various ways according to the specifications of the user terminal 10. For example, the maximum size of the instant messenger image 440 may be specified as a horizontal and vertical pixel size, or may be specified as a specific cost of the display of the user terminal 10, for example, 40% or less.

If the size of the messenger image 440 and the size of the conversation message generation area 421 are equal to or larger than predetermined values, the image processing unit 120 adjusts the size of the messenger image 440, The size of the IM image can be adjusted so that the ratio is below the predetermined value.

In another embodiment, when the size of the already identified conversation message creation area 421 is reduced to a predetermined size or less, the image processing unit 120 recognizes that the message creation area is not identified, and re-identifies the conversation message creation area can do. However, the criterion for determining that the identification of the message generation area is canceled after the identification of the message generation area by the priority image processing unit 120 may not be strict than the criterion for identifying the message generation area in step S140.

The messenger image displayed through the display is a three-dimensional character image and may be in a moving form. The messenger image can be implemented through the resource based on the messenger image characteristic information as described above. The messenger image 440 shown in FIG. 4B can be moved, and the user can select the messenger image 440 via the interfacing unit 180 to interact with the messenger image. For example, when the user selects the messenger image 440, the effect sound may be output or the movement of the messenger image may be changed.

The user may add an effect to the messenger image 440 or modify the color or movement of the messenger image 440 via the interfacing unit 180. [ Accordingly, the user can generate a conversation message including the desired IM image 440. The characteristic information of the messenger image may be different according to the operation of the user's messenger image 440, and the changed messenger image characteristic information may be included in the conversation message.

After the messenger image 440 is displayed on the display, the user can select the voice input button 450 provided through the interfacing unit 180 to add voice information to the messenger image 440. [ When the user selects the voice input button 450 and provides a voice message, the auditory information receiving unit included in the input unit 110 of the user terminal 10 receives voice information (step S160). The received voice information may be provided to the storage unit 160 through the control unit 130 or may be provided to the conversation message generation unit 150 immediately.

Here, the real image 410 may be displayed on the user terminal 10 and the messenger image 440 may be displayed while the user is inputting the voice information. The image processing unit 120 may extract the messenger image 440 displayed while the voice information is being input, as a dialogue image, and provide the extracted message image to the dialogue message generation unit 150. In response to a command such as selection of the voice input button 450, the controller 130 determines that the user generates a conversation message during the time when the voice information is input, and transmits the message to the image processing unit 120 only for the instant messenger image Extracts it as a dialog image and provides it to the dialog message generation unit 150. In this case, the conversation image can include all the information that the character moves in real time.

According to the embodiment, since the messenger image during the input of the voice information may be implemented according to the messenger image characteristic information, the specific movement or shape of the messenger image implemented in the receiver's user terminal 10 according to the messenger image characteristic information It is possible that the implementation type does not completely match the messenger image actually verified by the creator of the conversation message. In this case, only the characteristic information of the messenger image is included in the conversation message, and the user terminal that has received the conversation message generates the messenger image based on the characteristic information of the messenger image, You can also implement a new image.

The modulated voice may be output to the user while the voice information is input according to the embodiment. The user can predict how the voice information inputted by the user will be modulated and provided to the recipient. According to the embodiment, the voice information included in the conversation message according to the present invention can be modulated in various ways and the voice modulation scheme can be changed according to the user's selection.

The user terminal 10 may receive the recipient information to receive the conversation message currently being generated (step S170). The recipient information may include an identifier such as the recipient's telephone number.

The conversation message generation unit 150 may generate a conversation message including the conversation image received from the image processing unit 120, or may generate the messenger image including the messenger image characteristic information (step S180). The conversation message generation unit 150 may generate a conversation message including the voice information received from the input unit 110. [

Speech information can be obtained by analyzing the pitch interval, pattern analysis, bit operation and time delay modulation, or the like using the HPS (harmonic product spectrum) algorithm and the tempo detector library of pitch, spectrum, zero crossing rate and energy of voice data. And can be modulated using sampled voice data. Modulation of the voice information may be performed in the conversation message generation unit 150, the server 20, or the user terminal 10 receiving the conversation message.

The output condition of the conversation message may be set through the interfacing unit 180 of the user terminal 10 according to the embodiment. The conversation message according to the present invention can be output when a specific object is identified in the real image obtained through the user terminal 10. [ Accordingly, the user can set the output condition of the conversation message in the process of generating the conversation message through the user terminal 10. [ The dialog message output condition that can be selected by the user may include various external conditions that can be detected through the input unit 110. For example, the conversation message output condition may include the kind of object identified from the real image. The user can set a dialog message through the user terminal 10 to be visually displayed at the top of a particular object, such as a palm or a clock.

In addition, the output condition of the conversation message may include various conditions such as a specific time zone, a specific location, and a specific voice command, by which the user terminal 10 can sense the external condition. Various conversation message output conditions can be set through the user terminal 10. When the various output conditions are set as described above, the user who receives the conversation message can check the conversation message in various environments. In addition, according to the embodiment, a condition necessary for the conversation message to be output may be displayed on the output unit 140 of the user terminal 10 to guide the user to satisfy the conversation message output condition.

The control unit 130 may transmit the generated conversation message to the server 20 through the communication unit 170 (step S190). The transmission of the conversation message may be performed by selecting the transmit button 460 shown in FIG. 4B. In another embodiment, the control unit 130 may transmit the conversation message directly to another user terminal 10 through the communication unit 170 without going through the server 20. [

The server 20 stores the received conversation message in the storage unit 230 and records the received conversation message in the user management unit 220. [ In addition, the server 20 may transmit a conversation message to the recipient's user terminal based on the recipient information of the received conversation message (step S220). The recipient information may be included in the conversation message or delivered to the server 20 separately from the user terminal 10.

The server 20 can transmit the conversation message to the recipient's user terminal as soon as the conversation message is provided. In another embodiment, in the case where the server 20 conveys only the conversation message output condition of the conversation message to the recipient user terminal and receives information that the conversation message output condition is satisfied from the recipient's user terminal Other information included in the conversation message, for example, voice information and information for implementing a messenger image, may be transmitted.

In the method of providing a messaging service according to the present invention, the content may be purchased through the user terminal 10 before the generation of the conversation message indicated by " A " . However, the purchase of the content is not limited to the order of " A " shown in FIG. 3, but may be performed at any time as long as the content information can be included in the conversation message.

The process of purchasing contents to generate a conversation message will be described with reference to FIGS. 5 to 6C. 5A and 5B are sequence diagrams for explaining a step of purchasing contents and generating a conversation message. FIGS. 6A to 6C show an exemplary screen output through a display of a user terminal in a process of purchasing contents and generating a conversation message Respectively.

Referring to FIG. 5, an additional input is received through the user terminal 10 (step S171). The additional input may correspond to the selection of the content purchase button 470 shown in FIG. 6A. 6A to 6C, the same reference numerals are used for the same components as in FIGS. 4A and 4B. The components denoted by the same reference numerals perform substantially the same functions, so a detailed description thereof will be omitted.

In response to the user's further input, the user terminal 10 may connect to the content server 40 based on the user information (step S172). The user terminal 10 may provide a user identifier to the content server 40 to directly access the content server 40 or may access the content server 40 from the user terminal 10 via the server 20 without a separate authentication process have.

The content server 40 provides the content information to the user terminal 10 (step S211). The content server 40 may provide content information suitable for a user based on various information obtainable from the user terminal 10. [ For example, the content server 40 may provide content information suitable for a user by applying a recommendation algorithm such as a collaborative filtering algorithm based on the content and a characteristic parameter of the user. In addition, as the user selects a specific category, the content server 40 may provide the information of the contents classified into the category to the user terminal 10.

Contents to be sold by the content server 40 include various intangible goods, intangible goods that can be exchanged with the type merchandise, messenger images that can be used to generate conversation messages, effects applied to messenger images, and voice modulation filters can do.

6B, a content exhibition area 471 is provided on the screen where the real image 410 or the messenger image 440 is displayed, and various contents can be displayed together. In the case where the content that can be purchased and the IM image 440 are displayed together through the display, it may be useful when the content to be purchased is content used for a conversation message. For example, the user can select contents corresponding to characteristics of various available instant messenger images, and directly display the selected contents when they are applied to the instant messenger image.

In another embodiment, when selecting specific content in the content display area 471 through the user terminal 10 and requesting detailed information about the content, the content information requested in the entire display of the user terminal 10 Can be displayed. Also, when the user makes additional input such as selecting the content purchase button 470 in FIG. 6A, both the real image 410 and the messenger image 440 may disappear and move to the content purchase screen. The content purchase screen may be provided in a separate web page or in a separate application.

When the user requests purchase of specific contents (step S173), the contents server 40 can request authentication for payment to the user terminal 10 (step S212).

The user terminal 10 performs user authentication in response to the authentication request (step S174). The authentication of the user can be performed by the authentication unit 190 of the user terminal 10 shown in Fig.

When the authentication of the user is completed, the authentication information is transmitted (step S175). In response to the user's authentication, the content server 40 authenticates the buyer and processes the purchase request (step S215). The user terminal 10 can transmit the payment information of the user to the content server 40 together with the authentication information. Accordingly, the content server 40 can request a payment corresponding to the price of the content to the payment agent server such as the PG server according to the payment information associated with the payment information based on the authentication result.

According to the processing of the purchase request, the content server 40 delivers the purchased content to the user terminal 10 (step S214). The content server 40 may generate a purchase content identifier based on the recipient or sender information of the conversation message to the user terminal 10 and provide the purchase content identifier to the user terminal 10. [ According to an embodiment, the purchased content identifier may be a newly created web page access path based on recipient or sender information of the conversation message. Also, if the purchased content is content applicable in the application, it may be delivered in the form of an identifier indicating that the content is applied to the account of the recipient of the message.

The conversation message generation unit 150 of the user terminal 10 may generate a conversation message including the purchased content (step S181). Referring to FIG. 6C, it can be confirmed that the purchased content 480 is displayed together with the messenger image 440.

According to an embodiment, the content server 40 may deliver the purchased content to the server 20 rather than the user terminal 10. At this time, the purchase content delivered from the content server 40 may include information on which dialog message the purchase content is to be included in. The purchase content information delivered to the server 20 may include the creator of the conversation message to be included in the purchased content and the recipient information of the conversation message or may include an identifier of the conversation message to which the currently purchased content is linked.

The server 20 may link the purchased content received from the content server 40 with the received chat message from the user terminal 10 and re-process the chat message to be transmitted to the receiver.

In another embodiment, when a recipient who has received the conversation message requests content associated with the conversation message, the server 20 may separately provide the stored purchase content with the conversation message. At this time, the user terminal 10 having received the conversation message can request the contents associated with the conversation message including the identification information of the user terminal and the conversation message identification information.

The process of outputting the received conversation message will be described with reference to FIGS. 7 to 8B. FIG. 7 is a sequence diagram illustrating a process of outputting a conversation message, and FIGS. 8A and 8B are diagrams illustrating an exemplary screen in which a conversation message is output through a display of a user terminal.

Referring to FIG. 7, the server 20 transmits a conversation message to the user terminal 10 based on the recipient information specified for the conversation message (step S810).

Referring to FIG. 8A, when a conversation message is received at the user terminal 10, a red highlight indicating that a new conversation message has arrived at the inbox 490 via the interfacing unit 180 may be displayed.

If the user terminal 10 receives the input image 49 through the input unit 110, the user terminal 10 obtains the real image in real time through the input unit 110 (step S710). The image processing unit 120 of the user terminal 10 analyzes the acquired real image 415 in real time while the actual image 415 is being displayed as shown in FIG. 8A (step S720) It is determined whether a matching conversation message output area is identified (step S730).

In the present invention, the predetermined condition for identifying the conversation message output area and the predetermined condition for identifying the conversation message generating area may be the same or different. The predetermined condition for identifying the conversation message output area according to the embodiment may be set in the process of generating the conversation message as described above. That is, an output condition may be added to the conversation message, and the conversation message output area may be determined based on the conversation message output condition.

8A and 8B, the dialog message output area is exemplarily represented as corresponding to a human hand area.

The image processing unit 120 identifies a hand shape of a human being among real images obtained from the input unit 110 through real-time image analysis. As described above, the image processing unit 120 analyzes a real image frame by frame, If the feature point distribution and the matching ratio are equal to or greater than a predetermined value, it is determined that the hand point shape is present, and the identification information may be provided to the control unit 130 accordingly.

According to the embodiment, the output condition of the conversation message can be variously set in addition to designating the conversation message output area. The conversation message output condition may be displayed through the output unit 140 as '427' in FIG. 8A to guide the recipient of the conversation message to satisfy the output condition of the conversation message.

For example, if the dialog message output condition is a Gangnam station and a palm object is designated as a dialog message output area, a condition of 'Gangnam Station' and a condition of a message output area of 'palm' may be presented as shown in FIG. 8A have. The conversation message can be output only when it is confirmed that the user terminal 10 is located in the Gangnam station and the palm is identified in the real image. These conditions may be determined based on information obtained via the input 110 of the user terminal 10. [

According to the embodiment, only the conversation message output condition is transmitted from the conversation message to the user terminal 10, and when it is confirmed that the conversation message output condition is satisfied, the user terminal 10 transmits the conversation message to the server 20 And may request other associated chat messages.

Hereinafter, a description will be given of a case where the dialog message output condition is set only when the dialog message output area is identified in the real image.

If the conversation message output area is identified (step S730, YES), the user terminal 10 displays the messenger image in the conversation message output area based on the conversation message. The messenger image is an image implemented based on the messenger image characteristic information included in the conversation message or the conversation image included in the conversation message, is overlaid on the message output area on the real image through the image processing unit 120, Dimensional character image can be displayed.

Referring again to FIG. 8A, the image processing unit 120 may display the three-dimensional messenger image 445 on the identified hand-shaped conversation message output area 425. FIG. The voice information included in the conversation message may be outputted through the output unit 140 while the messenger image 445 is displayed through the display according to the embodiment (step S750).

In another embodiment, the sender of the conversation message can simultaneously check the output of the conversation message while the recipient receiving the conversation message confirms the conversation message, as shown in FIG. 8B. At this time, the sender image 495 is displayed on a part of the user terminal 10, and the recipient of the conversation message can check the sender image 495 and the conversation message together.

The control unit 130 of the recipient user terminal 10 may request a connection to the sender of the conversation message if the output of the conversation message is determined in the user terminal 10 of the recipient have. That is, when all of the output conditions of the conversation message are satisfied, the control unit 130 outputs a conversation message and requests the sender of the conversation message through the communication unit 170. If a request is made to the sender of the conversation message, the output of the conversation message may be held until the sender of the conversation message approves the connection.

If the purchase message is included in the conversation message (step S760, YES), the purchase content 485 may be displayed in an area adjacent to the messenger image 445 as shown in FIG. 8A. The user terminal 10 can request the information of the purchase content to the server 20 or the content server 40 in response to the user selecting the purchase content 485 or in response to the outputting of all the voice information included in the conversation message (Step S770).

According to the embodiment, the conversation message may store the identifier of the purchase content, and the server 20 may receive the purchase content information by providing the purchase content identifier to an external server such as the content server 40. [ The purchase content information received by the user terminal 10 may be purchase content itself, a bar code or QR code capable of exchanging purchased content, or a web page access path through which purchase content can be exchanged.

According to the embodiment, when it is determined that the receiver has confirmed all the information (messenger image, audio information, and purchase content information) included in the conversation message (step S780), the control unit 130 of the user terminal 10 transmits 20) (step S790)

According to the embodiment, the server 20 can inform the sender of the conversation message that the confirmation of the conversation message is completed when the confirmation of the conversation message is completed. The server 20 notifies the sender of the conversation message through the push alert or the like that the confirmation of the conversation message has been completed. In another embodiment, the control unit 130 transmits the confirmation message to the server 20 ) So that the sender and the recipient of the conversation message can perform the video conversation while watching the IM image together as shown in FIG. 8B.

In the present invention, a computer-readable storage medium storing a program for performing the above-described messaging service providing method may also be included in the scope of the present invention. For example, a storage medium storing computer-readable program instructions may be a hard disk, a floppy disk, magnetic media (e.g., magnetic tape), optical media (e.g., CD- A DVD (Digital Versatile Disc), magneto-optical media (e.g., a floptical disk), a hardware device (e.g., ROM, RAM, or flash memory, etc.). Also, the program instructions may include machine language code such as those generated by the compiler, as well as high-level language code that may be executed by a computer using an interpreter, etc. The hardware device described above may be implemented as one Or software modules, and vice versa.

The present invention has been described in detail with reference to the preferred embodiments shown in the drawings. These embodiments are to be considered as illustrative rather than limiting, and should be considered in an illustrative rather than a restrictive sense. The true scope of protection of the present invention should be determined by the technical idea of the appended claims rather than the above description. Although specific terms are used herein, they are used for the purpose of describing the concept of the present invention only and are not used to limit the scope of the present invention described in the claims or the claims. Each step of the present invention need not necessarily be performed in the order described, but may be performed in parallel, selectively, or individually. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It is to be understood that the equivalents include all components that are invented in order to perform the same function irrespective of the currently known equivalents as well as the equivalents to be developed in the future.

Claims (17)

Analyzing a real image obtained in real time by a user terminal to identify a first image region matching a predetermined first criterion;
Displaying a pre-stored messenger image visually above the identified first image area while displaying the obtained real image through a display of the user terminal; And
And generating a conversation message including a voice obtained by the user terminal and a messenger image displayed through the display while the real image and the messenger image are displayed together.
The method according to claim 1,
Wherein the generating the conversation message comprises:
And generating the conversation message including characteristic information of a messenger image displayed on the display during a time when the voice is acquired by the user terminal.
3. The method of claim 2,
Wherein the displaying the messenger image comprises:
And rendering and displaying the messenger image in three dimensions based on characteristic information of the messenger image.
The method of claim 3,
Wherein the displaying the messenger image comprises:
And adjusting and displaying the size of the pre-stored messenger image corresponding to the size of the identified first image area.
The method of claim 3,
Wherein the generating the conversation message comprises:
And modulating the acquired voice.
The method of claim 3,
Wherein the generating the conversation message comprises:
And setting a conversation message output condition including a reality image reference to which a messenger image included in the conversation message is to be output.
The method of claim 3,
Analyzing a real image obtained in real time by the user terminal when the conversation message is received, and identifying a second image region matching a predetermined second criterion;
And displaying, via the display, the messenger image on the visual upper part of the second image area based on the messenger image characteristic information included in the conversation message in the identified second image area while displaying the obtained reality image Displaying; And
And outputting the voice included in the conversation message together with the messenger image.
8. The method of claim 7,
Wherein the predetermined second criterion includes a match rate with a predetermined shape being equal to or greater than a reference value.
The method of claim 3,
Wherein the generating the conversation message comprises:
And generating the conversation message further including content purchased by connecting to the content server.
10. The method of claim 9,
Wherein the generating the conversation message comprises:
Generating the purchase content identifier based on recipient or sender information of the conversation message; And
And providing the purchased content identifier to the user terminal.
11. The method of claim 10,
Authenticating the purchaser through authentication information provided through the user terminal; And
Further comprising: requesting settlement of the content based on the authentication result and in accordance with the payment information associated with the settlement information.
11. The method of claim 10,
When a message output condition including a reality image reference in which the messenger image is displayed is set for the conversation message received and received,
And displaying information indicating the message output condition through a user terminal.
Analyzing a real image obtained in real time by a first user terminal to identify a first image region matching a predetermined first criterion;
Displaying a messenger image together visually above the identified first image area while displaying the obtained reality image through a display of the first user terminal;
Acquiring voice through the first user terminal while the messenger image is displayed;
Setting a chat message output condition for outputting a chat message including the messenger image and voice;
Generating the conversation message including the characteristic information of the messenger image, the acquired voice, and the conversation message output condition;
Receiving a conversation message from a second user terminal; And
Determining whether the second user terminal satisfies the dialog message output condition, and outputting the conversation message.
14. The method of claim 13,
Wherein the receiving of the conversation message by the second user terminal comprises:
And receiving a chat message output condition from the chat message first.
15. The method of claim 14,
Further comprising the step of the second user terminal outputting information indicating the conversation message output condition through a display.
15. The method of claim 14,
Wherein the receiving of the conversation message by the second user terminal comprises:
And requesting and receiving the entire conversation message if it is determined that the conversation message output condition is satisfied.
17. A computer-readable storage medium for performing the method according to any one of the preceding claims.

Images