KR100853122B1 - Method and system for providing Real-time Subsititutive Communications using mobile telecommunications network - Google Patents

Abstract

The present invention relates to an alternative video service method and system using a mobile communication network, which can provide a higher quality video call service by analyzing voice and video of a subject in real time during a video call and replacing it with a character. It relates to an alternative video service method and system used.

Alternative video service system using a mobile communication network of the present invention, a video processor for performing a real-time face tracking function for extracting the change of the feature point information of the image by analyzing the received video video signal, and the audio information by analyzing the received audio signal An RSC server including an audio processor configured to perform a real-time lip-sync function for extracting a video signal, and an RSC encoding module for generating a replacement video which is a video in real time by combining the video feature information and audio information with a character content; It features.

Mobile network, real time, alternative video

Description

Method and system for real-time alternative video service using mobile communication network {Method and system for providing Real-time Subsititutive Communications using mobile telecommunications network}

1 is a block diagram of a video production system according to the prior art.

2 is a block diagram of an alternative video service system according to a first embodiment of the present invention;

3 is a block diagram of an alternative video calling service platform according to a first embodiment of the present invention;

4 is a configuration diagram of an alternative video call server according to a first embodiment of the present invention.

5A and 5B are exemplary diagrams of an input image and a character image on which feature points are used for a substitute image service according to the present invention.

6 is a conceptual diagram of an alternative video service according to the first embodiment of the present invention;

7 is a call processing flowchart of an alternative video service method according to the first embodiment of the present invention.

8A and 8B illustrate an alternative video service method between a 2G terminal and a 3G terminal according to the first embodiment of the present invention.

9 is a view illustrating a configuration of a mobile communication terminal for an alternative video service according to a first embodiment of the present invention.

10 is a configuration diagram of an alternative video call service system according to a second embodiment of the present invention.

11 is a block diagram of a mobile communication terminal for an alternative video service according to a second embodiment of the present invention.

12 is a call processing flowchart in an alternative video service method according to a second embodiment of the present invention.

13 is a call processing flowchart of an alternative video service method according to a third embodiment of the present invention.

The present invention relates to an alternative video service method and system using a mobile communication network, which can provide a higher quality video call service by analyzing voice and video of a subject in real time during a video call and replacing it with a character. It relates to an alternative video service method and system used.

The substitute image refers to an image that replaces the actual image of the subject, such as a character (or an avatar). The substitute image is widely used in the fields of computer animation, mobile communication, and broadcasting. Many attempts have been made to provide alternate images similar to the images.

Korean Patent Application No. 10-2000-0055309, filed Sep. 20, 2000, describes a three-dimensional video authoring tool that supports motion, facial expression, lip-syncing, and lip-synced speech synthesis. Production system and method "is an invention related to a video production system having a lip sync function for matching the movement of the lips of an output voice and a substitute image, and has a configuration as shown in FIG.

The video production system 100 of FIG. 1 includes a voice information conversion engine 110 that performs a text to speech (TTS) function for converting text information input by a user into voice information, and each phoneme included in the voice. The phoneme information of the voice library 130 composed of a database of a plurality of face shapes set according to a mouth shape of a corresponding character and a predetermined facial expression model, and each phoneme information of the initial, neutral, and final characters inputted from the voice information conversion engine 110. The lip-sync generating engine 120 to change the mouth shape and the face shape of the character to be displayed on the screen by extracting and matching the mouth shape and the face shape of the character stored in the voice library 130 and the head that the character can take. The motion library 150 includes a database of character motion information related to the movement of the body, such as arms and legs, and the motion information stored in the motion library 150. Combining the motion of the character body generated by the animation creation engine 140 with the animation generation engine 140 that sets the motion of the chur character body, the mouth shape, the face shape of the character generated by the lip-sync generation engine 120, and the animation generation engine 140. It consists of a synthesis engine 160 for outputting in the form of a video.

By the way, the invention of the prior art of Figure 1 can change the shape of the mouth of the character in accordance with the phoneme of the input voice, but the face shape of the character and the motion of the body of the character is not linked to the phoneme of the input voice, so in advance Since the user's facial expressions and motions are combined with the shape of the mouth, the overall facial expression and motion of the character is not incongruous, and the actual image is input in one file unit, and the character image is generated in one file unit. Since it relies on the sequential method of joining two files together to create a video, the creation of a natural video character in which the facial expression of the character and the motion of the character are changed in real time according to the movement of the actual video input in the form of a video stream and an audio stream is achieved. It is difficult, so real-time communication is essential It is not adequate to replace the image used to call in behalf of the trust.

There are a number of prior arts that apply a technique similar to that of the prior art of FIG. 1 to a mobile communication network, but these are all merely insisting that the offline character generation method as shown in FIG. 1 can be simply applied to a mobile communication network. Therefore, there is no example in which a real-time character generation and replacement image providing method capable of combining a video stream and an audio stream input in real time through a mobile communication network in real time with a character image is provided.

In fact, off-line computer graphics programs such as 3D MAX, MAYA, etc., target VGA-level images with a resolution of 640 * 480 by default. To display full-color images, 3 bytes per pixel ( = bit) data amount, so the data amount of one screen is 640 * 480 * 24 = 7372800bit = 921Kbyte. There is no difficulty in processing the data using a workstation or a PC offline, but a problem occurs when the same image data is exchanged using a mobile communication network.

For example, when the video data is transmitted at a transmission rate (64 Kbps) supported by the MPEG4 compression method applied to a WCDMA network that supports a video call, t = 640 * 480 * 24/64000 = 115.2 (s). Since it takes less than about 2 minutes, it is virtually impossible to apply to a mobile communication network.

After all, in case of mobile communication, due to the limitation of transmission speed and transmission bandwidth, both video signal and audio signal should be compressed and transmitted at a high rate within the allowable range of each mobile communication network. Extract the feature points from the image included in the continuous video signal input in the form of, and extract only the voice of the caller from the compressed audio signal, and synthesize the extracted data with the character for the substitute image in real time (that is, There is a need for a quick and simple system and method capable of generating an overall latency of less than one second.

In addition, in the case of mobile communication, the data transmission state is often changed even during a call due to the movement of a user, the appearance of radio wave obstacles, the presence of a radio wave shadow area, and so on. do.

Furthermore, in order to realize the alternative video service provided in real time, a method and a system capable of generating and transmitting a replacement video in a simpler and more rapid manner should be devised by utilizing the characteristics of mobile communication only.

In view of the above problems, the present invention analyzes voice and video of a subject in real time during a video call and replaces it with a character in order to provide a service optimized for a mobile communication network in which the characteristics of real-time communication are important. An object of the present invention is to provide an alternative video service method and system using a mobile communication network, which can provide a video call service.

In addition, the present invention provides a service optimized for a mobile communication network in which the data transmission status changes rapidly according to the communication environment, the audio signal and the video signal complementary to each other according to the quality of the audio signal and the video signal It is an object of the present invention to provide an alternative video service method and system using a mobile communication network that can continuously generate and transmit an alternative video despite a change in a communication environment.

The present invention also provides a mobile communication network capable of generating and transmitting a substitute image having a higher transmission power by generating a substitute image based on a video signal and supplementing the mouth shape of the substitute image with a standardized mouth shape based on the audio signal. An object of the present invention is to provide an alternative video service method and system.

In addition, an object of the present invention is to provide an alternative video service method and system using a mobile communication network capable of alternative video call in the case of a hearing impaired by generating and transmitting an alternative video with the mouth shape according to the pronunciation.

In addition, the present invention provides a simpler and faster alternative image by generating personal metadata learned for each mobile communication user in consideration of the characteristics of the mobile communication in which one mobile communication terminal is allocated almost exclusively to one mobile communication user. An object of the present invention is to provide an alternative video service method and system using a mobile communication network, which can be generated and provided.

In addition, an object of the present invention is to provide an alternative video service method and system using a mobile communication network, which can confirm the transmission status of its own alternative video delivered to the other party in the alternative video call.

Another object of the present invention is to provide an alternative video service method and system using a mobile communication network, which can provide an alternative video service even when a call is made with an existing 2G terminal which is not provided with a video communication function.

In addition, it provides a real-time alternative video service method and system having a structure that can be applied to the third generation communication environment that can interoperate various communication networks, such as WCDMA, HSDPA, IP-TV, wired telephone network, Internet network and more advanced communication environment For the purpose of

The above and further objects and effects of the present invention will be more clearly understood in the detailed description of the construction of the invention detailed below.

In order to achieve the above object, the alternative video service system using the mobile communication network of the present invention includes a video processor for performing a real-time face tracking function to analyze the received video signal and extract the change of the feature point information of the image, and the received audio An RSC server including an audio processor configured to perform a real-time lip sync function by analyzing a signal and extracting voice information, and an RSC encoding module configured to generate a substitute image which is a video in real time by combining the image feature information and voice information with character content; It is characterized in that connected to the mobile communication network.

The video processor may include a video decoder that decodes the received video signal, and an RFT engine that extracts a change of contour feature points and individual feature point information of a reconstructed image included in the decoded video signal. And a face analysis module that determines whether the restored image is a human face image by comparing feature point information of the restored image with feature points of basic contours of a human face, and when the restored image is determined as a human face image. A face tracking module for tracking changes in individual feature points of the face in real time, and a motion tracking module for tracking feature point information of the contours of the face in real time when the reconstructed image is identified as a human face image. Of the feature points tracked by the tracking module and the motion tracking module. Screen is characterized in that the transmission to the RSC encoding module as the feature information of the character for replacing video services.

The audio processor may include an audio decoder that decodes the received audio signal, and an RLS engine that extracts a voice signal from the decoded audio signal and performs a real-time lip sync. The RLS engine filters the input audio signal. And an RTLS module for extracting a lip sync variable in real time by matching a mouth shape information database to a voice signal extracted from the voice analysis module and extracting only a signal having a predetermined frequency range as a voice signal. The variable may be transmitted to the RSC encoding module as character voice information specifying voice of the substitute video service character.

In addition, the RSC encoding module is configured to receive the character feature information and the character voice information and combine them with the character for the substitute image service to generate a substitute image in real time, and to render the generated substitute image; And a renderer and a video encoder for encoding and transmitting the rendered substitute image.

In addition, the mouth shape information database matched in the RTLS module is the mouth shape for all kinds of vowels, the changed mouth shape of the vowel according to the difference between the initial and the finality, the data on the process of changing the mouth shape according to the composition of each letter , And mouth shape data changed according to the frequency range of the voice signal.

In addition, the mouth shape information database matched by the RTLS module stores mouth shape data normalized to the voice of the reference frequency range set by the system and mouth shape data changed to the voice of the frequency range higher or lower than the reference frequency. The RTLS module is configured to extract a lip sync parameter according to the frequency range of the input voice signal when the frequency range of the voice signal extracted from the voice analysis module is different from the reference frequency range.

In addition, the RTLS module classifies the lip sync variable generated according to the voice parameter DNA included in the character voice information for each user terminal, repeatedly stores the data as personalized metadata, and learns the audio signal from the same user terminal in the future. The lip sync variable is extracted from the received audio signal using the learned personalized metadata as a default speech model upon reception.

The RSC encoding module may generate an outline and a face of the substitute image based on the image feature information, and correct a mouth shape of the substitute image by using a lip sync parameter included in the voice information. .

The RSC encoding module may generate a mouth shape of the substitute image based on the voice information, and generate a face portion of the substitute image except the mouth shape based on the image feature information.

In addition, the system interworks with the RSC server to perform call processing and replacement video service management for a replacement video call, and has an SGN server including replacement video service information including subscriber information and character information for each subscriber. It characterized in that it further comprises.

In addition, the system is connected to the user terminals using a WEB / WAP server, and performs the registration of the replacement image service and subscription of the character for the alternative video service requested from the user terminal, the alternative video service subscription and the alternative video service A CMS server for transmitting information on registration of a dragon character to the SGN server; And a storage device for storing the registered character.

In addition, the system MMSC for generating, storing and transmitting the multimedia message; And a message audio processor for performing a non-real time lip sync function by analyzing the audio signal included in the message stored in the MMSC and extracting the message voice information, and generating the video character in non real time by combining the message voice information with character content. And an LSS server having an alternative video message encoding module for transmitting the generated video character to the MMSC.

In the system, the RSC server, the LSS server, the SGN server, the CMS server and the storage device constitute an RSC service platform, and the RSC service platform further includes a media gateway for interworking with the mobile communication network. Characterized in that.

On the other hand, the alternative video service method according to the present invention using the alternative video system, the method comprising the steps of determining whether to subscribe to the alternative video service in the location register according to the request for the alternative video call call from the mobile terminal for the mobile communication; If the originating terminal is a terminal of an alternative video service subscriber, as a result of the determination, an alternative video call call is configured in the core network, and the RSC server activates a registered character for the calling terminal to prepare for an alternative video service. step; Generating a substitute image in real time by combining the image characteristic information and the audio information extracted from the video signal and the audio signal input from the calling terminal from the RSC server with the activated character; And performing a video call using a real-time substitute image by transmitting the substitute image to the called terminal through the core network in the RSC server.

Meanwhile, the mobile communication terminal for an alternative video service according to the present invention includes a camera unit for acquiring an actual image of a user, an audio unit including a microphone and a speaker for input / output of an audio signal, and a software for requesting an alternative video service. A key input unit including at least one of a key and a hot key, a display unit for displaying the real image and the substitute image, an RSC module for generating the substitute image, and a character and standard mouth shape information for the substitute image; A data memory for storing data, a program memory for storing a terminal operating program including a software program for performing the substitute image generating function, a transceiver for transmitting and receiving transmission signals, and the components At least one microcontroller to control the function of each component It characterized in that a control unit is configured to be composed of a processor.

The alternative video service method using the alternative video service mobile communication terminal may be performed in response to a request for a replacement video service using at least one of the soft key and the hot key of the mobile communication terminal. Generating an image; Determining whether to subscribe to a replacement video service at a location register of a mobile communication network according to the replacement video service request; And when the mobile communication terminal is a terminal of a substitute video service subscriber, performing a video call using a real-time substitute video by setting up a substitute video call call in the mobile communication network and transmitting the substitute video to a called terminal. Characterized in that it comprises a step.

Meanwhile, in the alternative video service system in which a part of an image processing part is implemented in a terminal according to another feature of the present invention, a video decoder for processing a video signal obtained from a camera, and a feature point of the actual image included in the video signal A telecommunication communication terminal configured to include an RSC module for performing character feature information extraction, including an RFT engine for generating character feature information for substitute images through extraction; And an audio processor connected to a core network of the mobile communication network and performing a real-time lip sync function to extract and extract voice information by restoring and analyzing an audio signal received in a compressed state from the mobile communication terminal, and character content for a substitute image. And an RSC server including an RSC encoding module for combining the voice information with character feature information for the substitute image to the mobile communication terminal to generate a substitute image which is a video in real time.

In addition, a method of performing an alternative video service using an alternative video service system in which a part of an image processing part is implemented in a terminal may be performed by the mobile communication terminal in response to a request for establishing an alternative video call call from the mobile communication terminal. Extracting the character feature information on the image; Determining replacement video service subscription information in the RSC server according to the replacement video call call setup request; In the RSC server, the registered character is activated according to the substitute video service subscription information, and the registered character is used for the voice information extracted by the audio processor and the character feature information from the mobile communication terminal. Generating a substitute image in real time by combining with a; And performing a video call using a real-time substitute image by transmitting the substitute image to a called terminal through the core network in the RSC server.

Hereinafter, with reference to an embodiment of the present invention shown in the accompanying drawings will be described in more detail the present invention.

2 is a block diagram of an alternative video service system according to a first embodiment of the present invention;

The alternative video service system of FIG. 2 uses a Wideband Code Division Multiple Access (WCDMA) network, which is one of 3G networks, as a core network, but uses a Session Initiation Protocol (SIP) transmission protocol used in IP (Internet Protocol) based video communication. Various protocols and codecs used in different communication networks such as SS7 protocol and H.324M protocol and H263 codec used in HSDPA (High Speed Downlink Packet Access) called H.323 codec technology and 3.5G mobile communication service. It can support the integrated technology and interwork with the media gateway 205 which can connect the time division multiplexing (TDM) traffic of the circuit network and the real-time transport protocol (RTP) traffic of the packet network for real-time transmission of data. Between 3G mobile users and landline video telephone users, and between 3G mobile users and web video telephone users. It is composed to support alternative video call service through heterogeneous communication network between 3rd generation communication environment that can interoperate various communication networks such as WCDMA, HSDPA, IP-TV, wired telephone network, Internet network and more advanced communication environment. Also has an applicable structure.

Therefore, even when using a different type of core network than that shown in FIG. 2 or interworking with other types of communication networks, the same or equivalent components as those of the RSC service platform 300 for performing an alternative video service function of the present invention are used. It is clear that the alternative video service system can be configured.

In FIG. 2, components of a typical WCDMA network include a plurality of mobile communication terminals 201 connected to a mobile communication network, a Node-B 202 serving as a base station, an RNC 203 serving as a controller, and an MSC 204 serving as an exchange. And an HLR / HSS 206 serving as a location register, an MMSC 207 serving as a multimedia message center, an AuC 208 serving as an authentication server, and a G / W 205 serving as a media gateway for interworking with heterogeneous communication networks.

Meanwhile, in order to provide an alternative video service according to the present invention, an RSC service platform 300 is added for an alternative video service, and the G / W 205 interworks with the RSC service platform 300 and a heterogeneous communication network. A content provider (CP) 404 and user terminals 401 and 402 are connected to each other through a heterogeneous communication network such as a WEB / WAP server 408.

Here, the mobile communication terminal 201 is a concept including all 4G terminals available in the future in addition to the 2G, 3G terminal, Node-B (202) is a base station for connecting the communication channel to the mobile communication terminal 201, RNC (Radio Network Controller) 203 is a controller that controls a plurality of Node-B (202) and interwork with the switch of the core network, MSC (Mobile Switching Center) 204 connects the RNC 203 with the core network An exchange that switches calls.

G / W (Media GateWay) 205 is a device for converting the format of the traffic format according to the communication network. In the packet voice delivery network, TDM traffic of the circuit network is transferred to the Asynchronous Transfer Mode (ATM) or IP traffic of the packet network. Trans-coding function to convert and signal switching function for call control.

The HLR / HSS (Home Location Register / Home Subscriber Server) 206 manages mobile subscriber information (ie, location information, authentication information, service information, authority and additional information, etc.) in real time, and the MSC / VLR 204. And additional service setting status such as incoming / outgoing, authentication, multimedia message, packet transmission, location information, and alternative video service through interworking with, AuC 208, MMSC 207, and RSC service platform 300 Provide other intelligent network services.

The AuC (Authentication Center, AC) 208 provides service subscriber authentication and wireless password using authentication keys and algorithms, and performs operations in conjunction with the HLR / HSS 206.

MMSC (Multimedia Message Service Center 207) is configured to include the functions of Short Message Service Center (SMSC), provides voice message, short message, multimedia message service, and any form including Internet e-mail connected to core network It also provides a message service to accommodate multimedia messages.

Real-time Substitutive Communications (RSC) service platform 300 is a core component provided for implementing the alternative video call of the present invention, and SGN (Signal) for call processing and service management for the alternative video call. Gateway Network (304) server, RSC server 306 for generating alternate video in real time, Lip-Sync Service (LSS) server 308 for non-real-time alternate video service, character content for alternate video and service subscriber Storage 307 for storing various data including service contents and learned personal metadata, and CMS (Contents Management Service) server 406 for character content management such as character registration and character template conversion. It is configured to include). The configuration and operation of the RSC service platform 300 will be described later in detail with reference to other drawings.

In order to provide an alternative video service, CP 404, which is a content provider, or users can register various character contents that can be used as a basic model of the alternative video to the CMS 406 through the WEB / WAP 408 server and store the storage device. 307, and users subscribe to the alternative video service using the mobile terminal 201, the wireless terminal 401 such as a wireless Internet terminal, or the wired terminal 402 such as a PC. And character setting.

3 is a block diagram of an RSC (alternate video call) service platform according to a first embodiment of the present invention.

First, the RSC server 306 restores and analyzes the received video signal to perform a real-time face tracking (RFT) function to extract a change in the feature point information of the image. An audio processor performing a real-time lip-sync (RLS) function for reconstructing and analyzing voice information, and image characteristic information from an RSC engine module to a character selected from a content DB of a storage device 307; And an RSC encoding module for combining the voice information to generate a video character.

SGN server 304 is a component that manages and manages alternative video service applications. The SGN server 304 includes a device driver, including an RSC server 306, a Contents Management Service (CMS) server 406, a storage device 307, and a Lip-Synce Service server 308. The other components of the RSC service platform 300 are controlled, and information set by the service subscriber in relation to the alternative video service (type of the alternative video character (ie, character ID), transmission time, method, mode, etc.) ) And controls the other components to perform the alternative video service according to the set service subscriber information.

Information on whether the caller has subscribed to the alternative video service and information on the terminal (eg, whether the terminal is an alternative video service) is stored in the HLR / HSS 206 to be used for call processing to the RSC service platform 300. Subsequent video service control after checking whether a subscription is made and the terminal type is performed by the SGN server 304. The ID of the character content to be used for the replacement video service is also transmitted by the SGN server 304 to the RSC server 306, which enables the RSC server 306 to identify each character.

The CMS server 406 registers the character content provided by the content provider or the substitute video service user and service environment information (eg, the type of the character for the substitute video, the transmission time, the method, the mode, etc.) set by the service subscriber. In addition, the registered service environment information is transmitted to the SGN server 304 so that the SGN server 304 can control the alternative video service. In addition, the CMS server 406 edits the template of the character content provided by the content provider, converts it into a form that can be applied in real time in the RSC server 306, and stores the converted character content in the storage device 307. Do this.

The LSS server 308 is a text or voice message generated and transmitted by the MMSC 207 at the request of the user terminal, or if there is no real-time call between the calling terminal and the called terminal (for example, power off the terminal, out of call rights, A non-real time alternative video service function that provides a substitute video multimedia message (MMS) by combining a character with a text or voice message generated and transmitted by the MMSC 207 by another additional service that automatically generates a message during a call). Do this. The LLS server 308 does not perform a real-time substitute image generation function because it combines the characters in the pre-recorded or generated by the TTS voice file.

The LSS server 308 may include a message audio processor (not shown) and an MMS encoding module (not shown) corresponding to the audio signal processor and the RSC encoding module 350 among the components of the RSC server 306 of FIG. 4 to be described later. And a TTS engine (not shown) for converting the input message into a voice message when the input message is a text message.

The message audio signal processor of the LSS server 308 performs a non-real-time lip sync function to extract message voice information by analyzing a message audio signal included in a message stored in the MMSC 207, and the MMS encoding module is an SGN server 304. And a character image set by the user or set in the system in conjunction with the storage device 307, and combined with the message voice information from the message audio signal processor to generate an alternative image having a mouth shape determined according to the message voice information. It performs the function of transmitting to the MMSC 207.

The storage device 307 provides all the character content registered in the CMS server 406 and transmits the character content to the RSC server 306 and the LSS server 308 under the control of the SGN server 304. do.

4 is a block diagram of an alternative video call (RSC) server according to a first embodiment of the present invention.

First, the RSC server 306 may include a video signal processor including a video decoder 310 and an RFT engine 320, an audio signal processor including an audio decoder 330 and an RLS engine 340, and an RSC encoding module ( 350).

The video decoder 310 decodes the video signal received through the mobile communication network and transmits the decoded video signal to the RFT engine 320.

The RFT engine 320 includes a face analysis module (FA) 322, a face tracking module (FT) 324, and a motion tracking module (MT) 326. It is composed.

The face analysis module 322 prestores feature point information on a basic outline of a human face using its own memory (not shown), and extracts and stores feature points forming the outline of the image input from the video decoder 310. It is checked whether the input image is a human face image by comparing with basic contour feature point information.

In this regard, FIG. 5A illustrates an input image and a character image on which contour feature points are displayed.

The left image of FIG. 5A is an actual image included in a video signal transmitted from a mobile communication terminal, and the right image is a character image generated by the RSC server 306. Both images show feature points (and connecting lines) for the basic contours of the human face (ie, relative positions of the head, eyebrows, eyes, nose, mouth, etc.), and many contour feature points (and connecting lines of the contour feature points) are replaced. It is used to quickly determine whether the input image is a face image of a person in order to determine whether to perform a video service.

As a result of determining the contour feature point in the face analysis module 322, when the input image is not a human face, the system may be configured not to provide a substitute image using a default character or to provide a substitute image service.

4, if the face analysis module 322 determines that the input image is a human face, the face tracking module 324 tracks the change of the individual feature points of the face portion of the input image, that is, the movement state in real time. The motion tracking module 326 tracks the contour feature point change of the input image in real time.

In this regard, FIG. 5B illustrates an input image and a character image in which individual feature points are additionally displayed.

In the left image, in addition to the contour feature points, individual feature points (ie, specific shapes such as head, eyebrows, eyes, nose, mouth, and ear) representing the features of the human face included in the input image are additionally displayed. On the right is a character image generated by reflecting the relative feature points and the relative positions of the individual feature points. The change of the facial expression of the head of the input image, etc., causes a change in the position of the contour feature point and the individual feature point, and the change in the position is tracked by the face tracking module 324 and the motion tracking module 326 in real time, respectively. Is reflected.

In the present invention, a plurality of individual feature points are added to each part to express specific shapes such as eyebrows, eyes, noses, mouths, and the like of the input image, thereby generating a character that reproduces the characteristics of the input image as it is.

4, the change in the feature points tracked and provided by the face tracking module 324 and the motion tracking module 326 is RSC encoding module 350 as the character feature information used to determine the motion and expression of the character for the substitute image. Is passed to the RSC animation engine 352).

Meanwhile, the audio decoder 330 decodes the input audio signal and provides the decoded audio signal to the RLS engine 340.

The RLS engine 340 may include a voice analysis module 342 for extracting a voice signal from an input audio signal, a real-time lip-sync module 344 for performing a real-time lip sync, and And a Voice Disguise Module 346 for modulating the lip-synced speech signal.

First, the voice analysis module 342 filters and removes signals in the remaining bands except for signals having a frequency corresponding to a predetermined voice band set in the present invention, in order to extract only voice from the input audio signal. Amplify.

That is, in general, the audible frequency of a person is about 20 to 20,000 Hz, and the frequency of the voice used for normal conversation is about 100 to 5,000 Hz. However, even when processing only a voice signal of about 300 to 3,400 Hz band when using mobile communication, Filtering only 300 ~ 3,400Hz signal considering the phonetic characteristic that more than% clearly recognizes the other party's voice.

The frequency band of 300 to 3,400 Hz may include other types of audio signals other than voice, but audio characteristics such as tone, tone, pattern regularity, and the like are distinguished from voice, thereby using the voice analysis module 332. In the present invention, only a voice signal except for a frequency having a different audio characteristic from a preset voice pattern is separated.

Most of these speech extraction methods utilize mobile communication characteristics that mainly process speech signals with amplitudes and frequencies within a limited range. Only voice signals can be extracted.

Subsequently, in the state in which the mouth shape according to most of the pronunciations used in the phone call is stored in the storage device 307 as a mouth shape database, the real-time lip sync module 334 is applied to the voice signal acquired by the voice analysis module 332. By extracting the speech parameter DNA unique to each included voice lip synch variable according to the individual voice pattern is extracted, the extracted lip synch variable is matched to correspond to one of the mouth shape stored in the mouth shape database.

In order to extract the correct lip sync parameters, the mouth shape stored in the mouth shape database includes the mouth shape for all kinds of vowels, and the changed mouth shape of the vowels due to differences in initial and final vowels (e.g. " In the case of "ga" and "wave", the state of change of mouth shape is different), and the process of changing the shape of mouth according to the characteristics of Yin, etc. must be considered. Haji includes pronunciations used in conversations) and key words and the process of changing them in an experimental way.

In addition, the mouth shape database includes mouth shape data changed to correspond to pronunciation higher or lower than the reference frequency in addition to mouth shape data standardized to correspond to each pronunciation belonging to a predetermined reference frequency range set by the system.

Thus, if, for example, the frequency range of the extracted speech signal is out of the reference frequency range due to the difference of the individual speech parameter DNA, the real-time lip sync module 344 matches the changed mouth shape data corresponding to the frequency range of the extracted speech signal. By doing this, it is possible to extract the lip sync parameters that can express the shape of the mouth more accurately.

Meanwhile, the lip sync variable extracted from the real time lip sync module 344 is sent to the storage device 307 under the control of the SGN server 304, and stored in a data field corresponding to the corresponding mobile communication terminal 201 to personalize it. Accumulated metadata, and thus, learning data about voice characteristics of each user is constructed, and personalized meta data instead of a lip sync variable corresponding to a standard voice in an alternative video call service using the same mobile communication terminal 201 in the future. Lipsync parameters based on the data are directly used as the default speech model for more accurate and faster mouth shape matching.

That is, the voice parameter DNA representing the unique characteristics of the input voice signal is extracted and stored separately as individual voice pattern metadata, and the unique characteristics of the corresponding voice signal are learned through repeated storage for a predetermined period or a predetermined number of times. By using this, it is possible to apply animation sinks according to individual voice characteristics by using learned metadata without standardizing morphing at the time of animation generation.

For example, mouth shape data applied when using an alternative video service for the first time is matched with a voice in a reference frequency range, but a user having an audio frequency range outside the reference frequency range performs an alternative video call for several seconds or more. In this case, since the lip sync variable representing the changed mouth shape data corresponding to the frequency range of the extracted speech signal is extracted and accumulated as personalized metadata, the corresponding user is not the reference frequency range in the future when the same video call of the same user is performed. You can instantly match changed mouth shape data to your voice frequency range.

The voice modulation module 346 stores various kinds of voices (for example, males, females, entertainers, bright voices, depressed voices, etc.) that vary in frequency band, tone, and intonation into a character voice database. It can be used as the character's voice instead of the caller's actual voice.

The lip sync parameters extracted from the real-time lip sync module 334 and optionally the character modulated voice variables provided by the voice modulation module 346 are transmitted to the RSC encoding module 350 as character voice information for determining the mouth shape and voice of the character. Is sent.

The RSC encoding module 350 generates a video by combining character feature information and character voice information from the RFT engine 320 and the RLS engine 340 with the character content and mouth shape data selected from the content DB of the storage device 307. For this, the RSC animation engine includes a Realtime Subsitutive Communications Animation Engine (352), an Image Renderer (354), and a Video Encoder (356).

Because different people have different pronunciation characteristics, different callers have slightly different mouth shapes for the same pronunciation, and even the same person may not only pronounce the same pronunciation with different mouth shapes, but also mouth shapes for different pronunciations. This can be almost similar. This phenomenon means that the shape of the mouth extracted from the face tracking method cannot accurately predict the voice.

Accordingly, the RSC animation engine 352 of the present invention uses the character feature information input from the face tracking module 324 and the motion tracking module 326, and the character input from the real-time lip sync module 344 and the voice modulation module 346. The voice information is combined in real time to generate a character video including the voice.

One method for generating a character video according to the present invention is a character set by the user among characters included in the character database of the storage device 307 in the RSC animation engine 352 (if there is no user's setting, a predetermined value is specified in advance). A default character), a feature image included in the character image information received from the real-time face tracking engine 320 is generated to generate a substitute image for transmission, and then according to the character voice information received from the real-time lip sync engine 340. The mouth shape data is retrieved from the storage device 307 to correct the character mouth shape of the replacement image for transmission. This correction is performed by adjusting the character mouth shape points of the substitute image to the mouth shape data.

Alternatively, another method for generating a character video, the mouth shape data according to the set character and the character voice information received from the real-time lip sync engine 340 is retrieved from the storage device 307 and the RSC animation engine 352 After synthesizing and generating a substitute image for transmission, character feature points included in the character image information received from the real-time face tracking engine 320 are reflected (except the mouth shape part) to compensate for the expression of the character of the transfer image for transmission. . At this time, the character feature points of the mouth shape included in the character image information are discarded without being used for generating the replacement image.

According to the above method, since the mouth shape portion of the character image included in the character image information is replaced with the mouth shape provided according to the character voice information, for example, in the case of "ㅏ" pronunciation, the character has the correct mouth corresponding to the "ㅏ" pronunciation. Allow them to express their shape.

In addition, in relation to the mouth shape complementary feature, the RLS engine 340 may apply a standard mouth shape generated by using standardized mouth shape data corresponding to a reference frequency range for each phoneme to the replacement image. In this case, the mouth shape of the substitute image is optimized and displayed in the content of the voice information in spite of the difference of the mouth shape of the individual user, and in particular, those who accept the information depending on visual information such as the shape of the mouth, such as a deaf person It has the effect of making the service available.

On the other hand, in the above case, it is assumed that the mobile communication environment is good, but in the case of the actual mobile communication environment, especially the mobile communication environment in the city, the communication signal transmission environment is constantly changed due to the movement of the caller and the appearance of obstacles. Will change. Therefore, in the situation where the quality of the video signal input to the video decoder 310 is excellent, the character image is generated based on the character image information. In the situation where the quality of the audio signal input to the audio decoder 330 is relatively excellent, the character voice is generated. It is preferable to generate a character image based on information.

To this end, the RSC animation engine 352 receives information (QoS) on the signal quality from the video decoder 310 and the audio decoder 330 to determine whether the video signal and the audio signal exceed the respective quality criteria, and the video Means (not shown) for determining a relatively good signal between the signal and the audio signal may be provided. Of course, when there is QoS data provided from other components in the mobile communication network, the signal determination means may not need to be separately provided.

The signal judging means may distinguish whether the signal is deteriorated into a temporary loss and a long term loss. Temporary loss is when a part of a frame of a video signal and an audio signal is lost, and a long-term loss is a case where a call is blocked or a signal reception is bad for a long time due to a change of communication environment.

When a temporary loss occurs in some frames of a video signal and an audio signal, the RSC animation engine 352 generates a character image and a mouth shape based on a signal in which no loss occurs, thereby generating a substitute image regardless of the temporary loss of the signal. Make sure to continue to deliver. For example, when the data of some sections of the video signal is lost, the shape of the mouth of the character image of the corresponding section is supplemented by using character voice information. When the data of some sections of the audio signal are lost, the characters of the corresponding section are lost. The mouth shape of the image is supplemented by using character image information.

On the other hand, when the signal reception is poor for a long time, if the audio signal is consistently below the quality standard, the character image and mouth shape are generated based only on the video signal, and if the video signal is continuously below the quality standard, Character shape and character image are generated based on the audio signal. As described above, in order to generate a character image based on an audio signal, a feature point distribution of a facial expression suitable for each pronunciation must be prepared in advance in a database.

In addition, the RSC animation engine 350 includes synchronization means (not shown) for synchronizing the video signal with the audio signal to generate a substitute image. The synchronizing means of the RSC animation engine 352 periodically synchronizes the character image and the character voice, and generates a synchronizing signal even when image complement is performed due to the variation of the quality of the video signal and the audio signal as described above. And character voice.

Subsequently, the image renderer 354 compiles the character image data to reduce the data amount and simultaneously synchronizes the character image and the character voice to perform the function of optimizing the movement of the character image to the most natural pattern. In addition, the image renderer 354 may apply various known image processing methods, such as an image interpolation method that naturally interpolates a sudden change of character image information.

Finally, the video encoder 356 encodes and transmits the character replacement video in an appropriate video format supported by the mobile communication network.

6 illustrates an alternative video service sequence according to the first embodiment of the present invention.

First, in the content uploading step (1), the CPs 404 register various character contents in advance in the CMS server 406. Even users who subscribe to the alternative video service may register their own character content with the CMS server 406 using their wired / wireless terminals 201, 401, 402.

Subsequently, in the (2) content purchasing and setting step, the service environment set by the user is registered in the CMS server 406 as the user subscribes to the alternative video service using his or her wired / wireless terminals 201, 401, 402. .

Then, (3) in the content ID registration step, the CMS server 406 notifies the HLR / HSS 206 whether to subscribe to the alternative video service and the subscriber terminal information, and notifies the SGN server 304 of the service registration details. The user registers the ID of the content registered by the user in the RSC server 306 to prepare to perform the replacement video service.

Now, if (4) the alternative video service subscriber requests an alternative video call using his terminal in (4) call attempt step, (5) in the call setup request / response step, the originating MSC 204, HLR / HSS 206 ) And the SGN server 304 interwork with each other to determine whether to subscribe to a service, and then set up a call for an alternative video service. (6) In the alternative video call processing step, the RSC 306 generates a replacement video to receive an incoming MSC 204. (7) In the alternative video call connection step, the alternative video call is performed in real time between the callers 201 and 401 and the called party 201.

On the other hand, when a call with the called party is impossible or when the caller wants to generate a voice message or a text message and transmit it as an alternative video message, the MMSC (not shown) interworks with the SGN server 304 and the LSS server 308. An alternative video MMS is generated using the character video set by the caller who is an alternative video service subscriber and then transmitted to the called party's mobile communication terminal.

The alternative video call via the RSC server 306 is a real-time service that provides a substitute video by analyzing a video signal and an audio signal input during a call in real time, but the alternative video service using the LSS server 308 is generated in advance. There is a difference in that it is a non-real-time service that provides a substitute image by analyzing stored voice messages (or voice messages converted from text messages) on a file basis.

7 illustrates a call processing procedure in the alternative video service method according to the first embodiment of the present invention.

For the alternative video service of the present invention, the user subscribes to the service in advance and sets a character to use for the alternative video call (S310). In this step, the user may upload a character written by himself and set it to use for an alternative video call.

Next, the service subscriber makes a call to the other party by using the calling terminal for the alternative video call to request a replacement video call (S320). The alternative video call call establishment request is performed by entering a telephone number of the called terminal with a specific service code or by entering a telephone number of the called terminal together with a softkey or a hotkey provided to the calling terminal.

Subsequently, an alternative video call setup request is transmitted through the outgoing MSC, and the HLR / HSS (not shown) connected to the outgoing MSC checks whether each terminal subscribes to the alternative video service and whether or not a video call is possible. If possible, the call establishment request is sent to the RSC platform via the originating G / W. The RSC platform performs preparation of an alternative video call such as user authentication and user setting character activation (S322), and then transmits a call setup request to the mobile communication network used by the called terminal.The alternative video call has passed through the called G / W and the called MSC. The receiving terminal receives the setup request (S324), and the call setup for the alternative video call is made by the alternative video call connection response (S328) of the called terminal (S328).

When the call setup is completed in the above step, a real-time substitute video call is performed between the calling terminal and the called terminal. When the calling terminal transmits the actual video through the sending MSC (S360), the calling G / W converts the video into a format that can be processed by the RSC platform (S330) and sends it to the RSC platform. A corresponding substitute image is generated (S340) and transmitted to the G / W of the called party's communication network, and the called G / W is converted into a media having a format appropriate for the communication method of the corresponding communication network (S350) and replaced by the called terminal via the recipient MSC. The video is transmitted (S370), and the called terminal also transmits the actual video of the user in response (S380), and the transmitted real video is converted into an alternative video on the RSC platform and transmitted to the calling terminal (S390). Video call is made.

The so-called media gateways such as outgoing G / W and incoming G / W shown in FIG. 7 are components for interworking heterogeneous communication networks, and include a transmission protocol and a media codec applied to each communication network, thereby transmitting them from the originating communication network. Converts the media data into media data that can be supported by the called party's communication network.

Therefore, when the alternative video system of the present invention is applied to a communication network that does not have such a media gateway in the core network, a means for converting the alternative video generated in the RSC platform into a format suitable for the communication network used by the called terminal should be provided to the RSC platform. do.

Meanwhile, during a normal call between the calling terminal and the called terminal, an alternative video call may be requested through a softkey or a hotkey provided in the terminal. In this case, an alternative video call preparation in the alternative video call setup section of FIG. 7 is prepared. Step S322 and all steps of the real-time substitute video call section are executed immediately.

Meanwhile, the present invention can be applied to a type of terminal (eg, 2G terminal) that does not support a video call, and FIGS. 8A and 8B illustrate a 3G terminal capable of a video call and a 2G terminal that does not support a video call. The alternative video service method of FIG.

First, FIG. 8A illustrates an alternative video service method between a 3G terminal of a caller who subscribes to an alternative video service and a 2G terminal of a called party who is not a subscriber. First, when the calling terminal requests an alternative video call, the HLR 206 managing the calling terminal checks whether the called terminal can make a video call in conjunction with the HLR of the called terminal, and when the called terminal is determined to be a 2G terminal, It is set to provide a caller-specific alternative video to the calling terminal, and provides an alternative video to the calling terminal in accordance with the voice input from the called terminal.

Meanwhile, FIG. 8B illustrates an alternative video service method between the caller's 2G terminal and the called party's 3G terminal subscribed to the alternative video service. The calling terminal cannot receive a substitute video to its own terminal, but can designate a substitute video of itself to be transmitted to the counterpart's terminal. First, when the calling terminal 2G requests an alternative video call, the HLR 206 managing the calling terminal confirms that the called terminal is a 3G terminal, and combines the caller's voice information with an alternative video character designated by the caller and replaces it. Create an image and send it to the called terminal.

In this case, when the called party is an alternative video service subscriber, the called party may designate to be represented as an alternative video of the other party of the character video set by the other party, regardless of the alternative video designated by the other party. In the case where the alternative video collides, it is preferable to give priority to the called party specified replacement video.

In short, an alternative video service subscriber having a terminal supporting a video call can designate both his / her own alternative video and an alternative video of the other party, and an alternative video service subscriber having a terminal not supporting a video call is transmitted to the other party. You can only specify your own replacement image.

9 illustrates a display configuration of an alternative video call terminal according to the first embodiment of the present invention.

In the alternative video call, the display 710 of the terminal displays a softkey 720 for requesting the alternative video call call setup in addition to the normal softkey. In addition, a hot key 730 for requesting an alternative video call may be separately provided on the keypad of the terminal.

In the first terminal screen of FIG. 9, a soft video 720 is used to perform an alternative video call setup request during a video call, and the alternative video is transmitted and displayed as the main video, and the real video is returned to the normal video call. The process of displaying the main image is shown.

Meanwhile, in FIG. 9, it should be noted that an image of the other party (which may be a real image or a substitute image) 712, a real image 713 of itself, and a substitute image 714 of oneself are simultaneously displayed on the terminal screen. . Since a conventional terminal is provided with an image input means such as a camera, it is common to express its own actual image. However, as shown in FIG. In addition to the technology of dividing the terminal, it is necessary to use a 3G or more terminal implemented with a technique for generating an alternative image in the terminal itself or a technique for allocating two outgoing and receiving channels to one terminal.

First, when a terminal is provided with two or more outgoing channels and a receiving channel, for example, a terminal used in a WCDMA network, which is a kind of 3G communication network, the dual channel technology is supported so that the caller is replaced by an RSC platform during an alternative video call. The video and the actual video (or replacement video) of the called party can be transmitted to the calling terminal through two channels, respectively, a voice traffic channel and a data traffic channel. Accordingly, as shown in FIG. All of his replacement images can be expressed.

Meanwhile, even when a substitute image is generated and transmitted by the terminal itself instead of generating and transmitting the substitute image in the RSC platform, as shown in FIG. This will be described in detail in the second embodiment of the present invention.

10 is a configuration diagram of an alternative video service system according to a second embodiment of the present invention.

10 shows that the RSC service platform is separated from the mobile communication network and implemented in the terminal 1000 of the mobile communication user. However, in the second embodiment, the CMS server 406 for content registration and management is provided in the mobile communication network. Different from the first embodiment, and the remaining configurations are the same to similar.

11 is a configuration of a mobile communication terminal equipped with an RSC module used in an alternative video service system according to a second embodiment of the present invention.

The RSC module-mounted mobile communication terminal 1000 of FIG. 11 includes a camera unit 1120 for acquiring an actual image of a user, an audio unit 1140 including a microphone and a speaker for input / output of an audio signal, and an alternative video service. A key input unit 1160 having a soft key and a hot key for request, a display unit 1180 for displaying actual images and a substitute image, an RSC module 1300 for generating a substitute image, and a character for a substitute image A memory 1140 including a data memory for storing data and a program memory for storing a terminal operating program including a software program for performing an alternative image generating function, and a transmitter (Tx) each responsible for transmitting and receiving a transmission signal. 1220 and a duplexer 1260 connected to a receiver (Rx) 1240 and an antenna to separate transmission and reception frequencies, and other components It is configured to include a control unit 1100 that is constituted by at least one microprocessor for controlling the functions of the components.

The RSC module 1300 corresponds to the RSC server of the first embodiment, and includes a video decoder 1310 and an RFT engine 1320 for video signal processing, an audio decoder 1330 and an RLS engine for audio signal processing. 1340 and character feature information from the RFT engine 1320 and character voice information from the RLS engine 1340 are combined with a set character stored in the memory 1140 to generate an alternative image, render it, and then transmit the mobile communication network. And an RSC encoding module 1350 for converting to a format suitable for bandwidth and transmission rate.

12 is a flowchart illustrating a call processing procedure in an alternative video service method according to the second embodiment of the present invention.

For the alternative video service of the second embodiment, the user subscribes to the service in advance, himself or herself creates a character to use for the alternative video call, or downloads it from the CMS server (or storage device connected to the CMS server) 406 An alternative video service subscription section (not shown) stored in the terminal must be preceded.

Next, a real-time substitute video call is performed between the calling terminal and the called terminal by an alternative video call call setup request from the calling terminal (S1322). At this time, since the calling terminal is a terminal equipped with an RSC module, an alternative video is generated by itself (S1320). ), Call setting is requested, and the calling party G / W converts the corresponding substitute video into a format suitable for communication between the G / W (S1324) and delivers it to the called party G / W. One substitute video is converted into a format suitable for its own communication network (S1326) and transmitted to the called terminal via the called MSC.

Subsequently, the called terminal receives a substitute image from the calling terminal (S1328), and correspondingly generates a substitute image of the called party in itself (S1330), and receives a substitute video call connection response signal including the generated substitute image of the called party. Transfer to the calling terminal while switching to the appropriate media format through the incoming G / W, outgoing G / W, outgoing MSC (S1332, S1334), and receiving and playing it at the calling terminal (S1338). do.

In the above process, the RSC service platform except the CMS server does not contribute to the alternative video call connection. However, when one of the two terminals is a 2G or 3G terminal that does not have an alternative image generation function, the RSC service platform intervenes to generate the alternative image and transmits the alternate image. The alternative video service is performed in the combined form of the example.

On the other hand, when the RSC module is implemented in the mobile communication terminal as in the second embodiment, since the generation of the replacement image proceeds regardless of the change in the mobile communication environment, it is unnecessary to use a signal quality discriminating means when generating the replacement image. Done.

In addition, in the second embodiment, as shown in FIG. 8, since the substitute image generated by the RSC module may be simultaneously displayed together with the actual image input to the mobile communication terminal, the user may transmit his / her actual image transmitted to the counterpart. It is possible to observe the representation of alternative video as well as the video, and to switch the transmission video to transmit the desired video among the actual video and the replacement video, and to use the repeated service to provide a smooth form of the real video. It is possible to determine the shape that is not used (e.g., the rotation angle of the head capable of generating the replacement video, the speed of movement, the distance between the terminal and the face), so that the alternative video service can be used more effectively by inputting the actual video in the appropriate form. do.

On the other hand, in the case of the second embodiment, the configuration for video signal processing, the configuration for audio signal processing, and the configuration for generating a substitute image are all mounted in the RSC module of the mobile communication terminal. As an intermediate configuration, only the video processing unit including the video decoder 1310 and the RFT engine 1320 of FIG. 11 is configured as the RSC module 1300 in the user terminal, and the audio decoder 1330, the RLS engine 1320, and the RSC. The encoding module may be implemented in an RSC server in the mobile communication network. The third embodiment relates to a case where the alternative image generation function is properly separated between the terminal and the server as described above.

13 is a flowchart illustrating a call processing procedure in an alternative video service method according to the third embodiment of the present invention.

In the case of the third embodiment, it is a matter of course that a preliminary procedure step of substituting an alternative video is performed.

Subsequently, when the mobile communication terminal requests the alternative video call call setup (S1210) and transmits the character feature information extracted using the video decoder and the RFT engine of the own terminal (S1205), the RSC server transmits the audio signal transmitted from the mobile communication terminal. Decodes and extracts the character voice information and combines it with the received character feature information and the set character to generate a substitute image, the generated substitute image is received at the called terminal via the destination G / W and the recipient MS (S1250).

Subsequently, the called terminal generates character feature information in the same manner (S1260), transmits an alternative video call connection response signal (S1270), generates an alternative video in the mobile communication network (S1290), and requires media switching (S1280, 1295). In this case, the replacement video call is performed by playing back the replacement video received at the calling terminal and receiving the receiving video (S1230).

In the third embodiment, since the mobile communication terminal processes the video signal and transmits only the character feature information to the mobile communication network, the limitation of the bandwidth and the signal transmission rate due to the transmission of the video signal itself can be greatly alleviated. Since the bandwidth for transmitting the audio signal is significantly smaller than the bandwidth required for the transmission of the video signal, the audio signal is not processed by the mobile terminal but transmitted to the mobile network as it is to be processed by the RSC server. Speed problems do not occur.

On the other hand, as a fourth embodiment, it can be assumed that both the video signal processing unit and the audio signal processing unit is implemented as an RSC module in the mobile communication terminal, in which case the RSC server includes an RSC animation engine, a renderer and a video encoder. It will consist only of the RSC encoding module. Those skilled in the art will be able to understand the features and advantages and disadvantages of the fourth embodiment through comparison with the first, second and third embodiments described above, and thus the description and description thereof will be omitted.

Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art to which the present invention pertains can implement the present invention in other specific forms without changing the technical spirit or essential features, and have been described above. The embodiments are to be understood in all respects as illustrative and not restrictive.

In addition, the scope of the present invention is specified by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents are included in the scope of the present invention. Should be interpreted as

According to the present invention, in order to provide a service optimized for a mobile communication network in which the characteristics of real-time communication are important, a higher quality video call service can be obtained by analyzing a voice and a video of a subject in real time during a video call and replacing it with a character. Provided is an alternative video service method and system using a mobile communication network.

In addition, according to the present invention, in order to provide a service of an optimized type for a mobile communication network in which the data transmission state changes rapidly according to the communication environment, the audio signal and the video signal are complementary depending on the quality state of the audio signal and the video signal. The present invention provides an alternative video service method and system using a mobile communication network capable of continuously generating and transmitting a replacement video despite a change in a communication environment.

In addition, according to the present invention, by generating a replacement image mainly on the video signal and by complementing the mouth shape of the replacement image with a standardized mouth shape based on the audio signal, it is possible to generate and transmit a replacement image with better transmission power, An alternative video service method and system using a communication network are provided.

In addition, according to the present invention, alternative video service method and system using a mobile communication network, which is possible to alternative video call in the case of the hearing impaired by generating and transmitting a replacement video mainly on the shape of the mouth according to pronunciation is provided.

Further, according to the present invention, in consideration of the characteristics of the mobile communication used in which one mobile communication terminal is almost entirely assigned to one mobile communication user, the replacement image is made simpler by generating personal metadata learned for each mobile communication user. Provided are an alternative video service method and system using a mobile communication network, which can be generated and provided quickly and quickly.

In addition, according to the present invention, there is provided an alternative video service method and system using a mobile communication network that can confirm the transmission status of its own replacement video delivered to the other party in the video call.

According to the present invention, there is provided an alternative video service method and system using a mobile communication network, which can provide an alternative video service even when a call is made with an existing 2G terminal which is not provided with a video communication function.

In addition, according to the present invention, a real-time alternative video service method having a structure that can be applied to the third generation communication environment that can interoperate with various communication networks, such as WCDMA, HSDPA, IP-TV, wired telephone network, Internet network and more advanced communication environment And a system is provided.

Claims (33)

In the alternative video service system using a mobile communication network, The video processor performs a real-time face tracking function for extracting video feature information representing a change in feature points of an image by analyzing a received video signal and a real-time lip sync function for extracting voice information by analyzing a received audio signal. An RSC server including an audio processor and an RSC encoding module which combines the video characteristic information and audio information with character content to generate a substitute video which is a video in real time, is connected to the mobile communication network, The audio processor includes an audio decoder that decodes the received audio signal, and an RLS engine that extracts a voice signal from the decoded audio signal and performs a real-time lip sync. The RLS engine filters the input audio signal and extracts only a signal having a predetermined frequency range as a voice signal, and matches the mouth shape information database to the voice signal extracted by the voice analysis module to generate a lip sync variable in real time. Contains an RTLS module to extract, The lip sync variable is transmitted to the RSC encoding module as voice information of the character for the substitute video service. The RTLS module classifies the lip sync parameters generated according to the voice parameter DNA included in the character voice information for each user terminal and repeatedly stores them as personalized metadata, and when receiving an audio signal from the same user terminal in the future And extracting the lip sync variable from the received audio signal using the learned personalized metadata as a default speech model. The method of claim 1, The video processor includes a video decoder for decoding the received video signal, and an RFT engine for extracting a change of contour feature points and individual feature point information of a reconstructed image included in the decoded video signal. The RFT engine may include: a face analysis module configured to compare the feature point information of the reconstructed image with the feature point information of the basic contour of the human face, and determine whether the reconstructed image is a face image of a person; and the reconstructed image is a face image of a person. Face tracking module for tracking in real time the change of the individual feature points of the face portion when determined as, and a motion tracking module for tracking the feature point information of the contour of the face portion in real time when the reconstructed image is determined as a human face image , The change of the feature points tracked by the face tracking module and the motion tracking module is transmitted to the RSC encoding module as feature information of a character for a substitute video service. delete The method of claim 1, The RSC encoding module may include: an RSC animation engine configured to receive the character feature information and the character voice information and combine them with the substitute image service character to generate a substitute image in real time, and a renderer to render the generated substitute image; And a video encoder for encoding and transmitting the rendered substitute image. The method of claim 4, wherein Mouth shape information database matched in the RTLS module is a mouth shape for all kinds of vowels, the changed mouth shape of the vowel according to the difference between the initial and the finality, data about the process of changing the mouth shape according to the composition of each letter, and Alternative image service system using a mobile communication network, characterized in that it comprises a mouth shape data that changes according to the frequency range of the audio signal. The method of claim 5, wherein The mouth shape information database matched by the RTLS module stores mouth shape data normalized to the voice of the reference frequency range set by the system and mouth shape data changed to the voice of the frequency range higher or lower than the reference frequency. And, if the frequency range of the voice signal extracted from the voice analysis module is different from the reference frequency range, extracting a lip synch variable according to the frequency range of the input voice signal. Service system. delete The method of claim 1, The RLS engine further includes a voice modulation module for modulating the voice signal lip-synced in the RTLS module, wherein the voice modulation module generates a character voice modulation variable corresponding to various voices having different frequency bands and tones, respectively. And providing the character voice information to the RSC encoding module together with the lip sync parameters extracted from the RTLS. The method of claim 8, The RSC encoding module generates a contour and a face of the substitute image based on the image feature information, and corrects a mouth shape of the substitute image by using a lip sync parameter included in the voice information. Alternative video service system using communication network. The method of claim 8, The RSC encoding module generates a mouth shape of the substitute image based on the voice information, and generates a face portion of the substitute image except the mouth shape based on the image feature information. Alternative video service system. The method according to claim 9 or 10, And an SGN server interworking with the RSC server to perform call processing and replacement video service management for a replacement video call, and having replacement video service information including replacement video service subscriber information and character information for each subscriber. Characterized in that, alternative video service system using a mobile communication network. The method of claim 11, Connected to user terminals using a WEB / WAP server, and performs registration of a substitute video service character and subscription of a substitute video service requested from the user terminal, and registers the substitute video service subscription and a substitute video service character. A CMS server for transmitting information about the SGN server; And Alternative storage service system using a mobile communication network, characterized in that it further comprises a storage device for storing the registered character. The method of claim 12, MMSC for generating, storing and transmitting multimedia messages; And A message audio processing unit for performing a non-real time lip sync function to analyze the audio signal included in the message stored in the MMSC and extracting the message voice information; and generating and generating a video character in non real time by combining the message voice information with character contents. And an LSS server having an alternative video message encoding module for transmitting the encoded video character to the MMSC. The method of claim 13, The RSC server, the LSS server, the SGN server, the CMS server and the storage device constitute an RSC service platform, and the RSC service platform further includes a media gateway for interworking with the mobile communication network. , Alternative video service system using mobile communication network. Using a mobile communication network that provides an alternative video service to an alternative video service system including an alternative video service platform connected to a core network of a mobile communication network including a base station, a controller, a switch, a location register, and a multimedia message center through a gateway. In the alternative video service method, The alternative video service platform, The video processing unit performs a real-time face tracking function which extracts and changes the feature point information of the image by restoring and analyzing the video signal received in the compressed state, and the audio information by restoring and analyzing the audio signal received in the compressed state. An RSC server including an audio processor configured to extract a real-time lip sync function, and an RSC encoding module configured to combine the image feature information and the voice information with character content to generate a substitute image which is a video in real time; Interworking with the RSC server to perform a call processing and replacement video service management for a replacement video call call, and has a replacement video service information including the replacement video service subscriber information and character information for each subscriber, and the alternative video service subscriber list An SGN server notifying the location register; Connected to user terminals using a WEB / WAP server, and performs registration of a substitute video service character and subscription of a substitute video service requested from the user terminal, and registers the substitute video service subscription and a substitute video service character. A CMS server for transmitting information about the SGN server; And A storage device for storing the registered character, The alternative video service method, Determining, by the location register, whether to subscribe to an alternative video service according to a request for establishing an alternative video call from a mobile terminal; If the calling terminal is a terminal of an alternative video service subscriber, as a result of the determination, an alternative video call call is established in the core network, and the RSC server activates a registered character for the calling terminal to prepare for an alternative video service. Doing; Generating a substitute image in real time by combining the image characteristic information and the audio information extracted from the video signal and the audio signal input from the calling terminal from the RSC server with the activated character; And And performing a video call using a real-time substitute image by transmitting the substitute image to the called terminal through the core network in the RSC server. The method of claim 15, The alternative video service platform may include a message audio processor configured to perform a non-real time lip sync function by analyzing an audio signal stored in a multimedia message and extract message voice information, and combine the message voice information with character content to display a video character in real time. It further comprises an LSS server having an alternative video message encoding module for generating and transmitting the generated video character to the MMSC, The alternative video service method, Transmitting a multimedia message from the MMSC to the LSS server; Extracting message voice information included in the multimedia message from the LSS server and combining the message voice information with a character registered for the calling terminal to generate a substitute image which is a video character; Transmitting the substitute image to the MMSC at the LSS server; And The MMSC further comprises the step of transmitting a replacement video message combined with the replacement video to the called terminal, alternative video service using a mobile communication network. The method of claim 16, The calling terminal has a hot key for an alternative video service, and the hot key is implemented with at least one of a softkey and a terminal button, and the request for establishing an alternative video call call from the calling terminal is a hot key for an alternative video service provided in the calling terminal. Alternative video service method using a mobile communication network, characterized in that performed by pressing. The method of claim 17, The storage device stores standardized mouth shapes for the registered character and default information on character facial features according to each mouth shape. The calling terminal is a terminal of an alternative video service subscriber and does not support a video call, and the called terminal is a terminal supporting a video call. The generating of the substitute image in real time by the RSC server may include generating the substitute image in real time by combining the audio information extracted from the audio signal input from the calling terminal with the default information. Alternative video service method using a mobile communication network. The method of claim 18, The called terminal is a terminal of an alternative video service subscriber and supports a video call. The alternative video service method, Generating a substitute video in real time by combining the video feature information and the audio information extracted from the video signal and the audio signal input from the calling terminal with the character registered for the called terminal in the RSC server; And  In the RSC server, in real time by ignoring the replacement image generated by the character registered for the calling terminal, and transmitting the replacement image generated by the character registered for the called terminal to the called terminal through the core network. And performing a video call using the replacement video. The method according to any one of claims 15 to 19, Classifying the voice information used in generating the substitute image by the RSC server for each calling terminal and repeatedly storing and learning the voice parameter DNA included in the voice information as personalized metadata in the storage device; And And using the learned personalized metadata as a default voice model when generating a replacement video for the same originating terminal in the future. A camera unit for acquiring an actual image of a user, an audio unit including a microphone and a speaker in charge of input / output of an audio signal, a key input unit including at least one of a softkey and a hotkey for requesting an alternative image service, and the actual A display unit for displaying an image and a substitute image, an RSC module for generating the substitute image, a data memory storing data including the character for the substitute image and standard mouth shape information, and performing the substitute image generating function A program memory for storing a terminal operating program including a software program for performing the operation, a transceiver for transmitting and receiving transmission signals, and at least one microprocessor connected to the components to control the functions of the components. It includes a control unit is configured, The RSC animation engine generates an outline and a face of the substitute image based on character feature information from the RFT engine when data of a part or the whole of an audio signal is lost, and character voice information from the RLS engine. Using to correct the shape of the mouth of the substitute image, The RSC animation engine generates a mouth shape of the substitute image based on the voice information when data of a part or all sections of a video signal is lost, and the mouth shape based on character feature information from the RFT engine. A mobile communication terminal for a replacement video service, characterized in that for generating a face portion of the replacement video except for. The method of claim 21, The RSC module, A video decoder for processing the video signal; An RFT engine generating character feature information for the substitute image by extracting feature points of the real image included in the video signal; An audio decoder for processing the audio signal; An RLS engine configured to generate character voice information for determining a mouth shape of the character for the substitute image by extracting voice information included in the audio signal and matching the standard mouth shape information; An RSC animation engine for combining the character feature information from the RFT engine and the character voice information from the RLS engine with the character for the substitute image stored in the data memory to generate the substitute image; A renderer for rendering a substitute image generated by the RSC animation engine; And And a video encoder for encoding the substitute image rendered by the renderer. The method of claim 22, And the control unit simultaneously displays the actual image of the user acquired by the camera unit and the substitute image of the user generated by the RSC module on the display unit. The method of claim 23, And the control unit additionally displays at least one of an actual image and a substitute image of a call counterpart received through the transceiver, on the display unit. The method of claim 24, The RLS engine repeatedly stores the voice parameter DNA included in the extracted voice information as the user's metadata in the data memory, and defaults the personalized metadata learned by the repetitive storage when the voice information is extracted in the future. The mobile communication terminal for an alternative video service, characterized in that used as. delete delete delete In the alternative video service system using a mobile communication network, RSC module to perform character feature information extraction, including a video decoder for processing a video signal obtained from a camera, and an RFT engine for generating character feature information for a substitute image by extracting feature points of the actual image included in the video signal. A telecommunications terminal configured to include; And An audio processing unit connected to a core network of the mobile communication network and performing a real-time lip sync function to restore and analyze audio signals received in a compressed state from the mobile communication terminal to extract voice information; And an RSC server including an RSC encoding module for generating a substitute video which is a video in real time by combining the voice information and character feature information for a substitute video to the mobile communication terminal. And repeatedly storing the voice parameter DNA included in the extracted voice information in the data memory as user metadata, and using personalized metadata learned by the storage as a default voice model when extracting voice information in the future. Alternative video service system using mobile communication network. The method of claim 29, The RFT engine compares the feature point information of the reconstructed image with the feature point information of the basic contour of the pre-stored human face to determine whether the reconstructed image is a human face image, and the reconstructed image is a human face image. A face tracking module that tracks the change of the individual feature points of the face part in real time when it is discriminated, and a motion tracking module that tracks the feature point information of the contour of the face part in real time when the reconstructed image is identified as a human face image; , The change of the feature points tracked by the face tracking module and the motion tracking module is transmitted to the RSC encoding module of the RSC server as feature information of the character for the substitute video service. system. The method of claim 30, The audio processor includes an audio decoder for restoring the received audio signal in the compressed state, and an RLS engine for receiving the audio signal from the audio decoder and extracting a voice signal to perform real-time lip syncing. The RLS engine filters the input audio signal and extracts only a signal having a predetermined frequency range as a voice signal, and matches the mouth shape information database to the voice signal extracted by the voice analysis module to generate a lip sync variable in real time. Contains an RTLS module to extract, The lip sync variable is transmitted to the RSC encoding module as voice information of the character for the alternative video service, alternative video service system using a mobile communication network. delete 32. A method of performing a replacement video service using the replacement video service system according to any one of claims 29 to 31, Extracting, by the mobile communication terminal, the character feature information of the actual video according to a request for establishing an alternative video call call from the mobile communication terminal; Determining replacement video service subscription information in the RSC server according to the replacement video call call setup request; In the RSC server, the registered character is activated according to the substitute video service subscription information, and the registered character is used for the voice information extracted by the audio processor and the character feature information from the mobile communication terminal. Generating a substitute image in real time by combining with a; And And performing a video call using a real-time substitute image by transmitting the substitute image to the called terminal through the core network in the RSC server.

Images