KR101701813B1 - Display apparatus and Method for changing Voice thereof - Google Patents

Abstract

A display device and a voice conversion method thereof are provided. A method of converting a voice of a display device, when a first video frame is input, detecting at least one entity included in the first video frame, storing the selected entity when one of the detected entities is selected, If one of the voice samples is selected, the selected voice sample is matched to the selected entity and stored. When a second video frame including the selected entity is input, the voice of the selected entity is converted into the selected voice sample and output. Thereby, the voice of the entity included in the video frame is changed to the selected voice sample, so that the user can receive the voice-customized contents.

Description

[0001] The present invention relates to a display apparatus and a method of converting the same,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a display device and a voice conversion method thereof, and more particularly, to a display device for customizing and converting audio of contents and a voice conversion method therefor.

In general, IPTV (internet protocol television) is a system for providing multimedia services such as audio and video information through network-based IP. At this time, the multimedia service may include live TV, video on demand (VOD), and time shifted programming. Conventionally, it has been possible to replace the face of an entity displayed in a video clip. At this time, the entity defines the face of a specific character selected by the user in the video clip. In addition, the face recognition method can be used to provide an effective replacement of the entity selected by the user in the video clip to another entity. In addition, conventionally, the conversation can be changed to the entity selected by the user. That is, the user could select the first conversation from the video and replace the first conversation with the second conversation. However, a technique for changing the voices of selected entities has been required.

Accordingly, there is a need to develop a method and system for customizing the voices of selected entities.

It is an object of the present invention to provide a display apparatus and a voice conversion method for customizing a voice of an entity selected by a user among entities included in an input video frame.

According to another aspect of the present invention, there is provided a method of converting a voice of a display device, the method comprising: detecting, when a first video frame is input, at least one entity included in the first video frame; If one of the detected entities is selected, storing the selected entity; If one of the pre-stored plurality of voice samples is selected, matching the selected voice sample to the selected entity and storing the selected voice sample; And converting a voice of the selected entity into the selected voice sample and outputting the selected voice sample when the second video frame including the selected entity is input.

Wherein the entity is a face of a person included in a video frame, and wherein the detecting comprises detecting at least one of the skin color, motion, size, shape and position of the at least one entity via the face detection module So that the face of the person included in the video frame can be detected.

The method may further include, when the at least one entity is detected in the input video frame, displaying the detected entity as a list in one area of the display screen.

And displaying the plurality of voice samples in a list in one area of the display screen when the entity is selected.

The step of storing the selected entity may further include storing a first identifier corresponding to the selected entity in a look-up table, and storing the selected voice sample may include storing a second identifier corresponding to the selected voice sample It can be stored in a look-up table.

The plurality of voice samples include at least one of a voice sample embedded in the display device, a recorded voice sample, and a voice sample input by the user, and the recorded voice sample and the voice input by the user The sample can be filtered by the voice subsampler module.

The outputting step may include determining whether the selected entity is included in the second video frame.

The outputting step may include: determining whether or not the selected entity included in the second video frame has a lips movement; And converting the voice of the entity into the selected voice sample and outputting the selected voice sample if it is determined that the lip movement of the entity is present.

According to another aspect of the present invention, there is provided a display apparatus including: a detector for detecting at least one entity included in a first video frame when a first video frame is input; A user interface unit for selecting an entity to perform voice conversion among the detected entities and receiving a voice sample matching the selected entity; A storage for storing the selected entity and the selected voice samples; And a controller for converting a voice of the selected entity into the selected voice sample and outputting the selected voice sample when a subsequent video frame including the selected entity is input.

And the detection unit detects at least one of the skin color, motion, size, shape and position of the at least one entity via the face detection module The face of the person included in the video frame can be detected.

A video processing unit for processing the input video frame; An audio processing unit for processing an audio signal corresponding to the input video frame; A display unit for outputting video frames processed by the video processing unit on a screen; And an audio output unit for outputting the audio signal processed by the audio processing unit in synchronization with the video frame, wherein the control unit converts the voice of the selected entity into the selected voice sample and provides the selected voice sample to the audio output unit The audio processing unit can be controlled.

When the at least one entity is detected in the input video frame, the controller may control the display unit to display the detected entity in a region on the screen.

In addition, when the entity is selected, the controller may control the display unit to display the plurality of voice samples in a list in one area of the screen.

The storage unit may store a first identifier corresponding to the selected entity and a second identifier corresponding to the selected voice sample in a look-up table.

The storage unit may store at least one of a voice sample stored in the display device, a recorded voice sample, and a voice sample input by the user, and the selected voice sample may include at least one of the built- And a voice sample input by the user.

The recorded voice samples and the voice samples input by the user can be filtered by the voice subsampler module.

In addition, the controller may search for and determine whether the selected entity exists in the face of the person included in the second video frame using the face detection sub-module.

The control unit may determine whether or not there is lip movement of the selected entity included in the second video frame, and convert the voice of the entity into the selected voice sample if it is determined that the lip movement of the entity is present And output it.

According to various embodiments of the present invention as described above, the voice of the entity included in the video frame is changed to the selected voice sample so that the user can receive the voice-customized content.

1 is a block diagram of a display device according to an embodiment of the present invention,
2 is a block diagram of an arrangement for customizing voices of selected entities in a display device, in accordance with an embodiment of the present invention;
3 is a flowchart illustrating a method for customizing a voice of an entity selected in a display device, according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a method of selecting and updating an entity using a first presentation module according to an embodiment of the present invention; FIG.
Figure 5 illustrates a user interface including a look-up table for entity selection, according to one embodiment of the present invention;
6 is a flow chart illustrating a method for selecting a voice sample for customizing voices using a voice sub-sampler module, in accordance with an embodiment of the present invention;
Figure 7 illustrates a user interface including a look-up table for selecting voice samples, according to one embodiment of the present invention,
8 is a flowchart illustrating a method of customizing a voice using a core processing module according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings. 1 is a block diagram of a display device 1 according to an embodiment of the present invention.

1, the display device 1 includes a video input unit 10, a detection unit 20, a video processing unit 30, an audio processing unit 40, a storage unit 50, an audio output unit 60, A display unit 70, a user interface unit 80, and a control unit 90.

The video input unit 10 receives video data composed of video frames from an external source connected wirelessly or wirelessly. At this time, the video input unit 10 may receive broadcast data from a broadcasting station or receive video data from a video input device such as a DVD player.

The detection unit 20 detects an entity from the video frame of the input image data. At this time, the entity may be an image of a specific character including a face or a face of a person included in the video frame. Therefore, the detection unit 20 can detect the entity included in the video frame using the face detection module. The detection unit 20 can use the skin color, motion, size, shape, and position of the entity when detecting the entity.

The video processing unit 30 processes video frames of input video data. That is, the video processing unit 30 performs image processing operations such as decoding and scaling of inputted image data.

The audio processing unit 40 processes the audio signal corresponding to the input video frame. At this time, under the control of the control unit 90, the audio processing unit 40 can process the voice of the entity included in the video frame to change.

The storage unit 50 stores various data and multimedia data for driving the display device 1. [ In addition, the storage unit 50 stores various modules for voice conversion of the display device 1. [

The audio output unit 60 outputs the audio signal processed by the audio processing unit 50. [ At this time, the audio output unit 60 may be implemented as a speaker.

The display unit 70 displays the processed video frames in the video processing unit 30. [

The user interface unit 80 receives a control command from the user to control the display device 1. [ In particular, a voice sample to be applied to the entity to be voice-converted and the entity to be voice-converted through the user interface unit 80 can be selected.

At this time, the user interface unit 80 may be implemented as a GUI (Graphic User Interface) and an input device such as a touch screen, a remote controller, and a pointing device.

The control unit 90 controls the entire display device 1 based on the control command received through the user interface unit 80. In particular, the control unit 90 may convert the voice for voice customization of the entity included in the video frame.

Specifically, when the first video frame is input through the video input unit 10, the control unit 90 controls the detection unit 20 to detect at least one entity included in the first video frame.

When at least one entity is detected, the control unit 9 controls the display unit 80 to display a list including at least one entity in one area of the display screen so that one of the at least one entity can be selected .

When a first entity to be voice-converted is selected from among the detected entities through the user interface unit 80, the controller 90 controls the storage unit 50 to store the selected first entity. At this time, the control unit 90 may control the storage unit 50 to store the first identifier corresponding to the selected first entity together with the selected first entity.

The control unit 90 controls the display unit 80 to display a list including a plurality of voice samples in one area of the display screen in order to select a voice sample matching the selected first entity. At this time, the plurality of voice samples include at least one of a voice sample built in the display device 1, a recorded voice sample, and a voice sample input by the user.

When a specific voice sample among a plurality of voice samples is selected through the user interface unit 80, the control unit 90 controls the storage unit 50 to store the selected voice sample and the selected first entity in a matching manner. At this time, the control unit 90 may control the storage unit 50 to store the second identifier corresponding to the selected voice sample.

When the second video frame is input, the control unit 90 determines whether or not there is an entity corresponding to the first entity selected in the second video frame. If there is an entity corresponding to the selected first entity among the plurality of entities included in the second video frame, the controller 90 converts the voice of the selected first entity into the selected voice sample and outputs the selected voice sample to the audio output unit 60 The processing unit 40 can be controlled.

In particular, the control unit 90 determines whether or not there is lip movement of the selected first entity included in the second video frame, and converts the voice of the first entity into the selected voice sample And controls the audio processing unit 40 to output it.

At this time, the controller 90 may change at least one of the tone color and the pitch of the peculiar characteristics of the voice of the first entity.

The voice of the first entity included in the video frame is changed to the selected voice sample by the display device 1 as described above so that the user can receive the voice-customized contents.

Hereinafter, the voice conversion method of the present invention will be described in more detail with reference to FIG. 2 to FIG.

Figure 2 is a block diagram of a configuration for customizing voices of selected entities in a display device, in accordance with an embodiment of the present invention. In particular, FIG. 2 is a diagram showing a configuration module of a display device 100 for voice conversion according to another embodiment of the present invention.

2, a block diagram of a configuration for customizing an entity's voice includes a first video frame 105, a face detection module 110, a first presentation module 115 for selection of an entity, Up table 120 for storing one entity, a second presentation module 125 for selecting a voice sample, a second identifier 130, a first identifier 195, an input video frame 135, The core processing module 145 includes a core processing module 145, a user input voice sample 165, a recorded voice 170, a voice subsampler module 180 and a voice database 190, A lips movement detection submodule 155 and a voice control unit 160. The voice subsampler module 180 includes a voice processing module 15 and a recording module 185. [

The first video frame 105 is displayed on the display device 100. As an example of the display device 100, a computer, an IPTV, a VOD, a CE (Consumer electronic) device, and an Internet TV may be used, but the present invention is not limited thereto. One example of the first video frame 105 may include, but is not limited to, scenes of movies, broadcast streams, live video, and video clips. The display device 100 receives the first video frame 105 over the network. The network may include, but is not limited to, a wireless network, the Internet, an intranet, a Bluetooth, a small area network (SAN), a metropolitan area network (MAN), and an Ethernet. The first video frame 105 includes a plurality of entities. At this time, the entity may be regarded as a plurality of characters displayed in the first video frame 105. A user may select a particular entity from a plurality of entities displayed in the first video frame 105 for voice customization. Hereinafter, a specific entity selected by the user is referred to as a first entity 140.

The user can perform voice setting options of the display device 100 for voice customization. For selection in the voice configuration option, the face detection module 110 may be activated to capture the first video frame 105. [ The face detection module 110 extracts at least one entity included in the captured first video frame 105. The face detection module 110 may utilize a plurality of unique properties to extract at least one entity displayed in the first video frame 105. At this time, one specific characteristic may include, but is not limited to, skin color, motion, size, shape, and position. In addition, the face detection module 110 may utilize various algorithms to extract the entities contained in the first video frame 105.

The entities included in the first video frame 105 may then be represented by the first presentation module 115 as a list. A particular entity may be selected for voice customization to the user from the entities listed in the first presentation module 115. For entity selection by the user, the first entity 140 is stored in a look-up table. The first entity 140 is then associated with a first identifier 195. That is, the first identifier 195 represents the first entity 140. In addition, the first identifier 195 exclusively represents the first entity 140. In addition, the look-up table 120 includes a second identifier 130. The second identifier 130 indicates a voice sample used for voice customization. A plurality of voice samples are stored in the voice database 190. A user may select a particular voice sample that may be used for voice customization from a plurality of voice samples stored in the voice database 190. [ Hereinafter, a specific voice sample selected by the user for voice customization will be referred to as a "selected voice sample ". In addition, the second presentation module 125 is used to list the voice samples stored in the voice database 190. The second presentation module 125 may allow the user to select a particular voice sample that may be used for voice customization.

The voice subsampler module 180 processes the voice samples selected by the user. But are not limited to, a recorded voice sample 170 as an example of the voice samples input by the user, an embedded voice sample (not shown) stored in the database, and a voice sample input by the user. Generally, a built-in voice sample is provided by a service provider. In addition, the voice subsampler module 180 enhances the sound quality of a particular voice sample by allowing the voice samples to pass through a smooth filter prior to storing the particular voice sample in the voice database 190. The voice subsampler module 180 also allows the voice recording module 185 to record voice samples in real time. In addition, the user may enter voice samples from the web into the voice subsampler module 180. Some of the voice samples input by the user and the recorded voice samples are processed by the voice processing module 175. [ The processed voice samples are input to the voice database 190. When registering a voice sample in the voice database 190, a second identifier 130 is generated. Each voice sample stored in the voice database 190 is stored in association with the corresponding second identifier 130. A second identifier generated for each voice sample stored in the voice database 190 is used to exclusively represent the voice samples. The plurality of voice samples are displayed in a list for the user to select using the second presentation module 125. The user may select a particular voice sample from the voice samples included in the list generated by the second presentation module 125. [ When selecting a specific voice sample for voice conversion, the second identifier 130 corresponding to the specific voice selected by the user is stored in the look-up table 130. [ The second identifier 130 may be used when mapping a particular voice sample selected by the user to the first entity 130 from the database 190. [

The core processing module 145, which includes a facial search submodule 150, a lip motion detection submodule 155 and a voice control 160, is coupled to the voice subsampler module 130. The core processing module 145 is a core part of the display device 100. The core processing module 145 determines whether or not the voice setting option has been performed by the display device 100. [

When the voice configuration option is performed, the core processing module 145 receives the input video frame 135. [ The video frame 135 may be considered a video clip that is used when the user implements voice customization. The core processing module 145 creates a first query for the look-up table 120. The first query is used to retrieve the first entity (140) selected by the user stored in the look-up table (120). The first entity 140, represented by the first identifier 195, is provided as input to the facial search submodule 150. The facial search submodule 150 performs capturing for a plurality of entities of the input video frame 135. [ The facial search submodule 150 searches for a match between a plurality of entities included in the input video frame 135 and the first entity 140. The core processing module 145 uses an image processing technique to search for a match between the plurality of entities included in the input video frame 135 and the first entity 140. The core processing module 145 operates the lip motion detection submodule 155 when an entity that matches the first entity 140 among the plurality of entities included in the input video frame 135 is searched. The lip motion detection sub-module 155 analyzes the input video frame 135 to determine a representation of the lip motion of the selected entity. When the lip motion of the selected entity contained in the input video frame 135 is detected, the lip motion detection submodule 155 generates an interrupt for the voice control unit 160. The voice control unit 160 generates a second query to retrieve the second identifier 130 stored in the look-up table 120. [ The voice control unit 160 also generates a third query for transmitting to the voice database 190 to retrieve the selected voice sample using the second query. The voice control unit 160 customizes voices by changing voice characteristics such as tone color and pitch. The voice conversion may be performed using a voice morphing method or the like. The voice morphing method can also be represented by a voice conversion method.

The look-up table 120 is used to map selected entities to be provided for voice conversion. The look-up table 120 stores the first entity 140, the second identifier 130 and the first identifier 195 for a certain period of time. When an entity is selected from the first presentation module 115, the corresponding first entity 140 enters the look-up table 120. In addition, the look-up table 120 generates the first identifier 195 when the first entity 140 enters the look-up table 120. Also, when a voice sample is selected from the second presentation module 125, the second identifier 130 enters the look-up table 120 for the selected voice sample.

When the lips movement of the selected entity in the input video frame 135 is detected, the look-up table 120 extracts the second identifier. The second identifier is used to invoke a voice sample that can be applied to the selected entity from the voice database 190. The voice control unit 160 extracts voice features such as tone color and pitch to customize the voice of the selected entity. At this time, the voice customizing is performed in the display device 100 without disturbing the user's viewing.

3 is a flowchart illustrating a method for voice customizing an entity selected in the display device 100, according to an embodiment of the present invention.

In step 210, at least one entity included in the first video frame is included. The at least one entity may be the face of the character contained in the first video frame. An example of the first video frame may include, but is not limited to, a video clip and a broadcast video. At least one entity included in the first video frame may be captured by the face detection module. The face detection module may utilize a plurality of unique properties to capture at least one entity included in the first video frame. Examples of unique properties include, but are not limited to, skin color, motion, size, shape, and position. In addition, the face detection module may use various algorithms to capture at least one entity included in the first video frame.

In step 215, at least one entity included in the first video frame is displayed as a list. The at least one list included in the first video frame may be performed by the first presentation module. The first presentation module also displays at least one entity included in the first video frame. The user selects a particular entity from at least one entity included in the list generated using the first presentation module.

In step 220, the user selects a first entity from at least one entity included in the first video frame. The entities included in the first video frame may be included in the list generated using the first presentation module. The selection of the first entity from at least one entity included in the first video frame may be performed by a user interface. At this time, the user interface may include a GUI (Graphic User Interface), a touch screen, and a command line interface, but the present invention is not limited thereto. As a first example, a user may select a first entity among at least one entity included in a first video frame by providing input to a first presentation module using a GUI.

In one embodiment of the present invention, the first entity selected by the user may be stored in a look-up table. The look-up table may be implemented to generate a first identifier for identifying a first entity. The first identifier indicates the first entity. Similarly, a plurality of first identifiers may be generated to represent a corresponding plurality of entities included in the first presentation module. And at least one entity may be stored in a look-up table. The look-up table may be implemented using a processor.

In yet another embodiment of the present invention, a hash table may be used to store at least one entity included in the first video table.

In step 225, a first voice sample is selected. The first voice sample may be stored in a voice database. Voice databases can be embedded and can be located remotely. The voice database includes a plurality of voice samples. In a first embodiment, a first voice sample may be represented using a second identifier. The second identifier indicating the first voice sample may be stored in a look-up table. Similarly, a plurality of second identifiers may be generated to represent a plurality of corresponding voice samples stored in the voice database.

In another embodiment, a hash table may be used to store a second identifier representing the first voice sample.

In step 230, it is determined whether there is a match between at least one entity included in the input video frame and the first entity. Whether a match between the first entity and at least one entity included in the input video frame is performed using the face search submodule. The facial search submodule compares the first entity with at least one entity contained in the input video frame. Digital image processing techniques may be used for comparison of at least one entity with a first entity. The face retrieval submodule also matches the first entity with at least one entity included in the input video frame. Matching the first entity selected by the user with at least one entity included in the input video frame and then determining lips movement of the selected entity among at least one of the entities of the input video frame.

In one embodiment of the invention, various face recognition algorithms can be used by the face detection submodule to search for matching of the first entity with at least one entity included in the input video frame.

In step 235, a first entity selected between at least one of the entities included in the input video frame is analyzed for lips movement. The lip motion of the selected entity may be detected by the lip motion detection submodule. The lip motion detection submodule may use a speech processing technique to analyze whether the selected entity has lip motion.

In a first embodiment of the present invention, the lip motion detection submodule determines the necessity of voice conversion. The lip motion detection sub-module detects the presence of lip motion. If lip motion is detected at the selected entity, the lip motion detection submodule begins the process of performing the voice conversion. However, if no lip motion is detected at the selected entity, the lip motion detection submodule bypasses the process of performing the voice conversion.

In another embodiment of the present invention, various algorithms can be applied to the lip motion detection submodule to detect the lip motion of the selected entity.

In step 240, the voice of the selected entity is transformed. The voices of the selected entities can be converted using the voice control unit. The voice transformation includes replacing the first voice sample with the selected entity voice. In one example, the first voice sample may be stored in a voice database. The voice database stores a plurality of voice samples. The voice control unit may use various voice synthesis techniques to convert the voice of the selected entity to the first voice sample.

In one embodiment of the invention, the lip motion detection submodule may activate the voice control unit to convert the voice of the selected entity to a first voice sample. In one example, the lip motion detection submodule may be enabled using an interrupt. The lip motion detection submodule generates an interrupt. The interrupt allows the voice control unit to convert the voice of the selected entity to a first voice sample. In addition, the voice conversion can be applied for a specific time. The specific time means the period during which the voice conversion occurs.

4 is a flowchart illustrating a method for selecting and updating an entity using a first presentation module, according to an embodiment of the present invention.

In step 310, the first video frame is received as an input to the face detection module. An embodiment of a video frame may include, but is not limited to, moving pictures, broadcast streams, live images, and video clips. Also, the first video frame includes a plurality of entities. The entity may be the face of the character included in the first video frame.

In step 315, the first video frame is captured using the face detection module. An example of an image capture technique may include, but is not limited to, digital image processing techniques and chroma key techniques.

In step 320, at least one entity included in the first video frame is extracted by the face detection module. Extraction of an entity may be accomplished by ascertaining a plurality of unique properties associated with the entity contained in the first video frame. Examples of unique properties include, but are not limited to, skin color, motion, size, shape, and location. Also, various algorithms can be used to capture at least one entity included in the first video frame.

In step 325, at least one entity included in the first video frame is displayed as a list. The first presentation module including at least one entity included in the first frame. The first presentation module displays at least one entity included in the first video frame. The user may select a particular entity from at least one entity listed in the list using the first presentation module. Hereinafter, a specific entity selected by the user will be referred to as a first entity.

In step 330, the user selects a first entity from at least one entity included in the first video frame. The entities included in the first video frame may be displayed as a list using the first presentation module. The selection of the first entity among the at least one entity included in the first video frame may be selected by the user interface. At this time, one example of the user interface may include a GUI, a touch screen, and an instruction line interface, but is not limited thereto.

In step 335, the first entity is stored in a look-up table. The lookup table may be implemented to generate a first identifier for identifying the first entity. The first identifier indicates the first entity. Similarly, a plurality of first identifiers may be generated to identify a corresponding plurality of entities contained in the first presentation module. At this time, at least one entity may be stored in a look-up table.

5 is a diagram illustrating a user interface including a lookup table for selecting an entity, in accordance with an embodiment of the present invention. 5 shows a display unit 405, a video frame 410 including at least one entity, a first entity 415, a second entity 420, a first identifier for identifying the selected first entity 415, And a look-up table in which the image data 430 is stored.

The display unit 405 displays at least one entity included in the video frame 410. Examples of the display unit 405 include a computer, an IPTV, a VOD, and an Internet TV, but the present invention is not limited thereto. At least one entity is detected from a video frame using a face detection module. At least one entity detected in the video frame is displayed as a list for user selection using the first presentation module. Here, the list displayed by the first presentation module includes a first entity 415 and a first entity 420 for user selection. As an example, the user selects the first entity 415. As another example, the user may select the second entity 420. The first entity selected by the user is stored in the look-up table 425. The look-up table 425 generates a first identifier 430 for the selected first entity 415. The first identifier 430 represents the selected first entity 415. In addition, the look-up table may generate another first identifier to represent the second entity. Similarly, a plurality of first identifiers may be generated to represent a plurality of entities corresponding to the first presentation module. At this time, at least one entity 425 is stored in a look-up table.

6 is a diagram for explaining a method of selecting a voice sample for voice customization according to an embodiment of the present invention. The voice subsampler module processes the voice samples input by the user. An example of a voice sample input by a user may include, but is not limited to, a recorded voice sample, a sample voice, and a voice sample input by the user.

In step 510, the user may be provided with an option to select a voice output from among a plurality of pre-processed voice samples stored in the voice database. The preprocessed voice may be a built-in voice sample. The built-in voice samples are stored in the voice database. Typically, a built-in voice sample is provided by a service provider. If the user desires to select a preprocessed voice for voice customization, the user may select a preprocessed voice sample output from a plurality of voice samples stored in the voice database, as described in step 525. [ Also, if the user does not want to use the preprocessed voice samples for voice customization, the user can use the recorded voice samples for voice customization. In step 515, the user can use the recorded voice samples using the recording module. If the user wishes to use the recorded voice samples for voice customization, the recording process begins, as shown in step 530. [ Also, if the user does not want the recorded voice samples for voice customization, the user may enter voice samples that may be used for voice customization, as shown in step 520. [ Also, in step 535, the recorded voice samples are processed using a voice subsampler module. The voice subsampler module removes various noises such as random noise and quantization noise included in the recorded voice samples. The voice subsampler module may enhance the sound quality of the recorded voice samples by applying a voice sample recorded with a smoothing filter prior to storing in the voice database, In step 540, the recorded voice samples are stored in the voice database.

Also, as in step 520, the voice samples input by the user for voice customization are processed using the voice subsampler module shown in step 535. In step 540, the voice samples input by the user are stored in the voice database.

7 is a diagram illustrating a user interface including a look-up table for selecting a voice sample, in accordance with an embodiment of the present invention. 6, a display unit 605, a recording module 610, and a look-up table 640 are included.

The display unit 605 displays at least one entity included in the video frame. Examples of the display unit include, but are not limited to, a computer, IPTV, VOD, and Internet TV. The user selects an entity from at least one entity included in the video frame. In one embodiment of the present invention, the selection may be performed by dragging the cursor to the selected entity, inputting it using the keyboard, or using the touchpad. For example, the selected entity may be a character included in a video frame, as shown at 635. The selected entities 635 are stored in the look-up table 640. The lookup table 640 also generates a first identifier 645. The first identifier 645 exclusively represents the selected entity 635. Similarly, a plurality of first identifiers for a corresponding plurality of selected entities are stored in a look-up table 640. As an example, the user desires to record a voice sample using the recording module 610. The available voice samples for voice customization of the selected entity 635 may include, but are not limited to, a robot voice sample 615, a celebrity voice sample 620, and a baby voice sample 625. The voice samples 625 as described above are stored in the voice database. Each voice sample stored in the voice database includes a second identifier that individually displays each. The selection of the voice samples as described above is stored in the voice database for the purpose of voice conversion of the user. When a voice sample for voice customization is selected, the second identifier 630 corresponding to the selected voice sample is stored in the look-up table 640. A second identifier 630 is used to retrieve a voice sample from the voice database. The selected voice samples can be used for customizing the voice of the selected entity.

8 is a flowchart illustrating a method of customizing a voice using a core processing module according to an embodiment of the present invention. In step 710, the core processing module receives the input video frame. The input video frame may include, but is not limited to, scenes of moving pictures, video clips, and broadcast streams. In step 715, the core processing module determines whether the user desires voice customization. If the user desires voice customization, the input video frame is analyzed, as shown in step 720. However, if the user does not want voice customization, the core processing module bypasses the process for voice customization.

In step 720, the input video frame is analyzed. The core processing module analyzes the video frame by capturing at least one entity contained in the input video frame. Capture of at least one entity may be performed using a face search submodule. The facial search submodule can capture entities by identifying various unique properties associated with a plurality of entities contained in an input video frame. Examples of unique properties include, but are not limited to, skin color, motion, size, shape and location. Also, various algorithms may be applied to capture at least one entity included in the input video frame.

In step 725, the core processing module creates a first query to retrieve the first entity selected by the user stored in the look-up table. The first entity selected by the user stored in the look-up table is provided as input to the facial search submodule.

In step 730, the core processing module retrieves a match between the first entity stored in the look-up table and the plurality of entities stored in the input video frame. The matching between the first entity stored in the look-up table and the plurality of entities stored in the input video frame can be retrieved using the face search submodule.

In step 735, the core processing module determines whether the first entity stored in the look-up table and the plurality of entities stored in the input video frame are matched. Once a match is found, the core processing module checks the input video image to determine whether the lip motion of the selected entity is present. However, if a match is not found, the steps for voice customization are bypassed, as shown in step 765.

In step 740, the core processing module checks the input video frame to determine whether there is a lip motion of the corresponding selected entity. The lip motion detection submodule is used to determine the lips movement of the selected entity. After determining whether there is a lips movement of the selected entity of the input video frame, an interrupt is generated for the voice control. However, if it is determined that there is no lips movement of the selected entity, the steps for voice customization are bypassed, as shown in step 765.

In step 745, the lip motion detection submodule generates an interrupt to be transmitted to the voice control unit. The interrupt is generated as a signal to implement voice customization for the selected entity. Interrupts are generated for delivery to the voice control according to the presence or absence of lips movement for the selected entity in the input video frame.

In step 750, the voice control unit generates a second query to be sent to the look-up table to recall the second identifier. The second identifier indicates the selected voice. The selected voice is used for customizing the voice for the selected entity. The second query is used for transmission of the second identifier representing the selected voice.

In step 755, the third query is generated to be transmitted to the voice database to retrieve the selected voice samples using the second identifier. The voice database stores a plurality of voice samples for voice customization. Each voice sample stored in the voice database is associated with a corresponding plurality of second identifiers. Then, a plurality of voice samples associated with the second identifier are stored in the voice database. The third query is used for transmission of the selected voice samples from the voice database.

In step 760, the voice of the selected entity is changed to the selected voice sample. The voice control unit changes the voice of the selected entity to the selected voice sample. The voice change performed by the voice control section includes changing the specification of voice such as tone color and pitch.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

10: image input unit 20:
30: video processing unit 40: audio processing unit
50: storage unit 60: audio output unit
70: display unit 80: user interface unit
90:

Claims (18)

A method of converting a voice of a display device,
Detecting, when a first video frame is input, at least one entity included in the first video frame;
If one of the detected entities is selected, storing the selected entity;
Displaying a plurality of previously stored voice samples in one area of the display screen when the entity is selected;
If one of the pre-stored plurality of voice samples is selected, matching the selected voice sample to the selected entity and storing the selected voice sample;
And converting a voice of the selected entity into the selected voice sample when the second video frame including the selected entity is input, and outputting the selected voice sample. The method according to claim 1,
The entity comprising:
A face of a person included in a video frame,
Wherein the detecting comprises:
Wherein a face of the person included in the video frame is detected using at least one of the skin color, motion, size, shape and position of the at least one entity through the face detection module. The method according to claim 1,
Further comprising: when the at least one entity is detected in the input video frame, displaying the detected entity in a list in one area of the display screen. delete The method according to claim 1,
Wherein storing the selected entity comprises:
Storing a first identifier corresponding to the selected entity in a look-up table,
Wherein storing the selected voice samples comprises:
And a second identifier corresponding to the selected voice sample is stored in a look-up table. The method according to claim 1,
Wherein the plurality of voice samples comprise:
A voice sample, a recorded voice sample, and a voice sample input by a user, the voice sample being embedded in the display device,
The recorded voice samples and the voice samples input by the user,
Wherein the voice signal is filtered by a voice subsampler module. 3. The method of claim 2,
Wherein the outputting step comprises:
And determining whether the selected entity is included in the second video frame. The method according to claim 1,
Wherein the outputting step comprises:
Determining whether the selected entity included in the second video frame has a lips movement;
And converting the voice of the entity into the selected voice sample if the lips movement of the entity is determined to exist. In the display device,
A detector for detecting at least one entity included in the first video frame when the first video frame is input;
A user interface unit for selecting an entity to perform voice conversion among the detected entities and receiving a voice sample matching the selected entity;
A storage for storing the selected entity and the selected voice samples; And
And a controller for converting a voice of the selected entity into the selected voice sample when the second video frame including the selected entity is input,
Wherein,
And controls the display unit to display a plurality of previously stored voice samples in one area of the screen when the entity is selected. 10. The method of claim 9,
The entity comprising:
A face of a person included in a video frame,
Wherein:
Wherein at least one of the skin color, motion, size, shape, and position of the at least one entity is used to detect a face of a person included in the video frame through the face detection module. 10. The method of claim 9,
A video processing unit for processing the input video frame;
An audio processing unit for processing an audio signal corresponding to the input video frame;
A display unit for outputting video frames processed by the video processing unit on a screen; And
And an audio output unit for outputting the audio signal processed by the audio processing unit in synchronization with the video frame,
Wherein,
And controls the audio processing unit to convert a voice of the selected entity into the selected voice sample and provide the selected voice sample to the audio output unit. 12. The method of claim 11,
Wherein,
And controls the display unit to display the detected entity in a list on one area on the screen when the at least one entity is detected in the input video frame. delete 10. The method of claim 9,
Wherein,
And stores a first identifier corresponding to the selected entity and a second identifier corresponding to the selected voice sample in a look-up table. 10. The method of claim 9,
Wherein,
A voice sample stored in the display device, a recorded voice sample, and a voice sample input by the user,
Wherein the selected voice sample comprises:
A voice sample recorded by the user, and a voice sample input by the user. 16. The method of claim 15,
The recorded voice samples and the voice samples input by the user,
And the signal is filtered by a voice subsampler module. 11. The method of claim 10,
Wherein,
Wherein the face detection submodule is used to search for and determine whether the selected entity is present in a face of a person included in the second video frame. 12. The method of claim 11,
Wherein,
And determining whether or not there is lip movement of the selected entity included in the second video frame. When it is determined that lip movement of the entity is present, the voice of the entity is converted into the selected voice sample and outputted. / RTI >

Images