KR101962126B1 - Multimedia device for accessing database according to result of voice recognition and method for controlling the same - Google Patents

Abstract

According to another aspect of the present invention, there is provided a method of controlling a multimedia device accessing a database according to a speech recognition result, the method comprising: extracting text data and audio data by scanning at least one channel; Storing data in a database (DB) according to a predetermined criterion; recognizing a voice signal from a user of the multimedia device; searching audio data corresponding to the recognized voice signal in the DB And displaying the web page search result when audio data corresponding to the recognized voice signal is not present in the DB.

Description

TECHNICAL FIELD [0001] The present invention relates to a multimedia device for accessing a database according to speech recognition results, and a control method thereof. [0002] MULTIMEDIA DEVICE FOR ACCESSING DATABASE ACCORDING TO VOICE RECOGNITION AND METHOD FOR CONTROLLING THE SAME [0003]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia device, and more particularly, to a multimedia device and a control method thereof for accessing a database in accordance with speech recognition results. The multimedia device may be applied to, for example, a digital television (DTV), a network TV, an IPTV, a smart TV, a web TV, a mobile device or a smart phone.

Recently, voice recognition technology using a multimedia device such as a mobile phone or a TV is being discussed. However, according to the related art, the database of the reference value for the response is not automatically updated, so that there is a limit to always check only the same fixed answer.

In addition, since the TTS (Text To Speech) service according to the related art is provided only by the voice of a designated speaker or a default speaker, there is a problem that various multimedia services can not be provided to a user.

An embodiment of the present invention proposes a solution for solving the above-mentioned problems and automatically updating a database optimized for the query value of the user.

In addition, another embodiment of the present invention proposes a method of providing lane data to a user in case of a database search failure corresponding to a question value of a user.

Another embodiment of the present invention is to define a protocol for providing a service in cooperation with a mapped multimedia data without outputting only the result of the speech recognition service as simple text.

According to another aspect of the present invention, there is provided a method of controlling a multimedia device accessing a database according to a speech recognition result, the method comprising: extracting text data and audio data by scanning at least one channel; Storing data in a database (DB) according to a predetermined criterion; recognizing a voice signal from a user of the multimedia device; searching audio data corresponding to the recognized voice signal in the DB And displaying the web page search result when audio data corresponding to the recognized voice signal is not present in the DB.

Furthermore, the multimedia device accessing the database according to the speech recognition result according to an embodiment of the present invention may include an extraction module that scans at least one channel and extracts text data and audio data, A memory for storing data in a database (DB) according to a preset reference; a recognition module for recognizing a voice signal from a user of the multimedia device; and audio data corresponding to the recognized voice signal, And a display module for displaying a web page search result when audio data corresponding to the recognized voice signal is not present in the DB.

According to an embodiment of the present invention, there is provided a solution for automatically updating a database optimized for a user's query value.

According to another embodiment of the present invention, there is provided a method of providing lane data to a user in the event of a database search failure corresponding to a question value of a user.

Another embodiment of the present invention defines a protocol for providing a service in cooperation with the mapped multimedia data without outputting the result of the speech recognition service as simple text.

1 is a block diagram illustrating a multimedia device accessing a database according to speech recognition results according to an embodiment of the present invention.
FIG. 2 is a detailed view of the recognition module shown in FIG. 1; FIG.
3 is a diagram illustrating text data and audio data extracted according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a database for each speaker that maps and stores virtual query values corresponding to extracted text data according to an embodiment of the present invention. Referring to FIG.
FIG. 5 is a diagram showing a genre-specific database in which virtual query values corresponding to extracted text data are mapped and stored according to another embodiment of the present invention.
FIG. 6 is a diagram illustrating a database for each preferred channel in which virtual query values corresponding to extracted text data are mapped and stored according to another embodiment of the present invention. FIG.
7 is a diagram illustrating a process of inputting a user's voice signal using a multimedia device according to an embodiment of the present invention.
FIG. 8 is a diagram showing a processing process when multimedia data corresponding to a voice signal of a user input in FIG. 7 is stored in a DB.
9 is a diagram showing a processing process in the case where the multimedia data corresponding to the voice signal of the user input in FIG. 7 is not stored in the DB.
10 is a flowchart illustrating a method of controlling a multimedia device accessing a database according to a speech recognition result according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

The suffix "module" and " part "for components used in the following description are given merely for ease of description, and the" module "and" part "

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention, but these may vary depending on the intention or custom of a person skilled in the art or the emergence of new technologies. In addition, in certain cases, there may be a term arbitrarily selected by the applicant, in which case the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the entire contents of the specification.

1 is a block diagram illustrating a multimedia device accessing a database according to speech recognition results according to an embodiment of the present invention. Hereinafter, a process of processing a voice signal transmitted from a user and a process of processing a broadcast signal transmitted from a broadcasting station or a server will be described with reference to FIG.

The network interface 110 not only receives general A / V data for implementing a broadcast screen from a broadcast station, a server, etc., but also receives teletext data of each broadcast. The caption data may be referred to as text data and mapped with audio data. Therefore, text data can be output at appropriate timing based on the sync information of the audio signal.

The extraction module 120 scans at least one channel to extract text data and audio data, and the memory 130 stores the extracted text data and voice data in a database (DB) according to predetermined criteria have. In addition, additional services may be provided by additionally using EPG information (ex: broadcast time, program name, performer information, etc.) of the extracted broadcast.

In particular, according to one embodiment of the present invention, the problem of the prior art that provides the same answer to the user's voice signal is solved and the database is automatically updated. With reference to FIG. 3 to FIG. 6, will be described later in detail.

For example, it extracts the main keywords included in the text data of the broadcast signal, and accumulates a database of reference words (for example, weather, sports, news, traffic accidents, fire, etc.) Further, when the question mark is included in the text data of the above-mentioned broadcast signal, the following answer is designed to be preferentially stored in the database.

The recognition module 140 recognizes a voice signal from a user of the multimedia device 100 and the search engine 150 designs to search audio data corresponding to the recognized voice signal in the DB. The recognition module 140 will be described later in detail with reference to FIG.

The display module 180 is designed to display a web page search result when audio data corresponding to the recognized voice signal is not present in the DB. Of course, if the audio data corresponding to the recognized voice signal exists in the DB, the graphic data of the corresponding multimedia data is outputted through the display module 180, and the audio data of the corresponding multimedia data is output to the speaker 170 The controller 160 designates the controller 160 to output the control signal.

According to the prior art, if the user's voice can not be recognized or the associated response data is not stored in the memory, no output is output or simply an error message is displayed. On the other hand, according to the embodiment of the present invention described above, there is an advantage that a user can quickly access a web page screen in which a recognized keyword is input.

FIG. 2 is a detailed view of the recognition module shown in FIG. 1; FIG. Hereinafter, a process of recognizing a user's voice signal by a multimedia device according to an embodiment of the present invention will be described with reference to FIG. However, FIG. 2 is an embodiment, and may be designed so that voice recognition is performed by another process.

Speech recognition techniques are generally based on pattern matching techniques. That is, characteristic parameters of a recognition target word or a phoneme are previously stored in a memory of a computer, and when a speaker's voice is input, the characteristic is extracted and analyzed, and the similarity with the characteristics of the previously stored word or phoneme is measured, And outputs it as a recognition result.

In addition, typical examples of recognition algorithms include Dynamic Time Warping (DTW), Hidden Markov Model (HMM), Neural Network, and the like.

Dynamic time warping (DTW) is a method of determining similarity by comparing a representative pattern with a given input pattern. Even if the same word is uttered, it has a different duration depending on the speaker, emotion, and surrounding environment. Therefore, the non-linear optimization of the discrepancy of the continuous length is performed based on the partial optimization and performs the overall optimization . The method is mainly used for small isolated word recognition of recognition target vocabulary, and it is easy to change the task contents of speech recognition system because it can easily create reference pattern.

The Hidden Markov Model (HMM) is the most widely used method for speech recognition with high recognition rate and convenient learning performance. It consists of a transition probability modeling the temporal change of speech and an output probability modeling the spectrum change . Rather than using the similarity between the input pattern and the output pattern, the similarity of the model is calculated by using a probabilistic estimate with a given model.

Neural Network is a method of using a neural network modeling the process of encrypting and decrypting information in the human brain. Artificial neurons with a simple parietal function are interconnected densely and perform various functions according to the connection between them. Parallel computation ability, fault tolerance, low number and learning ability.

On the other hand, the conventional speech processing apparatus recognizes the input speech based on a preset vocabulary and grammar, and responds to the recognized result. Therefore, according to the prior art, there is a limitation in that it is possible to accurately recognize only the voice without a word in the grammar.

A method for solving such a problem will be described with reference to FIG. However, implementation of speech recognition in another embodiment is also within the scope of the present invention, and the scope of the right should be interpreted according to the matters described in the claims.

First, the entity extracting unit 241 extracts at least one entity information and an upper entity name corresponding to the entity information from the input voice. For example, the voice signal is "the drama that Kim Tae-hee comes out of today?" .

The individual information means the sub-information constituting the contents of the inputted voice. Eventually, the speech may consist of one or more entity information. For example, it can be divided into "today", "Kim Tae-hee", "go", "come out", "drama"

Finally, all entity information in each sentence constituting the voice is semantically connected to each other, but if the voice is input which is not understood by the normal person, the entity information is not semantically connected.

On the other hand, an entity name means a category of all identifiable things in reality. These object names are organized hierarchically as the tree structure. This hierarchical semantic system can be called "ontology".

For example, one of the assignable objects, "today ", may be a visual name, a date, or a broadcast date. That is, the category to which "today" belongs is "time "," date " In this case, "time" is a higher concept than "date" or "broadcast date ", and" date "

As described above, a predetermined hierarchical relationship is established between a plurality of entity names. Thus, "time" is a higher entity name than "date ", and" date "is a higher entity name than" broadcast date ".

As a result, in the above-described example, the entity extracting unit 241 can extract entity information called "Today, Kim Tae-hee, Drama ", and the " date, star actor, broadcast genre, Can be extracted. The memory 230 stores a plurality of pieces of entity information and entity names in a database, and is designed to be updated periodically.

The entity extracting unit 241 detects a portion of the input voice signal that matches the entity information stored in the memory 230 and extracts the detected entity information from the memory 230.

In addition, the entity extracting unit 241 extracts an upper entity name corresponding to the extracted entity information from the memory 230. In the example described above, when the entity information "today" is read, the entity extracting unit 241 reads the name of the parent entity "date".

On the other hand, in the memory 230, not only a plurality of parent entity names but also a plurality of child object names belonging to respective parent entity names are stored in advance.

The focus determining unit 242 determines the focus using the individual information having the content requiring the response among the extracted individual information. In the above-described example, the object extracting unit 241 reads out the object names of the "today", "Kim Tae-hee", and "drama" parts by focusing on the noun portion in the "

The mapping unit 243 combines the three words read by the focus determination unit 242 and transmits the combination of the three words to the verification unit 244. The verification unit 244 accesses the memory 230 to obtain EPG information Lt; / RTI >

In other words, it reads the information about the broadcasting programs broadcasted today in the drama having the actor name Kim Tae-hee. Therefore, the recognition unit 245 is designed to recognize only the information on the specific drama among the drama appearing by Kim Tae-hee, among the programs scheduled to be broadcast today, for example.

3 is a diagram illustrating text data and audio data extracted according to an embodiment of the present invention. Hereinafter, a process using text data and audio data extracted according to an embodiment of the present invention will be described with reference to FIG.

According to recent data broadcasting, text data corresponding to audio data provided by drama, news, and entertainment programs are provided together. Therefore, it is possible to map and extract and store audio data and text data.

For example, as shown in FIG. 3, if a scenario of a broadcast program called Entertainment News is provided through a broadcast signal, texts and audio data are stored together with the contents of an answer by an actor Kim Tae-hee. In particular, it is possible to construct a database in which only the text data before each answer is terminated with a question mark, thereby guiding the user to provide a service of a type in which the user answers questions afterwards.

As described above with reference to FIG. 1 and FIG. 2, when the voice signal recognized by the user is the same as the question of the reporter shown in FIG. 3 (for example, how are they nowadays? , It is possible to output the contents of each of the actors Kim Tae-hee responded to by a specific actor's voice.

That is, in the case of such a design, there is an advantage that a voice service is not performed only by a fixed voice of a person, as in the prior art, and an audio service is provided by the voice of a desired person's favorite party. Therefore, it is possible to provide an interactive service as if the conversation is in a virtual reality.

FIG. 4 is a diagram illustrating a database for each speaker that maps and stores virtual query values corresponding to extracted text data according to an embodiment of the present invention. Referring to FIG. Hereinafter, with reference to FIG. 4, a description will be made of a process of establishing a database for each speaker to provide an interactive voice service according to an embodiment of the present invention.

As described above with reference to FIGS. 1 to 3, it is assumed that text data corresponding to audio data and a speaker are mapped through a broadcast signal.

First, the database 410 of the first celebrity designates the information about the age and the residence with the virtual query value and stores it by using the text data included in the broadcast signal.

Furthermore, the database 420 of the second celebrity is designed to map and store the information on the target and settlement locations together with the virtual query value using the text data included in the broadcast signal.

The reason why the virtual query values described above are mapped and stored is to decide whether or not to output the answers according to the degree of mapping between the keyword included in the voice signal recognized by the user and the virtual query value.

For example, if the answer matches only 75% of the recognized keyword, the corresponding answer is output in the form of audio data, etc. If the answer is below the 75% match rate, the recognized keyword is displayed in the form . Therefore, there is an advantage that the probability of transmitting erroneous information to the user can be minimized.

FIG. 5 is a diagram showing a genre-specific database in which virtual query values corresponding to extracted text data are mapped and stored according to another embodiment of the present invention. Hereinafter, a process for constructing a database for providing an interactive voice service according to a genre will be described with reference to FIG.

As described with reference to FIGS. 1 to 3, it is assumed that text data and genre information corresponding to audio data are mapped through a broadcast signal.

First, the database 510 of the first genre is designed to use the text data included in the broadcast signal to map and store information about the baseball result and the NBA news together with the virtual query value.

Further, the database 520 of the second genre is also designed to store the audience rating and the main actor information of the specific drama together with the virtual query value using the text data included in the broadcast signal.

The reason why the virtual query values described above are mapped and stored is to decide whether or not to output the answers according to the degree of mapping between the keyword included in the voice signal recognized by the user and the virtual query value.

In this manner, the audio and text data corresponding to the recognized user's voice signal are not designed to be searched in all the databases, but the genre database is searched for preferentially based on the genre information included in the recognized voice signal, There is an advantage that speed can be improved.

FIG. 6 is a diagram illustrating a database for each preferred channel in which virtual query values corresponding to extracted text data are mapped and stored according to another embodiment of the present invention. FIG.

In FIGS. 4 and 5, it is assumed that the database is automatically constructed without any special action of the user. However, an embodiment that more clearly reflects the needs of the user will be described with reference to Fig. 6, which will be described below.

In the case of a multimedia device capable of processing login information, a preferred channel can be set for each user using the same multimedia device. For example, the user A is likely to view channels 1, 2, and 3 for a long period of time while setting channels 1, 2, and 3 as a preferred channel, and the user B may watch channels 5, 6, and 7 for a relatively long time by setting them as preferred channels.

Therefore, it is preferable that the search is preferentially performed from the preference channel DB of the logged-in user, rather than searching all databases for answers corresponding to recognized voice signals of the user.

The preference channel DB 610 shown in FIG. 6 restricts only the text data and the audio data collected by the broadcast signal received from the preference channel set by the specific user among the broadcast signals received from the plurality of channels have. On the other hand, the other DB 620 stores text data and audio data included in a broadcast signal received from a channel other than the preferred channel.

For example, a user who registers 12 sports channels as a preferred channel is more likely to receive and receive sports-related audio services. Therefore, prior to the search for another database, the text / audio data corresponding to the voice signal recognized by the user (for example, the current A football game relay time?, The record of the B baseball player? It is designed to search preferentially in the channel DB.

7 is a diagram illustrating a process of inputting a user's voice signal using a multimedia device according to an embodiment of the present invention. Hereinafter, with reference to FIG. 7, a process of implementing a service to which the present invention is applied by a user watching a broadcast of a specific channel will be described.

First, the multimedia device 700 according to an embodiment of the present invention shown in FIG. 7 can be repeated with reference to the block diagrams shown in FIG. 1 and FIG. Of course, it is also within the scope of the present invention to apply other speech recognition technologies according to the needs of those skilled in the art.

Also, the multimedia device 700 may be implemented as a network TV, a DTV, a smart TV, a web TV, an Internet TV, HBBTV, or the like.

The user 701 watching an arbitrary broadcast channel screen 702 using the multimedia device 700 generates a voice signal irrelevant to the currently viewed broadcast screen 702 (for example, what is Kim Tae-hee?) do.

A method of analyzing and recognizing a voice signal by the multimedia device 700 according to an embodiment of the present invention has been fully described with reference to FIGS. 1 and 2, and a repeated description thereof will be omitted.

Further, the multimedia device 700 searches at least one database to determine text data and audio data corresponding to the voice signal generated by the user 701. [ In addition, the database has been fully described in Figs. 3 to 6 above.

However, an embodiment (FIG. 8) in which the multimedia device 700 finds corresponding text data and audio data corresponding to a voice signal and an embodiment (FIG. 9) in which the multimedia data 700 is not found will be described separately.

FIG. 8 is a diagram showing a processing process when multimedia data corresponding to a voice signal of a user input in FIG. 7 is stored in a DB. Hereinafter, the process of outputting the retrieved text data and audio data will be described in detail with reference to FIG.

7, the multimedia device 800 shown in FIG. 8 displays the graphic image data 802 corresponding to the voice signal of the user 801 instead of the previous broadcast screen 702. In FIG. For example, it may be a movie screen shot or a moving picture included in a keyword of a recognized voice signal.

In FIG. 8, the entire screen is changed to a completely different graphic as compared with FIG. 7. However, it is also possible to simultaneously output the existing broadcasting screen 702 and the graphic image data 802 in a PIP format And will do so.

In addition, the multimedia device 800 according to another embodiment of the present invention outputs audio data corresponding to a voice signal of the user 801 recognized through at least one speaker 804a and 804b. As described above with reference to Figs. 3 to 6, the audio data is stored in the database together with the text data.

In addition, the multimedia device 800 according to another embodiment of the present invention displays text data corresponding to audio data on one side 803 of the screen, so that even in an environment where audio data can not be heard, There are advantages.

9 is a diagram showing a processing process in the case where the multimedia data corresponding to the voice signal of the user input in FIG. 7 is not stored in the DB. Hereinafter, with reference to FIG. 9, a solution when the retrieved text data and audio data can not be output will be described.

The biggest problem of the conventional speech recognition service is that when there is an error in speech recognition or a corresponding answer does not exist in the database, there is no response or simply an error message is displayed.

9, the multimedia device 900 according to an exemplary embodiment of the present invention automatically displays a specific website connection screen 901 (FIG. 9) when a response corresponding to the recognized speech signal is not stored in the database ). That is, it is assumed that the multimedia device 900 is connected to an IP network through a network interface, for example, as shown in FIG.

In addition, the multimedia device 900 is designed to input only specific keywords (e.g., Kim Tae-hee) among the automatically recognized keywords in the search input window 902. Of course, it is also within the scope of the present invention to design the voice signal specialist (ex: now Kim Tae-hee?) Input by the user in the input window 902 as it is in Fig.

Accordingly, the user can be quickly provided with relevant information even when the database of the multimedia device 900 is not established. Further, according to another embodiment of the present invention, the user can directly store the audio data provided by the website in the database It can also be designed.

10 is a flowchart illustrating a method of controlling a multimedia device accessing a database according to a speech recognition result according to an embodiment of the present invention. Fig. 10 may be supplementary interpreted with reference to the description of Figs. 1 to 9 above.

A method of controlling a multimedia device accessing a database according to a speech recognition result according to an embodiment of the present invention includes scanning at least one channel (S1010) and extracting text data and audio data (S1020) .

Further, the extracted text data and voice data are stored in a database (DB) according to a preset reference (S1030), and a voice signal is recognized from the user of the multimedia device (S1040).

Then, the audio data corresponding to the recognized voice signal is searched in the DB (S1050), and if the audio data corresponding to the recognized voice signal is not present in the DB, the web page search result is displayed (S1060).

The database (DB) is designed to map and store virtual query values corresponding to the extracted text data, for example. Can be interpreted with reference to Figures 3 to 6 above.

If the database DB is constructed for each speaker of the audio data, the step S1050 may include the steps of: determining whether a person keyword included in the recognized speech signal corresponds to a specific speaker of the DB; And outputting audio data of a specific speaker through a speaker.

If the database (DB) is constructed for each genre of audio data, the step S1050 may include extracting a genre keyword included in the recognized voice signal, and searching for a specific database corresponding to the extracted genre keyword in a limited manner .

If the user has logged in, the step S1050 is to search the database of the preferred channel previously set by the user with the highest priority, and if there is no audio data corresponding to the recognized voice signal, And retrieving the remaining databases in descending order.

Therefore, according to an embodiment of the present invention, a realistic interaction service is provided. For example, by letting the speaker of the actual broadcast speak the story of the story, it gives the effect of talking to the person.

Furthermore, according to another embodiment of the present invention, a database for voice service is automatically updated. Therefore, there is an advantage in that it is possible to avoid stereotypical responses stuck in existing frameworks.

1 to 10 are described for convenience of explanation, it is obvious that the present invention incorporates some of the features of the drawings to implement other embodiments.

In this specification, both the invention and the invention of the method are explained, and the description of the two inventions can be supplementarily applied as necessary.

The method inventions according to the present invention can all be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium.

The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

110: Network interface
120: Extraction module
130: memory
140: recognition module
150: Search engine
160: Controller
170: Speaker
180: Display module

Claims (10)

delete delete delete delete delete delete 1. A multimedia device for accessing a database in accordance with speech recognition results,
An extraction module for extracting text data and audio data by scanning at least one channel from a broadcast signal, the extraction module preferentially extracting a keyword having a high frequency of use from text data of the broadcast signal;
A memory for storing the extracted text data and audio data in a database (DB) according to a predetermined criterion, the preset reference being at least one of genre information of at least one channel scanned from the broadcast signal, Comprising speaker information of audio data;
A recognition module for recognizing a voice signal from a user of the multimedia device;
A search engine for searching audio data corresponding to the recognized voice signal in the DB; And
And a display module for displaying a web page search result when audio data corresponding to the recognized voice signal is not present in the DB,
The search engine comprises:
And preferentially searches the DB corresponding to the genre information or the speaker information for audio data corresponding to the recognized voice signal based on genre information or speaker information included in the recognized voice signal. 8. The method of claim 7,
The database (DB)
And maps the virtual query value corresponding to the extracted text data and stores the mapped query value. 9. The method of claim 8,
The database (DB)
And the multimedia data corresponding to the extracted text data is additionally mapped and stored. 10. The method of claim 9,
Controls to output audio data corresponding to the recognized voice signal through a speaker,
A controller for controlling the display module to output the multimedia data corresponding to the recognized voice signal,
Lt; / RTI >

Images