KR102122918B1 - Interactive question-anwering apparatus and method thereof - Google Patents

Abstract

The present invention proposes a multi-modal based question-and-answer method that can improve a conventional unnatural form of a questionnaire, prevent a case where a question is impossible without knowing additional information, and ask a question as if talking to a person. In addition, it provides an environment that can be used in the advertising market because content such as images, videos, and audio can be exposed to potential product buyers in the form of a question-and-answer service.

Description

INTERACTIVE QUESTION-ANWERING APPARATUS AND METHOD THEREOF}

The present invention relates to an interactive question-and-answer apparatus and method for providing a correct answer to a question.

In a conventional question-answering apparatus, when hint information for finding a response in a query is insufficient in a process of obtaining a response to an input query, it is often impossible to obtain an accurate response. Therefore, in a conventional question-and-answer apparatus, a process of generating a query to include the hint information is required prior to the process of obtaining a response to the query. For example, in a situation where a questioner views a particular piece of work in an art museum, when the questioner wants to know the production year of the specific piece, the questioner may ask a question to include a hint related to the name of the painter of the specific piece or the title of the specific piece Must be created and entered into the device. If the questioner does not know exactly the artist's name or the title of the specific work, an appropriate query cannot be generated.

In order to solve this problem, a conventional question-and-answer apparatus is leading to generate an appropriate query through a conversation with a questioner over several turns. For example, if the questioner inputs a query including a short product name into the Q&A device, the Q&A device performs a process of confirming to the questioner the name of the product with the highest similarity to the product name. The process is repeated through dialogue over several turns between the interrogator and the Q&A device until the correct product is found.

As described above, in the conventional question-and-answer apparatus, the query is completed through a natural conversation between the questioner and the question-and-answer apparatus even when the questioner does not know the exact information related to the object to be known.

However, such an interactive query input method is inconvenient due to a dialogue repeatedly performed over several turns between a questioner and a question and answer device.

An object of the present invention for solving the above-described conventional problems is to create an interactive query between a questioner and a question-and-answer device without generating a query statement, and to provide an accurate response statement using a natural query statement as it is. It is to provide a response device and a method.

An interactive query response method in a user terminal according to an aspect of the present invention for achieving the above object, the computer processor, the multimedia content from the server and the meta for multiple entities of interest to the user in the multimedia content Receiving information; Receiving, by the computer processor, a query for the entity selected by the user among the plurality of entities from the multimodal interface; The computer processor extracting identification information of the entity selected by the user from the meta information; And the computer processor transmits the identification information of the entity and the query to the server, and as a response to the query, the response including the correct answer candidate constrained by the identification information of the entity is the And receiving from the server.

In an interactive query response method in a server according to another aspect of the present invention, the computer processor is provided with the meta information including identification information for a plurality of entities of interest to the user in the multimedia content and the identification information. Generating attribute information; Transmitting, by the computer processor, the multimedia content and the meta information to a user terminal; Receiving, by the computer processor, a query statement for the entity selected by the user among the plurality of entities and identification information of the entity selected by the user extracted from the meta information from the user terminal; And generating, by the computer processor, a response to the query based on the attribute information given to the identification information of the entity selected by the user, and transmitting the response to the user terminal.

In an interactive question-and-answer apparatus including a server according to another aspect of the present invention, the server includes multimedia content, meta information including identification information for a plurality of entities of interest to the user in the multimedia content, and the identification A storage unit for storing attribute information assigned to the information; And transmitting the multimedia content and the meta information to a user terminal, and receiving a query statement for the entity selected by the user among the plurality of entities and identification information of the user selected entity extracted from the meta information from the user terminal. And a computer processor that generates a response to the query based on attribute information assigned to the identification information of the entity selected by the user and transmits the response to the user terminal.

According to the present invention, by providing the correct response to the natural query directly, the cumbersome task of changing the natural query to an unnatural query through repeated dialogue between the questioner and the device to include hints related to the response Can be omitted. Furthermore, it is possible to provide an accurate response to a natural query that has no hint related to the response or has very little information.

1 is a block diagram of an interactive question-and-answer apparatus according to an embodiment of the present invention.
FIG. 2 is a configuration diagram of the user terminal illustrated in FIG. 1.
FIG. 3 is a configuration diagram of the multi-modal interface illustrated in FIG. 2.
FIG. 4 is a configuration diagram of the content server shown in FIG. 1.
5 is an example of information stored in the storage unit shown in FIG. 4.
FIG. 6 is a diagram illustrating the configuration of a question and answer server shown in FIG. 1.
7 to 9 are diagrams showing a data structure of meta information according to embodiments of the present invention.
FIG. 10 is a view for explaining screen coordinates illustrated in FIG. 7.
11 is a view showing a data structure of attribute information according to an embodiment of the present invention.
12 is a flowchart illustrating a method for answering questions in a user terminal according to an embodiment of the present invention.
13 is a flowchart illustrating a method for answering questions in a server according to an embodiment of the present invention.
14 is a detailed flowchart of step S240 shown in FIG. 13.

It will be described in detail focusing on the parts necessary to understand the operation and operation according to the present invention. In describing the embodiments of the present invention, descriptions of technical contents well known in the technical field to which the present invention pertains and which are not directly related to the present invention will be omitted. This is to more clearly communicate the subject matter of the present invention by omitting unnecessary description.

In addition, in describing the components of the present invention, different reference numerals may be assigned to components having the same name according to the drawings, and the same reference numerals may be assigned to different components. However, even in such a case, it does not mean that the corresponding component has different functions according to embodiments, or does not mean that it has the same functions in different embodiments, and the function of each component is the corresponding embodiment You should judge based on the description of each component in.

Prior to the description of the embodiments of the present invention, the term entity referred to throughout the specification is defined.

'Entity' is information that is expected to be of interest to a user in multimedia content including images, audio, and videos, and is not included in multimedia content.

Exemplary types of entities include place name, event, year of production, person, household appliance, clothing, actor name, place name, producer, clothing worn by actors, shoes, bags, price, color, etc. The present invention is not limited thereto, and various example types may be further included according to types of contents.

Hereinafter, an interactive query response device according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram of an interactive question-and-answer apparatus according to an embodiment of the present invention.

Referring to FIG. 1, an interactive query response device according to an embodiment of the present invention includes a user terminal 100, a communication network 200, and a server 300.

The user terminal 100 is provided with a communication function to be able to access the communication network 200, it is possible to communicate with the server 300 through the communication network 200.

The user terminal 100 may receive content and meta information about the content from the server 300 through the communication network 200. The content may be multimedia content in various fields such as politics, economy, society, education, broadcasting, entertainment, sports, home shopping, and the like. The multimedia content may include image content, video content, and audio content. The meta information is information on a plurality of entities predicted to be of interest to the user in the content, and may be referred to as'meta data'. The meta information may include identification information identifying a plurality of entities. The identification information may be configured to have attributes of multi-modal information output from the multi-modal interface provided in the user terminal 100.

The user terminal 100 recognizes a multi-modal input input from the multi-modal interface to generate a natural query for the entity selected by the user among the multiple entities.

Exemplary types of multi-modal input include voice input, keyboard (or pen) input, mouse input, pen input, touch screen input, gesture input, etc. In the present invention, for convenience, voice input or keyboard (or pen) input is used. It is assumed that a natural query is generated from the recognized results. However, the present invention is not limited thereto, and may be implemented using two or more different inputs.

The user terminal 100 extracts identification information of a selected entity from the meta information received from the server 300 using the multi-modal interface.

The user terminal 100 transmits the identification information of the extracted entity and the natural query to the server 300 through the communication network 200, and receives a response to the query from the server 300. It is received through the communication network 200.

The identification information of the entity serves to greatly increase the probability of a correct answer of a response to the query in the server 300. That is, the server 300 constrains the correct answer candidate with the attribute given to the identification information of the entity, and generates a response message including the correct answer candidate. Accordingly, the user terminal 100 of the present invention can omit unnecessary work of changing a natural query statement into an unnatural query statement so that a hint for an entity is included in order to increase a probability of correct answer of the response statement as in the related art.

The server 300 includes a content server 310 and a question and answer server 330.

The content server 310 may be a server operated by a content provider.

The content server 310 transmits the multimedia content produced by the content provider to the user terminal 100 through the communication network 200.

The content server 300 generates meta information about a plurality of entities predicted to be of interest to the user in the multimedia content, and transmits the meta information to the user terminal 100 together with the multimedia content.

The content server 310 receives, from the user terminal 100, identification information of an entity selected by the user using a multi-modal interface and a query for the entity selected by the user among the plurality of entities.

The query and response server 330 receives the query and identification information of the entity, the content server 310, and generates a response to the query based on attribute information assigned to the entity's identification information. .

The question and answer server 330 provides the response text to the user terminal 110 through the content server 310.

FIG. 2 is a configuration diagram of the user terminal illustrated in FIG. 1.

Referring to FIG. 2, the user terminal 100 may be implemented as a computing device having a communication function. The computing device may be, for example, a smart phone, a tablet, a laptop, a desktop PC, a wearable device, a smart TV having a communication function, a household appliance such as a smart washing machine, or a smart refrigerator. Other shopping malls, tourist attractions, museums, etc. It may be a kiosk (kiosk).

The user terminal 100 that can be implemented as the computing device includes a computer processor 110, a multi-modal interface 120, a communication interface 130, a memory 140, a storage unit 150, a voice output unit 160 And a bus 170 connecting them.

The computer processor 110 controls the overall operation of the user terminal 100.

Computer processor 110 may include at least one general purpose processor that executes multiple algorithms. The general-purpose processor may include a graphics processor specialized in graphics computation. The algorithm may include, for example, an algorithm related to speech recognition, speech synthesis, image recognition, and the like. In the present invention, there is no feature in limiting the algorithm, and the description thereof is replaced by a known technique.

The computer processor 110 analyzes multi-modal input information output from the multi-modal interface 120 to recognize the multi-modal input.

The computer processor 110 generates a natural query for the entity of interest to the user from the multimedia content transmitted from the server 300 based on the recognized multi-modal input.

In order to generate the query, the computer processor 110 may generate a query using multi-modal input including voice input and/or keyboard (pen) input.

For example, the computer processor 110 may recognize a user's speech voice based on a speech recognition algorithm, and generate a text-type query based on the recognition result.

As another example, the computer processor 110 may recognize a keyboard (pen) input, and generate a text type query based on the recognition result. In order to recognize keyboard input, the computer processor 110 may provide an input window for inputting a query on the display screen provided in the user terminal 100.

The computer processor 110 extracts identification information of an entity of interest to the user from meta information transmitted from the server 300 based on the recognition result of recognizing the multi-modal input.

In order to extract the identification information of the entity from the meta information, for example, the computer processor 110 receives the touch coordinates of the entity selected by the user from the multi-modal interface 120, and of the entity corresponding to the received touch coordinates The identification information can be extracted from the meta information.

As another example, the computer processor 110 calculates an input time at which the user's speech voice corresponding to the query is input from the multi-modal interface 120, and identifies the entity's identification information corresponding to the input time from the meta information. Can be extracted. Here, the input time at which the user's speech voice is input may be a time counted from the playback start time of the multimedia content (video content or audio content). As another example, the computer processor 110 calculates an input time at which a keyboard (or pen) input corresponding to the query is input from the multi-modal interface 120, and identification information of an entity corresponding to the input time. Can be extracted from the meta information.

The multi-modal interface 120 generates a plurality of multi-modal input information for an entity selected by a user among a plurality of entities included in multimedia content.

In order to generate a plurality of multi-modal input information, the multi-modal interface 120 is a voice input unit 120-1, a keyboard input unit 120-3, a pen input unit 120-5, as shown in FIG. It includes a mouse input unit 120-7, a touch screen input unit 120-9, and a gesture input unit 120-11.

The voice input unit 120-1 converts a user's speech voice corresponding to a query to digital voice input information, and although not shown, the speech collector of a microphone such as a microphone and the user's speech voice collected by the voice collector It may include an audio processor to convert the voice input information.

The keyboard input unit 120-3 may include a keyboard that allows a user to directly type a query for the selected entity.

The pen input unit 120-5 may include an electronic pen that allows a user to directly write a query for an entity selected by an input window provided on a display screen.

The mouse input unit 120-7 may include a mouse that allows a user to click a desired query from a list of query statements for the entity selected by the user. Here, the list of query statements may be provided by the server 300. The query statement list may be a list generated through pre-learning of expected query statements for an entity selected by a user in multimedia content.

The touch screen input unit 120-9 may include a touch screen or a display device equipped with a touch panel capable of providing touch coordinates of a user-selected entity.

The gesture input unit 120-11 may include a wearable device capable of providing a gesture input to the entity selected by the user, an acceleration sensor and a gyro sensor attached to the body, and a camera sensor detecting user movement. The gesture may be, for example, a finger gesture pointing to a specific entity in content displayed by the user on the display screen.

Referring again to FIG. 2, the communication interface 130 serves to interface the communication network 200 and the user terminal 100.

The communication interface 130 serves to convert data or information generated by the computer processor 110 according to a communication protocol defined in the communication network 200 and transmit the converted data to the server 300 through wired or wireless communication. Do it.

The memory 140 provides a working space, that is, a memory space, for the computer processor 110 to process and process information received from the multi-modal interface 120 and information received from the server 300. The memory 140 includes volatile and nonvolatile memory.

The storage unit 150 stores multimedia content and meta information received from the server 300.

The voice output unit 160 converts and outputs a response message received from the server 300 into voice. Such conversion may be performed based on a known speech synthesis algorithm.

FIG. 4 is a configuration diagram of the content server shown in FIG. 1.

Referring to FIG. 4, the content server 310 includes a computer processor 311, a memory 313, a storage unit 315, a communication interface 317, and an output unit 319.

The computer processor 311 controls the overall operation of the content server 310.

The computer processor 311 may include at least one general purpose processor that executes multiple algorithms. The general purpose processor may include a graphic processor specialized in processing graphics.

The computer processor 311 generates meta information on a plurality of entities of interest to the user in multimedia content, and stores the generated meta information in the storage unit 315.

The computer processor 311 generates attribute information for a plurality of entities of interest to the user in the multimedia content, and stores the generated attribute information in the storage unit 315.

To generate meta information, the computer processor 310 classifies a number of entities of interest to the user in multimedia content. Entity classification may be performed by an entity classification model. The entity classification model is a learning model that learns the correlation between multimedia content and entities expected to be of interest to the user in the multimedia content. In order to learn the correlation, a deep learning technique, which is a kind of machine learning, may be used.

The computer processor 311 configures meta information by assigning identification information to each of a plurality of entities classified according to the entity classification model.

The computer processor 311 configures attribute information including attribute names and attribute values in identification information assigned to each of a plurality of entities classified according to the entity classification model.

The computer processor 311 stores meta information and attribute information together with the multimedia content in the storage unit 315.

The storage unit 315, as shown in FIG. 5, is a storage 315-1 for storing multimedia content, a storage 315-3 for storing an entity classification model, and a storage for storing the meta information and the attribute information. (315-5).

The computer processor 311 controls the communication interface 317 to transmit the multimedia content stored in the storage unit 315 and meta information about the multimedia content to the user terminal 100.

The computer processor 311 transmits the multimedia content stored in the storage unit 315, meta information about the multimedia content, and attribute information corresponding to the meta information to the query response server 330, the communication interface 340 To control. Accordingly, the content server 310 and the question and answer server 330 share the multimedia content, meta information about the multimedia content, and attribute information corresponding to the meta information.

The communication interface 317 serves to interface the communication network 200 and the content server 310. The communication interface 317 converts the multimedia content and meta information about the multimedia content according to a communication protocol defined in the communication network 200 under the control of the computer processor 311, and the converted data is wired or It is transmitted to the user terminal 100 through wireless communication.

The computer processor 311 receives the identification information for the entity selected by the user and the query for the entity selected by the user from the user terminal 100 and transmits it to the query response server 330.

Meanwhile, the memory 313 provides a workspace in which programs, execution instructions, and the like used by the computer processor 310 can be executed to generate the meta information.

The output unit 319 may include a display device displaying meta information generated by the computer processor 310 to a server administrator and an audio device outputting audio.

6 is a configuration diagram of a question and answer server illustrated in FIG. 1.

Referring to FIG. 6, the question and answer server 330 receives identification information of an entity selected by a user from the content server 310 and a query for the entity selected by the user from the content server 310, and attributes assigned to the entity Based on the information, a response to the received query is generated, and the generated response is transmitted to the user terminal 100 through the content server 310. In this case, the query and response server 330 may directly transmit the generated response text to the user terminal 100 without going through the content server 310.

To this end, the question and answer server 330 includes a computer processor 331, a memory 333, a storage unit 335, a communication interface 337 and an output unit 339.

The computer processor 331 controls the overall operation of the question-and-answer server 330 and generates a response to the query received from the content server 310. At this time, the computer processor 331 generates a response to the query based on the attribute information corresponding to the identification information received together with the query from the content server 310.

To generate the response, the computer processor 331 executes a question-and-answer algorithm. That is, the question-and-answer algorithm executed by the computer processor 331 generates a response message based on a database in which attribute information given to the identification information received from the content server 310 is stored. The question and answer process processed based on the question and answer algorithm will be described in detail below.

The memory 333 provides an execution space of a question-and-answer algorithm executed by the computer processor 331, and includes volatile and nonvolatile memory.

The storage unit 335 stores meta information provided by the content server 310 and attribute information corresponding to the meta information.

The communication interface 337 serves to interface the query and answer server 330 and the content server 310.

The output unit 339 includes a display device displaying a response generated by the computer processor 331 and an audio device outputting audio.

Although the content server 310 and the question and answer server 330 are described in this embodiment as separate, they can be integrated into one server.

7 to 9 are diagrams illustrating a data structure of meta information generated by the content server shown in FIG. 1.

In FIG. 7, when the multimedia content is image content, a data structure of meta information is illustrated. The meta information in the image includes an image file path 71, screen coordinates 73 and 75 in which an entity included in the image is located on a display screen, a unique identifier of the entity (77, URI), and an entity attribute name (79). It includes. Here, the screen coordinates (73, 75), when defining a virtual rectangular area surrounding the entity, the upper left coordinates (73) corresponding to the upper left corner of the rectangular area and the lower right corner of the rectangular area And the corresponding lower right coordinate 75. FIG. 10 shows an example of the screen coordinates. The image displayed on the display screen 10 of the user terminal 100 is an image in which three bags are displayed on the display stand 12, and in this image, the entity is a bag. . When the touch coordinates touched on the display screen 17 by the user to select a bag located in the middle of the three bags are located within the rectangular area including the upper left coordinate 73 and the lower right coordinate 75, the user The terminal recognizes the bag located in the center as an entity selected by the user, and extracts a unique identifier (URI_50) of the recognized entity from meta information. The user terminal 100 transmits the extracted unique identifier (URI_50) and a query for the entity to the server 300. In this case, the query may be'how much is that bag'. In the illustrated query, there is no hint information for the product number, size, and color of the bag, but the unique identifier (URI_50) serves as a substitute for the hint information. Accordingly, the user terminal 100 may omit the process of changing the illustrated natural query statement into an unnatural query statement to include hint information about the product number, size, and color of the bag.

In FIG. 8, when the multimedia content is video content, a data structure of meta information is shown. Meta information in the case of video content includes a video file path 81, a time section 83, 85 in which an entity of interest in the video content is played, a unique identifier 87, URI of the time section, and It includes the entity attribute name (89) assigned to the unique identifier (87). The time periods 83 and 85 include a playback start time 83 and a playback end time 85. If the user wants to know the place name appearing on the currently played video content, the query may be "where is that place", where the input time at which the user's speech voice corresponding to the query is input is the playback start time (83 ) And the playback end time (85), the user terminal 100 in the meta information shown in FIG. 8, time intervals (83, 85) defining the playback start time (83) and the playback end time (85) ) To extract the unique identifier (URI_100). The user terminal 100 transmits the query and the unique identifier (URI_100) to the content server 310.

In FIG. 9, when the multimedia content is audio content, a data structure of meta information is shown. Meta information in the case of audio content includes an audio file path 91, a time section 93, 95 in which an entity of interest in the audio content is played, a unique identifier 97, URI of the time section, and It includes the entity attribute name (99) assigned to the unique identifier (97). Similar to the method of extracting an identifier from the video content described above, the user terminal 100 may extract a query statement for the entity selected by the user from the audio content and a unique identifier for the entity from meta information. In the case of audio content, an example type of an entity may be a'song title', and an example type of a query may be "What song title is coming out now?".

FIG. 11 is a diagram illustrating a data structure of attribute information generated by the content server shown in FIG. 1.

Referring to FIG. 11, the attribution information is information shared by the content server 310 and the question-and-answer server 330 and includes an attribute name 23 and an attribute value 25 assigned to the unique identifier 21. . Two or more attribute names 23 and attribute values 25 may be assigned to the same unique identifier (URI_50). For example, an entity with a URI of'URI_50' has three attribute names consisting of a name, producer, and year of production, and three attributes consisting of "Last Judgment", "Michelangelo", and "Around the 16th Century" corresponding to each attribute name. The attribute value can be assigned, and the entity whose URI is'URI_300' can be assigned two attribute names consisting of a name and a singer and two attribute values consisting of "LET IT BE" and "Beatles" corresponding to each attribute name. have.

In the above-described embodiment, it is described that the content server 310 generates attribute information and provides it to the query and answer server 330, but on the contrary, the content server 310 generates the attribute information from the question and answer server 330 You can also provide In this case, meta information may also be generated by the question and answer server 330 and provided to the content server 310.

12 is a flowchart illustrating a method for answering questions in a user terminal according to an embodiment of the present invention, and it is assumed that the subject of each step below is the computer processor 110 provided in the user terminal 110.

Referring to FIG. 12, first, in step S110, the computer processor 110 receives multimedia content and meta information about the multimedia content from the server 300. Here, the meta-information is information on a plurality of entities of interest to the user in the multimedia content, and is information that can identify the plurality of entities.

Subsequently, in step S120, the computer processor 110 receives a query for the entity selected by the user from among the plurality of entities from the multi-modal interface 120. The query may be provided by voice input, keyboard input and pen input included in the multi-modal interface.

Subsequently, in step S130, the computer processor extracts identification information of the entity selected by the user from the meta information. An example of extracting the identification information of an entity from the meta information is a process in which the computer processor receives multi-modal input information identifying the entity selected by the user from the multi-modal interface, and the multi-modal within the meta information And searching for identification information of the entity corresponding to the input information and extracting the searched identification information from the meta information. Another example of extracting the identification information of the entity from the meta information, when the multimedia content is video content, calculates an input time for inputting a user's speech voice corresponding to the query based on a playback start time of the video content And extracting the identifier of the entry played at the calculated input time from the meta information. Another example of extracting the identification information of the entity from the meta-information is a process of calculating an input time at which a user voice corresponding to the query is input based on a playback start time of the audio contents from the multi-modal interface and the And extracting the identifier of the entry included in the audio content played at the input time from the meta information.

Subsequently, in step S140, the query statement and the identification information of the entity are transmitted to a server.

Subsequently, in step S150, the computer processor receives a response to the query from the server. At this time, the response statement includes a plurality of correct candidate candidates restricted by the identification information of the entity or the attribute information assigned to the identification information of the entity.

13 is a flowchart illustrating a method for answering a query in a server according to an embodiment of the present invention. For convenience of description, the following steps are one server in which the content server 310 and the query response server 330 are integrated. It is assumed to be performed in. However, when classifying the performance subjects of the following steps, the performance subjects of steps S210 to S230 may be the content server 310 shown in FIG. 1, and the performance subjects of steps S240 and S250 are question and answer shown in FIG. 1 It may be a server 330.

Referring to FIG. 13, first, in step S210, the server 300 generates meta information and attribute information for multimedia content. The meta information includes identification information for a plurality of entities of interest to the user in multimedia content. The method of generating the meta information includes the process of classifying a plurality of entities expected to be of interest to the user in the multimedia content using a pre-trained entity classification model, and identifying each of the plurality of classified entities Generating the identification information to have the attribute of the multi-modal input information and comparing the multi-modal input information of the multi-modal interface with the information and generating the meta information configured to include the generated identification information. do. Exemplary types of identification information included in the meta information include a unique identifier of each of the plurality of entities, screen coordinates in which each of the plurality of entities is displayed on a display screen of the user terminal, and video content including the plurality of entities is played. It may include a time period, a time period in which audio content including the plurality of entities is played, and an attribute name indicating the attribute of each of the plurality of entities. The attribute information may include an attribute name and an attribute value assigned to the unique identifier (URI). An example of the attribute name and attribute value is replaced with the description with reference to FIG. 11.

Subsequently, in step S220, the server 300 transmits the multimedia content and the meta information to the user terminal 100.

Subsequently, in step S230, the server 300 receives a query statement for the entity selected by the user from among the plurality of entities, and identification information of the entity selected by the user extracted from the meta information from the user terminal.

Subsequently, in step S240, the server 300 generates a response to the query based on the attribute information (attribute name and attribute value) assigned to the identification information of the entity selected by the user. That is, the server 300 generates a response statement including a correct answer candidate constrained by attribute information assigned to the identification information of the entity selected by the user.

Next, in step S250, the server 300 transmits the response to the user terminal 100.

14 is a detailed flowchart of step S240 shown in FIG. 13.

Referring to FIG. 14, in step S240-1, the server 300 analyzes a query statement transmitted from the user terminal 100, and recognizes a question-oriented vocabulary in the query statement based on the analysis result.

Query analysis may be performed based on a language processing algorithm including, for example, morphological analysis, syntax analysis, semantic analysis, and pragmatic analysis. . Since the present invention is not characteristic of such a language processing algorithm, the description thereof is replaced by known technology.

The question-oriented vocabulary can be defined as a word indicating the object of the query. For example, in the question "What is the year of production of this picture?", the question-oriented vocabulary is'Picture', and in the question "How much is that red bag?", the question-oriented vocabulary is'Bag', and "Here In the question "Where are you?", the question-oriented vocabulary is "Here", and in the question "Who is the singer of the song coming out now?"

Subsequently, in step S240-3, the server 300 determines the attribute of the question-oriented vocabulary based on the identification information transmitted from the user terminal 100. For example, the server 300 searches the database in which the attribute information shown in FIG. 11 is stored and searches for the same identifier as the identifier (URI) included in the identification information transmitted from the user terminal 100. When the same identifier is identified in the database, the attribute information assigned to the identified identifier, that is, the attribute value (25 in FIG. 11) is determined as the attribute of the question-oriented vocabulary.

If there are multiple attribute values assigned to the identified identifier, all of the multiple attribute values are determined as attributes of the question-oriented vocabulary. For example, if the identifier transmitted from the user terminal 100 is URI_50, and URI_50 is assigned three attributes consisting of'name','producer', and'production year', as shown in FIG. 11, 3 All of the dog's attributes are determined as attributes of the question-oriented vocabulary.

Subsequently, in step S240-5, the server 300 selects a word representing the determined attribute as a candidate for a correct answer, and generates a response message including the selected candidate for a correct answer. When there are a plurality of words representing the determined attribute, a plurality of correct candidates may be selected. A plurality of response statements each including a plurality of correct candidate candidates may be transmitted to the user terminal 100. The user terminal provides a plurality of response texts to the user through the display screen, and the user selects the correct answer desired by the displayed response text.

The embodiments described above are examples, and those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical spirits within the equivalent range should be interpreted as being included in the scope of the present invention.

Claims (18)

In a method for answering questions at a user terminal including a computer processor communicating with a server through a communication network,
The computer processor receiving, from the server, multimedia content and meta-information for a plurality of entities expected to be of interest to the user in the multimedia content;
Receiving, by the computer processor, a query for the entity selected by the user from among the plurality of entities from a multimodal interface;
The computer processor extracting identification information of the entity selected by the user from the meta information; And
The computer processor transmits the identification information of the entity and the query to the server, and as a response to the query, the response including the correct answer candidate limited by the identification information of the entity is sent to the server It includes; receiving from;
The meta information,
The server classifies a plurality of entities using a pre-trained entity classification model, and generates the identification information for each of the classified entities based on the multi-modal interface. How to answer questions.
In claim 1, The extraction step,
Receiving multi-modal input information identifying the entity selected by the user from the multi-modal interface;
Searching for identification information of the entity corresponding to the multi-modal input information in the meta information; And
Extracting the retrieved identification information from the meta information
Interactive question and answer method in a user terminal that includes.
The method of claim 1, wherein when the multimedia content is image content displayed on a display screen of the user terminal, the extracting step includes:
Receiving touch coordinates of the entity selected by the user from the multi-modal interface; And
Extracting the identification information of the entry corresponding to the touch coordinates from the meta information
Interactive question and answer method in a user terminal that includes.
In claim 1, If the multimedia content is video content, the extracting step,
Calculating an input time at which a user voice corresponding to the query is input based on a playback start time of the video content from the multi-modal interface; And
Extracting identification information of an entry reproduced at the calculated input time from the meta information;
Interactive question and answer method in a user terminal that includes.
In claim 1, If the multimedia content is audio content, the extracting step,
Calculating an input time at which a user voice corresponding to the query is input based on a playback start time of the audio content from the multi-modal interface; And
Extracting identification information of an entry included in the audio content played at the input time from the meta information
Interactive question and answer method in a user terminal that includes.
In claim 1, The identification information,
The unique identifier of each of the plurality of entities, the coordinates of a region in which each of the plurality of entities are displayed on the display screen of the user terminal, a time period during which video content including the plurality of entities is played, and the plurality of entities And a time period during which the audio content is played and an attribute name indicating the attribute of each of the plurality of entities.
In the interactive question-and-answer method in a server including a computer processor communicating with a user terminal through a communication network,
Generating, by the computer processor, meta information including identification information for a plurality of entities of interest to the user in the multimedia content and attribute information given to the identification information;
Transmitting, by the computer processor, the multimedia content and the meta information to a user terminal;
Receiving, from the user terminal, a query statement for the entity selected by the user among the plurality of entities and identification information of the entity selected by the user extracted from the meta information; And
And generating, by the computer processor, a response to the query based on the attribute information given to the identification information of the entity selected by the user, and transmitting the response to the user terminal,
The step of generating the meta information,
Classifying a plurality of entities expected to be of interest to the user in the multimedia content using a previously learned entity classification model;
Generating the identification information for each of the classified plurality of entities based on a multi-modal interface; And
Generating the meta information including the generated identification information
Interactive question-and-answer method in the server comprising a.
delete In claim 7, The step of generating the identification information based on the multi-modal interface,
Generating the identification information to have an attribute of the multi-modal input information, in order to compare the multi-modal input information and the identification information output from the multi-modal interface; And
Generating the meta information to include the generated identification information
Interactive question-and-answer method in the server comprising a.
In claim 7, the identification information,
Each unique identifier of the plurality of entities, a screen coordinate in which the plurality of entities are located on a display screen, a time period in which entities included in video content are played, a time period in which entities included in audio content are played, and the plurality of entities Interactive question-and-answer method in the server that includes an attribute name indicating each attribute.
In claim 7, The step of transmitting the response to the user terminal,
Analyzing a query statement transmitted from the user terminal, and recognizing a question-oriented vocabulary in the query statement;
Determining attribute information of the question-centered vocabulary based on identification information transmitted from the user terminal;
Selecting the attribute information as a candidate for the correct answer and generating the response statement including the selected candidate for correct answer
Interactive query response method in the server that includes.
In claim 11, The step of determining the attribute information,
Searching the database in which the attribute information is stored, and searching for the same identifier as the identifier included in the identification information transmitted from the user terminal; And
If the same identifier is identified in the database, determining attribute information assigned to the identified identifier as attribute information of the question-oriented vocabulary.
Interactive query response method in the server that includes.
In an interactive question-and-answer apparatus including a server including a computer processor communicating with a user terminal through a communication network,
The server,
A storage unit for storing multimedia information, meta information including identification information for a plurality of entities of interest to the user in the multimedia content, and attribute information assigned to the identification information; And
The multimedia content and the meta information are transmitted to the user terminal, and among the plurality of entities, a query statement about the entity selected by the user and identification information of the entity selected by the user extracted from the meta information are received from the user terminal. , A computer processor that generates a response to the query based on the attribute information assigned to the identification information of the entity selected by the user and transmits the response to the user terminal,
The computer processor,
In the multimedia content, a plurality of entities expected to be of interest to the user are classified using a pre-trained entity classification model, and the identification information for each of the classified multiple entities is generated based on a multi-modal interface, The interactive question-and-answer device that generates and stores the meta information including the generated identification information in the storage unit.
delete The computer processor of claim 13,
In order to compare the multi-modal input information and the identification information output from the multi-modal interface, the interactive question-and-answer device to generate the identification information to have the attributes of the multi-modal input information.
The computer processor of claim 13,
Each unique identifier of the plurality of entities, a screen coordinate in which the plurality of entities are located on a display screen, a time period in which entities included in video content are played, a time period in which entities included in audio content are played, and the plurality of entities And generating the meta information to include the identification information including an attribute name indicating each attribute, and storing the meta information in the storage unit.
The computer processor of claim 13,
Analyze the query statement transmitted from the user terminal, recognize the question-oriented vocabulary in the query statement, determine the attribute information of the question-centered vocabulary based on the identification information transmitted from the user terminal, and determine the determined attribute information. An interactive question-and-answer device that selects a candidate as a correct answer and generates the response sentence including the selected correct answer candidate.
The computer processor of claim 17,
The storage unit in which the attribute information is stored is searched to search for an identifier identical to the identifier included in the identification information transmitted from the user terminal, and when the same identifier is confirmed in the storage unit, the attribute information assigned to the identified identifier is recalled. An interactive question-and-answer device that is determined by attribute information of a question-oriented vocabulary.

Images