KR102498403B1 - Apparatus and method for gathering training set for natural language to sql system - Google Patents

Abstract

The present invention relates to collection and analysis of a training set for an NL2SQL system that converts natural language into SQL. , and by visualizing the result, it is possible to collect a high-quality NL2SQL system training set even if you are not a SQL expert, which has the effect of enabling developers to easily analyze the cause of errors while saving costs.

Description

Training set collection device and method for a system that converts natural language to SQL

The present invention relates to a database management system, and more particularly to a system for converting natural language to SQL.

For relational databases and natural language queries, natural language to SQL translation (NL2SQL) is finding the SQL statement that corresponds to the natural language query. Recently, many methods using deep learning technology have been developed for such NL2SQL.

The main task in developing a NL2SQL system is to design a good encoder and decoder, but two things are needed to improve the quality of an NL2SQL system. First, a large number of high-quality, labeled training data, and second, a tool for finding false translations. This is because the accuracy of the system is highly dependent on the training set.

Crowdsourcing is used to obtain a lot of training data, but compared to other machine learning problems, the NL2SQL system is an expensive problem because it requires workers to have a good knowledge of SQL queries and a given database schema. there is.

In addition, although representative NL2SQL models use an encoder decoder model, there is a problem in that it is difficult to properly understand what role each input feature plays in the output.

The inventors of the present invention have made research efforts to solve the problems of data collection and analysis of the prior art NL2SQL system. The present invention has been completed after much effort to complete a database natural language interface system that can save costs by enabling workers to easily search and collect data, visualize data, and analyze mistranslated queries.

It is an object of the present invention to provide an apparatus and method for obtaining a high-quality training set at low cost and analyzing mistranslated queries.

Meanwhile, other unspecified objects of the present invention will be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

Training set collection device for NL2SQL system according to the present invention,

A table creation unit for generating an extract table (Table Excerpts) composed of a part of the database in a given database; a search unit that translates a user's natural language question generated for the extraction table into a SQL query using a Natural Language to SQL (NL2SQL) system and displays a result of performing the SQL query; and an analysis unit that analyzes a pair of a natural language question and a SQL query, which is a training pair marked as incorrect by a user as a result of performing the SQL query, and proposes a model interpretation, error highlighting, or system. do.

The table generator may perform relation sampling, attribute sampling, and tuple sampling in the given database to generate the extract table.

a history database for storing a result of performing the SQL query in the search unit; and a training pair database for storing pairs of natural language questions and SQL queries marked as correct among results of performing the SQL query by the search unit.

The table creation unit creates a new extraction table by perturbing the extraction table, and the search unit verifies the SQL query using the new extraction table.

The analyzer may perform the model analysis by analyzing the inaccurate training pair by an integrated gradient method and a conductance method.

The analyzer normalizes the SQL query of the incorrect training pair and the given correct query using query rewriting, and compares the grammatical structures of the two normalized queries to visualize the incorrect part of the SQL query of the incorrect training pair. It is characterized in that the error highlighting is performed by doing.

The analyzer may translate the user's natural language question using a plurality of other NL2SQL systems other than the NL2SQL system, and recommend a NL2SQL system whose translated SQL query matches the given exact SQL query.

A training set collection method according to another embodiment of the present invention,

(a) in a given database, creating Table Excerpts which are part of said database; (b) translating the user's natural language question generated with respect to the extraction table into a SQL query using a Natural Language to SQL (NL2SQL) system, and displaying a result of performing the SQL query; and (c) analyzing a pair of a natural language question and a SQL query, which is a training pair marked as incorrect by a user as a result of performing the SQL query, and suggesting model interpretation, error highlighting, or a system; includes

The step (a) is characterized in that relation sampling, attribute sampling and tuple sampling are performed in the given database to generate the extract table.

After the step (b), storing the result of performing the SQL query in a history database; and storing a pair of a natural language question and an SQL query marked as correct among results of performing the SQL query in a training pair database.

In the step (a), a new extract table is created by perturbing the extract table, and in the step (b), the SQL query is verified by the new extract table.

The step (c) is characterized in that the model analysis is performed by analyzing the inaccurate training pair by an integrated gradient method and a conductance method.

The step (c) normalizes the SQL query of the incorrect training pair and the given correct query using query rewriting, and compares the grammar structures of the two normalized queries to determine whether the SQL query of the incorrect training pair is correct. It is characterized in that the error highlighting is performed by visualizing the part.

The step (c) is characterized by translating the user's natural language question using a plurality of other NL2SQL systems other than the NL2SQL system, and recommending an NL2SQL system whose translated SQL query matches the given exact SQL query.

According to the present invention, it is possible to collect a high-quality training set for training a system that converts natural language into SQL.

In addition, by providing visualization data for analyzing data for training, workers can more easily collect training data and correct incorrect data.

On the other hand, even if the effects are not explicitly mentioned here, it is added that the effects described in the following specification expected by the technical features of the present invention and their provisional effects are treated as described in the specification of the present invention.

1 is a schematic structural diagram of a training set collection device according to a preferred embodiment of the present invention.
2 and 3 show an implementation example of a training set collection device according to a preferred embodiment of the present invention.
4 is a schematic flowchart of a training set collection method according to another preferred embodiment of the present invention.
※ It is revealed that the accompanying drawings are exemplified as references for understanding the technical idea of the present invention, and thereby the scope of the present invention is not limited.

Hereinafter, with reference to the drawings, look at the configuration of the present invention guided by various embodiments of the present invention and the effects resulting from the configuration. In the description of the present invention, if it is determined that a related known function may unnecessarily obscure the subject matter of the present invention as an obvious matter to those skilled in the art, the detailed description thereof will be omitted.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above terms may only be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'. can Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those skilled in the art unless otherwise defined.

Hereinafter, with reference to the drawings, look at the configuration of the present invention guided by various embodiments of the present invention and the effects resulting from the configuration.

1 is a schematic structural diagram of a training set collection device according to a preferred embodiment of the present invention.

A training set collection device 1 for an NL2SQL system according to the present invention includes a table creation unit 10, a search unit 20, a collection unit 30, and an analysis unit 40. In addition, a training pair database 50 and a history database 60 may be further included.

The table creation unit 10 creates an extract table (Table Excerpts) through sampling in a given database D.

To create a small and intuitive extract table, perform 1) relation sampling, 2) attribute sampling, and 3) tuple sampling in database D.

For relationship/property sampling, accumulated SQL queries are used to calculate the relevance of different relationships and properties. The higher the frequency in which two schema entries (ie, relationships or attributes) appear together, the more related they can be determined.

To this end, a connected subgraph is extracted from a schema graph in which each node is a relationship or attribute and each edge implies an affiliation of a combinable relationship between two relationships or a relationship of attributes. In particular, it randomly selects relationships with probabilities proportional to the frequency of their appearance in the SQL query log. It then randomly and repeatedly selects new relationships that can be combined with previously selected relationships. Here, the selection probability is proportional to the co-occurrence frequency of the selected relationships.

After selecting the relationships, similarly, the properties are selected by considering the co-occurrence frequency of the selected relationships.

After the sampling phase is over, an extract table is created using this. It is created by 1) performing all outer joins between all selected relations, 2) projection on all selected attributes to obtain table t _D , and 3) sampling some tuples from t _D .

Table 1 below shows an example of an extract table generated from IMDB (International Movie Database).

[Table 1]

First, three attributes (Movie, Cast, and Actor) are selected by relationship sampling, where casting represents a many-to-many relationship between a movie and an actor. Next, six attributes are selected by attribute sampling. Table _tD can then be obtained by performing projections on all six attributes and all outer joins of the three relations. Finally, sampling five tuples from t _D gives Table 1, an excerpt table.

To obtain a more intuitive extract table, the extract table should contain tuples with the same partial key value. Considering the key {Movie.mid, Actor.aid} in Table 1, the first two tuples have the same partial key value of 2627, so they represent the same movie. This allows workers to come up with meaningful groups and query sets for the excerpt table.

The search unit 20 searches the database by translating the user's question into an SQL query. For this, users can choose the NL2SQL system and database.

2 is an example of a screen displayed to the user by the search unit 20.

The search unit 20 first randomly selects and displays an extract table 210 from the selected database.

When the search unit 20 receives a natural language question q _nl 220 for the extract table 210, it is translated into an SQL query q _sql using the selected NL2SQL system, and an execution result 230 is visualized.

For example, if the user enters a natural language question (220) “Find the titles of the movies which “Brad Pitt” starsred in”, it is “SELECT T1.title FROM movie AS”. T1 JOIN cast AS T3 ON T1.mid = T3.msid JOIN actor AS T2 ON T3.aid = T2.aid WHERE T2.name = “Brad Pitt” is converted into an SQL query, and the search unit 20 extracts the table ( In step 220, the titles of corresponding movies are searched and output (230). Looking at the result, the user checks (240) whether it is correct.

During this process, depending on the excerpt table, an incorrect query may produce the same response as a correct query. For example, in the excerpt table t of Table 1, when the user asks the question “How many performers are there?” You can create SQL queries. Even if the number of movies is generated as a response by this SQL query, the user will judge it as an accurate response and mark the translation as correct. This is because both the number of actors and the number of movies shown in Table 1 are 3.

To reduce this error, the table generator 10 verifies the SQL query by creating another extract table t'. The culling table t' can be created by perturbing culling table t and having it verified by other users.

The resulting SQL queries end up being queries against the original database, not against the extract tables. This is because you can collect complex queries including joins as well as simple queries against a single excerpt table.

Since the search unit 20 generates an extract table by performing SQL queries including 1) combination of selected tables, 2) projection of selected columns, and 3) sampling, the SQL query q1 for the extract table is converted into the original SQL query q1. This can be converted to SQL query q2 against the database.

The conversion is done by 1) substituting the join predicate in the FROM clause of q1, not the table, and 2) replacing the columns in q1 with the corresponding columns in the source database.

For example, in Table 1, the question “Who acted in movie 120?” is converted by the NL2SQL system into the SQL query “SELECT actor_name FROM table_excerpt WHERE movie_title='120'”, which is “SELECT actor.name FROM actor JOIN cast ON actor.aid = cast.aid JOIN movie ON cast.mid = movie.mid WHERE movie.title = '120'.”

That is, the previous query is displayed to the user, but the later converted query is collected by the search unit 20 . Of course, both the conversion result for the extract table and the conversion result for the original database can be analyzed.

The collection unit 30 collects training pairs. The collection of training pairs is invisible to the user as it is done as a background task while the user is doing other things.

Training pairs marked as correct by the user in the search unit 20 are stored in the training pair database 50 . If the user marks the training pair as incorrect, it is stored in the history database 60 and transmitted to the analyzer 40 as well.

In the collection unit 30, all bundles (q _nl , q _sql , feedback) consisting of natural language questions, SQL queries, and feedback are stored in the history database 60.

The analysis unit 40 receives a question to be analyzed by the user and receives an accurate SQL query for the selected question. It also performs model interpretation, error highlighting, and system suggestions.

3 is an example of a screen displayed to a user by the analyzer 40 .

The analysis unit 40 uses an integrated gradients method for model analysis. The integrated gradient method evaluates the contribution of each word (w) in a natural language or schema to the model output.

The analysis unit 40 calculates the contribution between the word w and each element of the output sequence generated by the decoder, and the contribution of w is obtained by adding all the contributions by w. Each word is visualized as a letter in a different color according to its contribution. For example, green can mean positive contribution and red can mean negative contribution.

The analyzer 40 may also use a conductance method to visualize the importance of internal hidden units of the model. The analyzer 40 evaluates the conductance of each hidden state. Since hidden states exist at the intersection of elements of sequences and layers (for example, if the sequence length is 5 and the number of layers is 3, 5x3 = 15 hidden states exist), the conductance is It is visualized as a heat map, where the x-axis is the elements of the sequence and the y-axis is the layer.

By these methods, the model's behavior can be understood, so that erroneous translations can be analyzed to improve the model and make troubleshooting easier.

In FIG. 3 , the user can easily find the location of the error through the red line 330 displayed on the scroll bar. Since all errors are stored in the history database 60, they can be loaded to find the error location.

For error verification, the user inputs a correct query 310 . The answer query 310 to “How many performers are there?” will be “SELECT count(*) FROM actor AS T1”.

When the user asks the question “the number of all actors”, “actors” included in the natural language question have a relationship with “actors”. If the contribution 320 of the word "actor" analyzed by the analyzer 40 is too low, it can be seen that the model does not fully understand the word. In the example of FIG. 3 , the contribution 320 of a word is expressed as a color density. Therefore, we will be able to modify the model by replacing the model's encoder structure with a pre-trained language model.

The analysis unit 40 then performs error highlighting.

The analysis unit 40 compares the selected SQL query to a given correct query and visualizes which part of the selected SQL query is incorrect. The analyzer 40 normalizes the two queries using query rewriting and compares the grammatical structures of the two normalized queries.

Finally, the analysis unit 40 translates the natural language question using another NL2SQL system not selected by the user, and then recommends the NL2SQL system that generated an accurate translation. If there is no system that provides accurate translation, the analysis unit 40 recommends the most accurate NL2SQL system by calculating a similarity score between the translated SQL query of each system and the ground truth. In order to calculate a similarity score between the two SQL queries, the analysis unit 40 normalizes the two SQL queries.

To determine correctness of a translation, analysis unit 40 uses a multi-staged validation tool, which determines semantic equivalence between a translated query and a given correct SQL query.

The analysis unit 40 first performs two queries on a given database instance and database instances obtained from the database test technique, and compares the results. If the execution results are different, the two queries are determined not to be equivalent. If the execution results are the same, the two are compared by a prover for semantic equality of the two SQL queries. If the proof tool fails to determine equivalence, the analyzer 40 may convert the two queries by query transformation and compare their grammatical structures. A relational database management system (RDBMS) or the like may be used for query conversion.

4 is a schematic flowchart of a training set collection method according to another preferred embodiment of the present invention.

The training set collection method of the present invention may be performed by a training set collection device including one or more processors and a control unit.

To collect a training set, an extract table is first created through sampling in a given database (S10).

To create an extract table, 1) Relation sampling, 2) Attribute sampling, and 3) Tuple sampling can be performed in the database. The detailed method for creating a table is as described above.

When the extraction table is created, the user performs a table search step (S20).

In the table search step, the database is searched by translating the user's natural language question into an SQL query. For this, users can choose the NL2SQL system and database.

To reduce translation errors in SQL queries, the table creation step (10) and the search (20) steps can be repeated. At this time, a new extract table can be created by perturbing the old table and the user can verify it again.

After the search is over, training pairs that are pairs of natural language questions and SQL queries are stored in the database (S30).

Training pairs checked as correct by the user are stored in a training pair database, and training pairs checked as incorrect are stored in a history database and causes of errors are analyzed.

In the analysis step, model analysis is performed (S40).

When the user selects a training pair that has been checked as inaccurate, an accurate SQL query for it is received, and for model interpretation, the output contribution of words in the natural language question and hidden states within the model are analyzed using the integrated gradient method and conductance method. .

By these methods, the model's behavior can be understood, so that erroneous translations can be analyzed to improve the model and make troubleshooting easier.

Error highlighting may also be performed in the analysis step (S50).

In the error highlighting step, the syntax structure of the two queries can be compared by comparing the selected SQL query to a given exact query and visualizing which part of the selected SQL query is incorrect.

Finally, a better system can be proposed based on the analysis result (S60).

After translating the natural language question using another NL2SQL system not selected by the user, we recommend the NL2SQL system that produced the correct translation. If there is no system that provides accurate translation, the most accurate NL2SQL system is recommended by calculating the similarity score between the translated SQL query of each system and the ground truth.

According to the training set collection apparatus and method for the NL2SQL system according to the present invention as described above, high-quality training data for NL2SQL can be collected by users who are not SQL experts, and developers can easily analyze erroneous SQL translations. there is

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the scope of protection of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention belongs.

Claims (14)

a table creation unit that performs sampling on a given database and generates an extract table (Table Excerpts) composed of a part of the database;
a search unit that translates a user's natural language question generated for the extraction table into a SQL query using a Natural Language to SQL (NL2SQL) system and displays a result of performing the SQL query; and
As a result of performing the SQL query, a pair of natural language questions and SQL queries, which are training pairs marked as incorrect by the user, is analyzed to perform model interpretation, error highlighting, and other NL2SQL systems other than the NL2SQL system Training set collection device for the NL2SQL system, including; analysis unit for performing the proposal of.
According to claim 1,
Characterized in that the table creation unit performs relation sampling, attribute sampling and tuple sampling on the given database to generate the extract table.
According to claim 1,
a history database for storing a result of performing the SQL query in the search unit; and a training pair database for storing pairs of natural language questions and SQL queries marked as correct among the results of performing the SQL query in the search unit.
According to claim 1,
The table generator creates a new extraction table by perturbing the extraction table;
Wherein the search unit verifies the SQL query by the new extract table.
According to claim 1,
Characterized in that the analysis unit performs the model analysis by analyzing the inaccurate training pair by an integrated gradient method and a conductance method.
According to claim 1,
The analyzer normalizes the SQL query of the incorrect training pair and the given correct query using query rewriting, and compares the grammatical structures of the two normalized queries to visualize the incorrect part of the SQL query of the incorrect training pair. A training set collection device for an NL2SQL system, characterized in that performing the error highlighting by doing.
According to claim 1,
The analyzer translates the user's natural language question using a plurality of other NL2SQL systems other than the NL2SQL system, and proposes another NL2SQL system in which the translated SQL query matches the given exact SQL query. Characterized in that , a training set collection device for the NL2SQL system.
A training set collection method performed by a training set collection device comprising one or more processors:
(a) performing sampling on a given database to create table excerpts that are part of the database;
(b) translating the user's natural language question generated with respect to the extraction table into a SQL query using a Natural Language to SQL (NL2SQL) system, and displaying a result of performing the SQL query; and
(c) Analyzing a pair of natural language questions and SQL queries, which are training pairs marked as incorrect by the user as a result of performing the SQL query, performing model interpretation, performing error highlighting, and excluding the NL2SQL system A method of gathering a training set for a NL2SQL system, including performing suggestions from other NL2SQL systems.
According to claim 8,
Wherein step (a) performs relation sampling, attribute sampling and tuple sampling on the given database to generate the extract table.
According to claim 8,
After the step (b), storing the result of performing the SQL query in a history database; and storing a pair of a natural language question and an SQL query marked as correct among results of performing the SQL query in a training pair database.
According to claim 8,
Step (a) creates a new extraction table by perturbing the extraction table;
Wherein step (b) verifies the SQL query by the new extract table.
According to claim 8,
Wherein step (c) performs the model analysis by analyzing the inaccurate training pairs by an integrated gradient method and a conductance method.
According to claim 8,
The step (c) normalizes the SQL query of the incorrect training pair and the given correct query using query rewriting, and compares the grammar structures of the two normalized queries to determine whether the SQL query of the incorrect training pair is correct. A training set collection method for an NL2SQL system, characterized in that performing the error highlighting by visualizing a part.
According to claim 8,
The step (c) translates the user's natural language question using a plurality of other NL2SQL systems other than the NL2SQL system, and performs a proposal of the other NL2SQL system in which the translated SQL query matches the given exact SQL query Characterized by a training set collection method for the NL2SQL system.

Images