KR102623561B1 - Method for generating metadata for automatically determining type of data and apparatus for determining type of data using a machine learning/deep learning model for the same - Google Patents

Abstract

A metadata generating device using a machine learning/deep learning model according to an embodiment of the present invention is a data type - the data type is numeric, character, categorical, and date. The method of automatically determining and generating metadata includes the following steps: (a) primarily determining whether the field value of the received data is a date type; (b) the determination result of step (a) above, If it is primarily determined that it is not a date type, applying the data type determination conditions included in the metadata creation rule to the field value of the received data to secondarily determine whether it is a categorical type or a date type. , (c) If, as a result of the determination in step (b), it is secondarily determined that it is not one of the categorical and date types, the field name of the received data is changed to one of the data types included in the metadata creation rule. (d) applying the field mapping table for determination to determine whether it is a numeric type, character type, categorical type, or date type and generating metadata; and (d) the generated metadata. It includes the step of updating the metadata creation rules by learning with a machine learning/deep learning model.

Description

Method for automatically determining the type of data and generating metadata and data type determination device using machine learning/deep learning model for this {METHOD FOR GENERATING METADATA FOR AUTOMATICALLY DETERMINING TYPE OF DATA AND APPARATUS FOR DETERMINING TYPE OF DATA USING A MACHINE LEARNING/DEEP LEARNING MODEL FOR THE SAME}

The present invention relates to a method for automatically determining the type of data and generating metadata, and a data type determination device using a machine learning/deep learning model for this purpose. More specifically, it relates to a method and device that can easily and conveniently determine the data type used to generate metadata for application to machine learning/deep learning models, which are artificial intelligence models.

In order to apply data to a wide range of artificial intelligence (AI) models, including machine learning/deep learning models, the data should be used rather than applying raw, unorganized data as is. It is necessary to first identify and identify metadata, which is structured data that can be explained.

Meanwhile, in the case of structured data, meta data such as data type of each field, statistical characteristics of values, distribution such as maximum/minimum/average/standard deviation, range corresponding to outliers, data version, and name of data and field, etc. The performance of the model can be significantly improved by applying it to the artificial intelligence model while the data is clearly generated. This can be seen as a kind of preprocessing process for the data, and in the past, specialized personnel with high labor costs were used to run Python ( The main method was preprocessing, which involved exploring data using a high-level programming language such as Python or manually determining the data type of a field.

Meanwhile, during the data preprocessing process for metadata creation, the determination of data type is more important than anything else. Since artificial intelligence models can basically only receive numeric data, non-numeric data such as text or category can be used as input. Data must be converted to numeric data, and even if it is text-type data, not only the statistical information used but also the pre-processing method itself varies depending on whether it is general text-type data, categorical data representing a specific category, or date-type data. Because.

However, the conventional data type determination method had a problem in that the type determination results for the same data were different depending on the capabilities of the experts and their respective perspectives on the data, which could even affect the performance of the artificial intelligence model. In addition, since it is handled directly by professional personnel, it is bound to take a lot of time and high labor costs. In the case of startups or small and medium-sized businesses that do not have sufficient financial resources, they are unable to perform data type determination or require internal personnel to process data over a long period of time. By performing the type determination task, there was even a problem that the performance of the released product was degraded or the timing of product launch was delayed.

The present invention reflects these problems and provides a method and device that can easily and conveniently perform the generation of metadata corresponding to the data preprocessing process, and more specifically, the determination of the data type used in the creation of metadata, without processing by specialized personnel. It's about.

Republic of Korea Patent Publication No. 10-2310598 (2021.10.01)

The technical problem that the present invention aims to solve is the creation of metadata corresponding to the data preprocessing process for structured data, and more specifically, the ability to easily and conveniently determine the data type used to create metadata without processing by specialized personnel. The aim is to provide a method for automatically determining the type of data and generating metadata, and a data type determination device using machine learning/deep learning models for this purpose.

The technical problem to be solved by the present invention is to continuously improve the performance of machine learning/deep learning models by continuously learning the results of generating metadata for structured data, and more specifically, the results of determining the data type used to generate metadata. The aim is to provide a method for automatically determining the type of data that can be used to generate metadata, and a data type determination device using machine learning/deep learning models for this purpose.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

A metadata generating device using a machine learning/deep learning model according to an embodiment of the present invention to achieve the above technical problem is a data type - the data type is numeric. , including character type, category type, and date type - The method of automatically determining and generating metadata includes (a) first determining whether the field value of the received data is a date type, (b) the above ( As a result of the judgment in step a), if it is initially determined that it is not a date type, the data type determination conditions included in the metadata creation rule are applied to the field value of the received data to determine whether it is a categorical or date type. (c) If it is determined secondarily that it is not one of the categorical and date types as a result of the determination in step (b), the field name of the received data is used to generate the metadata. A step of applying the rule to the Field Mapping Table for determining the data type included in the rule to finally determine whether it is a numeric type, character type, categorical type, or date type, and generating metadata; and (d) ) It includes the step of learning the generated metadata with a machine learning/deep learning model and updating the metadata creation rule.

According to one embodiment, the determination in step (a) may be based on a publicly available programming language.

According to one embodiment, the data type determination condition included in the metadata creation rule of step (b) is that the ratio of the number of unique field values to the total number of field values of the received data is a predetermined ratio. It includes a first condition as to whether the data is less than a certain percentage, and if it is less than a predetermined percentage according to the first condition, the type of the received data may be categorical.

According to one embodiment, the predetermined ratio may be any one between 0 and 0.5.

According to one embodiment, the data type determination condition included in the metadata creation rule of step (b) is that the field value of the received data is a year pattern, a month pattern, a day pattern, an hour pattern, a minute pattern, and a second pattern. It includes a second condition regarding whether it includes one or more patterns, and if it includes one or more patterns according to the second condition, the type of the received data may be a date type.

According to one embodiment, the field mapping table of step (c) includes a field name/full name ID table in which one or more (full name ID, reference count) is mapped to each of one or more field names, and the one or more full names. For each ID, a data type mapping table may be included in which the probability of being a numeric type, the probability of being a character type, the probability of being a categorical type, and the probability of being a date type are mapped.

According to one embodiment, the field mapping table of step (c) includes a full name, field description, and reference count for each of the one or more full name IDs, and one or more data type mapping tables included in the data type mapping table. A full name information table corresponding 1:1 to each full name ID may be further included.

According to one embodiment, after step (d), (e) determining whether the data type finally determined in step (c) is a numeric type or a date type, (f) the ( As a result of the determination in step e), if it is determined that it is either a numeric type or a date type, step (g) sorting the field values of the received data to determine whether it contains duplicate field values; and (g) step (f) above. As a result of the determination, if it is determined that it contains duplicate field values, calculating the interval value between two neighboring field values among the sorted field values, (h) among the calculated interval values between the two field values. (i) determining whether the ratio of the largest number of interval values is more than a predetermined ratio; and (i) determining that the received data is a time series dataset if it is determined that it is more than a predetermined ratio as a result of the determination in step (h). It can be included.

According to one embodiment, (f´) if, as a result of the determination in step (e), it is determined that the received data is not one of the numeric and date types, determining that the received data is a non-time series dataset, (g ´) If, as a result of the determination in step (f), it is determined that it does not contain duplicate field values, determining that the received data is a time series dataset; and (i´) As a result of the determination in step (h), a predetermined If it is determined that the ratio is not higher, a step of determining that the received data is a non-time series dataset may be further included.

A data type determination device using a machine learning/deep learning model according to an embodiment of the present invention to achieve the above technical problem includes one or more processors, a network interface, and a memory that loads a computer program executed by the processor. and storage for storing large-capacity network data and the computer program, wherein the computer program determines, by the one or more processors, (A) whether the field value of the received data is one of a numeric type and a character type; (B) If, as a result of the judgment of the operation (A), it is primarily determined that it is either a numeric type or a character type, the data included in the metadata creation rule in the field value of the received data An operation that secondarily determines whether the type is a categorical or date type by applying the type determination condition, (C) As a result of the judgment of the operation (B) above, a secondary judgment is made that it is not a categorical or date type. If it is determined, the field name of the received data is applied to the Field Mapping Table for determining the data type included in the metadata creation rule to determine one of numeric type, character type, categorical type, and date type. (D) an operation to finally determine whether the type is and generate metadata, and (D) an operation to update the metadata creation rule by learning the generated metadata with a machine learning/deep learning model.

According to the present invention as described above, a data type determination device using a machine learning/deep learning model goes through a three-step decision process to determine the type of the received data as one of numeric type, character type, category type, and date type. Since it can be determined automatically, the data type can be determined easily and conveniently without processing by specialized personnel, thereby preventing unnecessary consumption of time and cost, and has the effect of allowing the preprocessing process for the data to be performed easily and conveniently. .

In addition, the metadata creation results are continuously learned using machine learning/deep learning models to update the metadata creation rules, which can improve the accuracy of subsequent metadata creation and, more specifically, data type determination. there is.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram showing the overall configuration of a data type determination device using a machine learning/deep learning model according to a first embodiment of the present invention.
Figure 2 is a flowchart showing representative steps of a method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention.
Figure 3 is a diagram illustrating an exemplary field mapping table.
Figure 4 is another diagram illustrating a field mapping table as an example.
5 shows a method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention, where the field name of the received data is included in the full name information table in the field name/full name mapping table. However, if the full name indicated by the field name is included in the full name information table, and the field name of the received data is not included in the field name/full name mapping table and the full name information table, it is preceded by This is a flowchart showing the steps.
FIG. 6 is flowchart 1 showing a portion separated from the flowchart shown in FIG. 5.
FIG. 7 is flowchart 2 showing a portion separated from the flowchart shown in FIG. 5.
Figure 8 is a diagram illustrating how the reference count is updated in the field mapping table according to steps S300-1 to S300-5.
Figure 9 is a diagram illustrating how the reference count is updated in the field mapping table according to steps S300-2' to S300-6'.
Figure 10 is a diagram illustrating how a new field name is updated in the field name/full name mapping table according to steps S300-2' to S300-6'.
Figure 11 shows the data type determination device 100 using a machine learning/deep learning model receiving data, applying the metadata creation rule to generate metadata, and then updating the metadata creation rule by learning it. This is an exemplary drawing.
Figure 12 shows a method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention, by adding a method for discriminating between a time series dataset and a non-time series dataset performed after step S240. This is the flow chart shown.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms, and the present embodiments are merely intended to ensure that the disclosure of the present invention is complete and to provide common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Meanwhile, the terms used in this specification are for describing embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated in the phrase.

As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of the referenced component, step, or operation as well as one or more other components, steps, or operations. .

Figure 1 is a diagram showing the overall configuration included in the data type determination device 100 using a machine learning/deep learning model according to the first embodiment of the present invention.

However, this is only a preferred embodiment for achieving the purpose of the present invention, and some components may be added or deleted as needed, and of course, the role played by one component may be performed by another component.

The data type determination device 100 using a machine learning/deep learning model according to the first embodiment of the present invention includes a processor 10, a network interface 20, a memory 30, a storage 40, and data connecting them. It may include a bus 50, and may be implemented as an independent device itself, or may be implemented in the form of a tangible physical service server such as an in-house system or space rental system, or an intangible cloud service server. will be.

The processor 10 controls the overall operation of each component. The processor 10 may be a CPU (Central Processing Unit), MPU (Micro Processer Unit), MCU (Micro Controller Unit), or any of the types of processors widely known in the technical field to which the present invention pertains, and may be a machine learning model processor. Alternatively, it can be implemented with an artificial intelligence model processor, such as a deep learning model processor. In addition, the processor 10 may perform an operation on at least one application or program to perform a method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention.

The network interface 20 supports wired and wireless Internet communication of the data type determination device 100 using a machine learning/deep learning model according to the first embodiment of the present invention, and may also support other known communication methods. Accordingly, the network interface 20 may be configured to include a corresponding communication module.

The memory 30 stores various information, commands, and/or information, and receives one from the storage 40 to perform the method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention. The above computer programs 41 can be loaded. In FIG. 1, RAM is shown as one of the memories 30, but of course, various storage media can also be used as the memory 30.

Storage 40 may non-temporarily store one or more computer programs 41 and large-capacity network information 42. This storage 40 may be non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, hard disk, removable disk, or in the technical field to which the present invention pertains. It may be any of the widely known computer-readable recording media.

The computer program 41 is loaded into the memory 30, and one or more processors 10 perform (A) an operation to primarily determine whether the field value of the received data is a date type, and (B) the (A) operation. As a result of the judgment, if it is primarily determined that it is not a date type, it is secondarily determined whether it is a categorical or date type by applying the data type determination conditions included in the metadata creation rule to the field value of the received data. (C) If, as a result of the judgment of the operation (B), it is secondarily determined that it is not one of the categorical and date types, the field name of the received data is included in the metadata creation rule. An operation that applies to the Field Mapping Table for determining the data type to finally determine whether it is a numeric type, character type, categorical type, or date type and generates metadata, and (D) the above generation. An operation that updates the metadata creation rule can be performed by learning metadata with a machine learning/deep learning model.

The operation performed by the computer program 41 briefly mentioned above can be viewed as a function of the computer program 41, and a more detailed description will be provided by automatically determining the type of data according to the second embodiment of the present invention and This will be explained later in the description of how to generate data.

The data bus 50 serves as a path for moving instructions and/or information between the processor 10, network interface 20, memory 30, and storage 40 described above.

If the data type determination device 100 using the machine learning/deep learning model according to the first embodiment of the present invention described above is implemented in the form of an independent device, a user terminal (not shown) sends data to the device through the network. It will be possible to transmit or receive generated metadata, and if implemented in the form of a server, the automatic metadata generation function provided by a dedicated application linked to the server and network can be provided to the user terminal (not shown). .

Hereinafter, a method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention will be described with reference to FIGS. 2 to 10.

Figure 2 is a flowchart showing representative steps of a method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention.

However, this is only a preferred embodiment in achieving the purpose of the present invention, and of course, some steps may be added or deleted as needed, and one step may be performed by being included in another step.

In addition, each step is assumed to be performed by the data type determination device 100 using the machine learning/deep learning model according to the first embodiment of the present invention, and the implementation form is performed in the form of a server, and a user terminal (not shown) ), any terminal with a network function, such as a smartphone, smart watch, smart glasses, laptop computer, tablet computer, PDA, PDP, PMP, etc. may be used, but in the following description, the user terminal (not shown) We will continue the explanation assuming that this is a desktop PC.

First, the data type determination device 100 using a machine learning/deep learning model first determines whether the field value of the received data is a date type (S210).

Here, the entity that transmits data so that the data type determination device 100 using a machine learning/deep learning model can receive the data may typically be a user terminal (not shown), and the machine is not a user terminal (not shown). The data type determination device 100 using a learning/deep learning model may crawl from the Internet or access a specific database and download it by itself or by operation of an administrator, and the subject transmitting the data is not particularly limited. .

The data type determination device 100 using a machine learning/deep learning model that has received the data can store it in an internal storage space such as the storage 30 or an external storage space connected through a network, and the field value of the data is immediately stored without storage. You can first determine whether this is a date type.

Meanwhile, step S210 is performed by the data type determination device 100 using a machine learning/deep learning model, or more specifically, Python installed in the data type determination device 100 using a machine learning/deep learning model. ), etc., and accordingly, the primary determination of the data type in step S210 may be about a data type that can be recognized by the corresponding programming language, such as Python.

In this case, according to Python, the recognizable data types can be largely numeric, character, and date types. Here, in the case of numeric types (= numeric types), types consisting of numbers including integers and real numbers, character types (= In the case of text type, string type), it may be a type composed of text, and the date type may be a type composed of a format representing a date that a programming language can recognize.

The data type determination device 100 using a machine learning/deep learning model is based on Python, and when all field values of the data are made up of numbers, it is primarily judged as a numeric type and a format representing the date, for example, YYYY-MM. If it consists only of formats such as -DD, YYYY/MM/DD, etc., it can be judged primarily as a date type, and the rest can be judged as a character type.

As a result of the determination in step S210, if it is initially determined that the field value of the received data is not a date type, the data type determination device 100 using a machine learning/deep learning model creates metadata in the field value of the received data. By applying the data type determination conditions included in the rule, it is secondarily determined whether it is a categorical type or a date type (S220).

The secondary judgment in step S220 is performed only when it is primarily determined that the field value of the received data is not a date type as a result of the judgment in step S210, and the field value of the received data is a date type as a result of the judgment in step S210. If it is initially determined that the data type is a date type, step S220 is not performed and the data type is finally determined to be a date type, and the process ends.

In this way, the reason why step S220 is performed only when it is primarily determined that the field value of the received data is not a date type as a result of the determination in step S210 is that in the case of numeric types and character types that are not date types, only numbers or letters are used. This is because even if it is configured, it may not be a numeric type or character type in the true sense, but a categorical type or date type.

For example, if the field values for data called customer rating are 1, 2, 3, 4, and 5, it will be judged as a numeric type according to step S210 because it consists only of numbers, but in the true sense, it is a category representing the customer rating. This is because it is a field value corresponding to, and similarly, if the field value for quality data is very good, good, normal, bad, or very bad, it is not a numeric or date type, so it will be judged as a character type according to step S210, but it is not a numeric or date type. This is because, in meaning, it is a field value that corresponds to a category indicating quality. Additionally, if the field value for some data is 20211111, it is not a date format that the programming language can recognize, so it will be judged as a numeric type rather than a date type according to step S210, but in true meaning, it is a date format such as November 11, 2021. This is because it is a field value corresponding to a date, and similarly, if the field value for some data is November 11, 2021, unless the date format is specified separately, it will be judged as a character type rather than a date type according to step S210, but in the true sense, it will be judged as a character type rather than a date type. This is because it is a field value representing a date, so cases where an accurate detailed judgment of the data type is required may be limited to numeric and character types.

Data type determination using a machine learning/deep learning model To secondarily determine whether the field value of the data received by the device 100 is a categorical or date type, determine the data type included in the metadata creation rule. Conditions are used, which will be explained below.

One of the data type determination conditions included in the metadata creation rule is the first condition for whether the ratio of the number of unique field values to the total number of field values of the received data is less than a certain ratio, and the first condition is If it is less than a predetermined ratio depending on the condition, the type of the received data may be a categorical type, and this corresponds to the condition for determining whether it is a categorical type among numeric and character types.

Let me explain with an example. In a dataset consisting of 1,000 samples, if there are 10 unique field values among the field values of a field called "Customer Rating" that was initially determined to be a numeric type by the programming language, the unique field value is compared to 1,000, the total number of field values. The ratio of 10, which is the number of , is 0.001, which is 10/1,000, which is a very small value close to 0, a value that can be considered almost non-existent from a statistical perspective, and 1,000 datasets contain unique field values. Multiple sets with identical field values - The field values for the multiple sets themselves are different from each other, for example, 100 sets with a field value of 1, 200 sets with a field value of 2, and 500 sets with a field value of 3. , 100 with a field value of 4, 99 with a field value of 5, and 1 with a field value of 6 (unique field value) - so it is a categorical type rather than a numeric type. In the above example, It can be identified as a categorical type with 1, 2, 3, 4, and 5 as categories.

Meanwhile, the predetermined ratio that serves as the standard for determining whether or not it is less than 0.5 may be any one between 0 and 0.5. As the predetermined ratio increases, even if the number of unique field values increases, it is determined as a categorical type and the accuracy of determination decreases. It may be desirable for the manager of the data type determination device 100 using a machine learning/deep learning model to set the predetermined ratio close to 0, and the metadata generated in step S240, which will be described later, is machine learning /When updating the metadata creation rules by learning with a deep learning model, the predetermined ratio may also be adjusted based on the learning results.

Another condition for determining the data type included in the metadata creation rule is whether the field value of the received data includes one or more of the following patterns: a year pattern, a two-letter month pattern, a day pattern, an hour pattern, a minute pattern, and a second pattern. This is the second condition for whether or not, and if it includes one or more patterns according to the second condition, the type of the received data may be a date type, and the condition for determining whether it corresponds to a date type among numeric and character types is It applies.

In the previous example, if the field value is 20211111 or November 11, 2021, it will be judged as a numeric type and a character type, respectively, but in true meaning, they all correspond to a date type, so in these cases, the exact data type can be confirmed by judging it as a date type. This is for discrimination purposes.

More specifically, the year pattern that the second condition can include is a pattern that includes two or four numbers and letters representing the year, such as year, Y, year, etc., and the month pattern is one letter or two letters. Includes a number and a letter representing the month, such as month, M, Month, etc., or a character that represents the month by itself without including a number, such as an abbreviation for the month such as Jan, Feb, etc. It is a pattern that includes letters that describe the month in its entirety, such as January, February, etc., and a day pattern is a pattern that includes one or two letters of numbers and letters representing the day, such as day, D, Day, etc. , a minute pattern is a pattern that contains one or two letters of a number and a character representing the minute, such as minute, m, min, minute, etc., and a second pattern is a pattern that contains one or two letters or a number after the decimal point and a second. Since it is a pattern that includes the characters it represents, for example, seconds, s, second, etc., the above pattern information is only an example, and as many different patterns can be added as conditions to determine whether it corresponds to a date type. Of course.

As a result of secondarily determining whether it is a categorical or date type according to step S220, it was secondarily determined that it is not either a categorical or date type, that is, either a numeric type or a character type. In this case, the data type determination device 100 using a machine learning/deep learning model applies the field name of the received data to the field mapping table for determination of the data type included in the metadata creation rule to generate a number. It is finally determined whether the type is type, character type, category type, or date type, and metadata is generated (S230).

Unlike using the field values of the data in the preceding steps S210 and S220, there is a difference in using the field names of the data in the step S230, and the metadata creation rule is based on not only the preceding data type determination conditions but also a field mapping table for data type determination. (Field Mapping Table) may be further included, and in step S230, the type of data can be finally determined using the field mapping table.

Figure 3 is a diagram illustrating a field mapping table by way of example. Referring to Figure 3, the field mapping table has one (full name ID, reference count) for each of one or more field names as shown on the left side of Figure 3. As shown in the field name/full name mapping table mapped above and on the right, for each of one or more full name IDs, there is a probability of being a numeric type, a probability of being a character type, a probability of being a categorical type, and a probability of being a date type. It may include a data type mapping table to which probabilities are mapped.

Referring to the field name/full name ID table, the field names CUSNUM, PRCS_DT, PRCSDT, and CSTMRGRAD are (2,1), (1,3)/(4,2)/(3,1), (1,3), respectively. )/(3,1), (5,2)/(2,1) can be confirmed to be mapped, so multiple full name IDs can be mapped to one field name, and the field name/full name Since the ID table is sorted in order of the full name ID with the highest reference count for each field name, the data type of the field name can refer to the full name ID with the highest reference count.

Let me explain with an example. If the field name of the received data is CSTMRGRAD, you can search for the field name CSTMRGRAD in the field name/full name ID table and check the full name ID 5, which is the full name ID with the highest reference count of 2, and the full name ID 5. After searching 5 in the data type mapping table, the categorical type, which is the type with the highest probability of 0.75, can be determined as the final type of the data. If the field name is PRCS_DT, the corresponding field in the field name/full name ID table You can search the name PRCS_DT and check full name ID 1, which is the full name ID with the highest reference count of 3dmf. After searching the full name ID 1 in the data type mapping table, you can find the date type with the highest probability of 0.90. The type type can be judged as the final type of data.

The above explanation assumes that the field name of the received data is included in the field mapping table, more specifically, the field name/full name mapping table. The field name of the received data is included in the field name/full name mapping table. Cases where it is not present may occur as many times as possible, and for such cases, the field mapping table may further include a full name information table.

Figure 4 is another diagram illustrating a field mapping table. Referring to Figure 4, in the upper right corner compared to Figure 3, there is a full name, field description, and reference count for each of one or more full name IDs. It can be confirmed that it further includes a name information table, and the full name ID included in the full name information table can correspond 1:1 with each of the full name IDs included in the data type mapping table at the bottom right.

Here, the full name information table is used when the field name of the received data is not included in the field name/full name mapping table, but the full name information table includes the full name that the field name represents, and the field name of the received data is the field name. /Full name mapping table. If the full name is also not included in the full name information table, the problem is how to determine the data type. In this case, prior to determining the data type, the full name of the data field name must be set and recommended, and the field name must be added to the field name/full name mapping table, which will be explained below.

5 shows a method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention, where the field name of the received data is included in the full name information table in the field name/full name mapping table. However, if the full name indicated by the field name is included in the full name information table, and the field name of the received data is not included in the field name/full name mapping table and the full name information table, it is preceded by It is a flowchart showing the steps, and FIGS. 6 and 7 are diagrams showing the drawing shown in FIG. 5 separated due to size issues of the accompanying drawings.

First, the data type determination device 100 using a machine learning/deep learning model sets the full name of the data field name and sets the field name of the received data to the field name/full name mapping table included in the field mapping table for the recommendation rule. Determine whether it is included (S300-1).

Here, the field name corresponds to an abbreviation, and the explanation will be continued using the case where the field name of the received data is PRCS_DT as an example.

Referring again to FIG. 4, it can be seen that the left table, the field name/full name mapping table, contains various field names, and the data type determination device 100 using a machine learning/deep learning model determines the type of data received. Since the field name is PRCS_DT, it can be determined whether the field name PRCS_DT is included in the field name/full name mapping table, and it can be determined that it is included because it is included in column 1, row 3.

If it is determined to be included as a result of the determination in step S300-1, the data type determination device 100 using a machine learning/deep learning model searches for the full name ID with the highest number of references to the field name (S300-2 ).

Referring to Figure 4 again, it can be seen that one or more (Full Name ID, reference count) matches each field name in the field name/full name mapping table, which is the left table, where the full name ID is the field name/full name mapping table, which is the table on the top right. It corresponds to an ID indicating a specific full name included in the full name information table, and the reference count indicates how many times the field name was used as a specific full name within the data type determination device 100 using a machine learning/deep learning model. It corresponds to a number. The higher the number of references to a specific full name ID, the more frequently the field name is used as the full name corresponding to that specific full name ID. Search for the full name ID with the highest number of references. The purpose of this is to recommend the full name that is most likely to correspond to the field name based on experience so far.

Applying this to the previous example, since the field name of the received data was PRCS_DT, according to step S300-2, full name ID 1 with a reference count of 3 can be selected by searching for the highest reference count.

Afterwards, the full name ID with the highest reference count searched by the data type determination device 100 using a machine learning/deep learning model is searched in the full name information table included in the field mapping table, and the full name mapped thereto is received. Set the full name of the metadata field name for the data (S300-3).

Since the full name ID with the highest reference count was previously searched in step S300-2, if the full name ID is searched in the full name information table, the matching full name can be set as the full name of the metadata field name for the data.

Applying this to the previous example, since full name ID 1, which is the full name ID with the highest number of references to the data field name PRCS_DT, was searched, full name ID 1 is searched in the full name information table shown in the upper right corner of Figure 4. And, the full name mapped to this, Process Date, can be set as the full name of the metadata field name PRCS_DT for the received data.

Meanwhile, the setting here also includes the meaning of recommendation. In a state where it is not clear exactly which full name the field name abbreviation corresponds to, the machine learning/deep learning model uses the full name that is most likely to correspond to it based on the number of references. This is because the data type determination device 100 using It will be said.

If the full name of the metadata field name for the received data is set, the data type determination device 100 using a machine learning/deep learning model refers to the field name of the data included in the field name/full name mapping table above. The reference count for the full name ID with the highest count is increased by 1 (S300-4), and the reference count for the full name ID with the highest reference count previously searched included in the full name information table is increased by 1 (S300- 5).

Since this step S300-5 sets the full name of the field name using the field mapping table, it can be seen as updating the field mapping table, more specifically the field name/full name mapping table and the full name information table, to the latest state. Applying this to the previous example, as shown in FIG. 8, among PRCS_DT, which is the field name of the corresponding data included in the field name/full name mapping table on the left, the reference count for full name ID 1 with the highest previously searched reference count is 3. This is updated to 4 by increasing 1, and the reference count for full name ID 1 included in the full name information table on the right is updated to 12 by increasing 1 from 11.

Steps S300-1 to S300-5 described above relate to the case where the field name of the received data is included in the field name/full name mapping table, and the case where it is not included will be described below.

Referring again to FIG. 4, if NO is followed in step S300-2, that is, if it is determined not to be included as a result of the determination in step S300-1, the data type determination device 100 using a machine learning/deep learning model receives Receive the full name of the metadata field name for the data from the user (S300-2′)

This is because the preset field name/full name mapping table does not include the abbreviation corresponding to the field name, so by receiving one or more characters that are clues to the full name directly from the user, the corresponding full name is recommended from the full name information table. This is to do so, and the explanation will be continued by taking as an example that the letters received from the user are two letters Cu.

Afterwards, the data type determination device 100 using a machine learning/deep learning model predicts the letters or words that follow the full name of a metadata field name of one or more characters received from the user and recommends the completed full name ( S300-3′)

Since the character received from the user in step S300-2′ is Cu, the data type determination device 100 using a machine learning/deep learning model selects a full name starting with Cu among the full names included in the full name information table. Customer Number and Customer Grade can be recommended.

If a completed full name is recommended, the data type determination device 100 using a machine learning/deep learning model determines whether the recommended completed full name is selected by the user (S300-4′), and as a result of the determination, it is selected by the user. If so, the reference count for the selected full name included in the full name information table is increased by 1 (S300-5′), and matched to the selected full name among one or more field names included in the field name/full name mapping table. Increase the reference count for the specified field name by 1 (S300-6′)

This means that the user has selected the completed full name recommended in step S300-4′, meaning that the full name is included in the full name information table, and the field name matched to the full name is also included in the field name/full name mapping table. Since it can be seen that it is included in , the field mapping table is updated to the latest state by increasing the reference count by 1.

Applying this to the previous example, if the user selected Customer Grade among the recommended full names Customer Number and Customer Grade, the reference count of the full name Customer Grade included in the full name information table on the right is changed from 2 as shown in Figure 9. It is updated to 3 by increasing 1, and in the field name/full name mapping table on the left, the number of references in the field name CSTMRGRAD for full name ID 5, which is the full name ID of the full name Customer Grade, is increased from 2 to 3. is to update.

In steps S300-2′ to S300-6′ described above, the field name that exactly matches the field name of the data is not included in the field name/full name mapping table, but the full name for the field name is included in the full name information table. This refers to a case where a field name with substantially the same meaning is included in the field name/full name mapping table. For example, if the field name of the data is CUGRAD, it is not included in the field name/full name mapping table, so S300- Step 2 will be implemented, but if the letters received from the user for the full name of the field name CUGRAD are CU, and Customer Grade is selected among the recommended full names, CSTMRGRAD included in the field name/full name mapping table is actually It is considered as referenced.

Separately, the data type determination device 100 using a machine learning/deep learning model adds a field name not included in the field name/full name mapping table, for example, CUGRAD, as shown in FIG. 10, and You can update the new field name by entering the matching full name ID, 5, but specifying the reference count as 1. This is based on the premise that one full name can be mapped to multiple field names. , Conversely, it goes without saying that one field name may match multiple full names, and whether to increase the reference count of an existing field name that has substantially the same meaning as the newly added field name, for example, CSTMRGRAD, by 1. This may depend on the administrator's settings, and FIG. 10 shows the non-increased state as an example.

This time, we will explain the case where the completed full name recommended in step S300-4′ is not selected by the user.

As a result of the determination in step S300-4′, if the recommended completed full name has not been selected by the user, the data type determination device 100 using a machine learning/deep learning model determines the full name of the metadata field name for the received data. All are received from the user (S300-5″), and it is determined whether the full name information table includes the full name of the metadata field name all received from the user (S300-6″).

If the recommended completed full name was not selected by the user, it is likely that the full name information table does not include the full name for the field name intended by the user. However, depending on the user, even if it is included, the recommended completed full name is not selected. The full name may not be selected intentionally, and to ensure greater accuracy, the full name received from the user is searched again in the full name information table, and if it is determined to be included, a machine learning/deep learning model is used. The data type determination device 100 increases the number of references to the full name of the metadata field name included in the full name information table by 1 (S300-7″), and this is explained in S300-7″ above. Since it is the same as the description for step 5′, detailed description will be omitted to prevent duplicate description, and then step S300-8″, which will be described later, is performed.

On the other hand, if it is determined that it is not included as a result of the judgment in step S300-6″, the data type determination device 100 using a machine learning/deep learning model enters the full name of the metadata field name received from the user in the full name information table and the corresponding Add the full name ID and set the reference count to 1 (S300-7′″).

Afterwards, the data type determination device 100 using a machine learning/deep learning model determines whether the field name/full name mapping table includes the full names of the metadata field names received from the user and the field names of the data mapped ( S300-8″), if it is judged not to be included as a result of the judgment, add the full name or abbreviation for the metadata field name received from the user to the field name/full name mapping table, and add the full name of the metadata field name received from the user in the field name/full name mapping table, and add Add the full name ID for the full name of all received metadata field names, but set the reference count to 1 (S300-9″). If it is determined to be included as a result of the judgment, the field name is added at step S300-7″. Increase the reference count of the full name ID whose reference count was increased by 1 (S300-10″).

The above S300-9″ step is to update the field name/full name mapping table to the latest state if the field name is included in the full name information table but is not included in the field name/full name mapping table. S300- Step 7′″ is to update both the full name information table and the field name/full name mapping table to the latest state if they are not included in both the full name information table and the field name/full name mapping table. Since the update is the same as the previous explanation, detailed explanation will be omitted to prevent redundant description.

After the above-described steps S300-1 to S300-5, S300-2′ to S300-6′, and S300-5″ to S300-10″ are all processed, data using machine learning/deep learning models is collected. The type determination device 100 reorganizes the field mapping table based on the field name, the number of references to the field name, the full name, and the number of references to the full name. In the case of the field name/full name mapping table, the number of references to the field name is We will reorder the mapping list according to the updated reference count.

As described above with reference to FIGS. 4 to 10, all processes of steps S300-1 to S300-5, steps S300-2′ to S300-6′, and steps S300-5″ to S300-10″ are completed. If the field name of the received data is not included in the field name/full name mapping table, but the full name information table contains the full name indicated by the field name, and the field name of the received data is the field name. /If the full name mapping table and the full name are also not included in the full name information table, the field name and full name for all received data can be included in the field name/full name mapping table and the full name information table, respectively. By searching the ID and applying the corresponding full name ID to the data type mapping table, the data type with the highest probability value can be searched and determined as the final type of data.

Let us return to the description of FIG. 2 again.

If the type of received data is finally determined and metadata is created, the metadata creation rules are updated by learning the generated metadata with a machine learning/deep learning model (S240).

Here, as mentioned above, the metadata creation rule includes data type determination conditions and a field mapping table including a field name/full name ID mapping table, data type mapping table, and full name information table. Step S240 is a meta data creation rule. This can be said to be a step of updating all of these to the latest learning state according to the data generation results. In Figure 11, the data type determination device 100 using a machine learning/deep learning model receives data and sets metadata creation rules. An example of how to create metadata by applying it and then learning it to update the metadata creation rules is shown as an example.

So far, a method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention has been described. According to the present invention, the data type determination device 100 using a machine learning/deep learning model goes through a three-step decision process to determine the type of the received data as any one of numeric type, character type, categorical type, and date type. Since it is possible to determine the data type easily and conveniently without processing by specialized personnel, unnecessary time and cost consumption can be prevented, and the preprocessing process for the data can be performed easily and conveniently. In addition, the metadata creation results are continuously learned with a machine learning/deep learning model to update the metadata creation rules, thereby improving the accuracy of subsequent metadata creation and, more specifically, data type determination.

Meanwhile, although not separately explained, in the method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention described above, an artificial intelligence model is required for the metadata creation rule to include a data type determination condition. It would be necessary to adopt a machine learning/deep learning-based NLP model that learned natural language sentences in advance.

Furthermore, in the method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention, as shown in FIG. 12, after step S240, the type of data finally determined in step S230 is a number. A step (S250) of determining whether the received field is one of a type (a field indicating an order, such as an integer type) or a date type. As a result of the determination in step S250, if it is determined that the received type is one of a number type and a date type, A step of sorting the field values of the data to determine whether they contain duplicate field values (S260). If it is determined that the data contains duplicate field values as a result of step S260, the difference between two neighboring field values among the sorted field values is A step of calculating the interval value (S270), a step of determining whether the ratio of the largest number of interval values among the calculated interval values between the two field values is more than a predetermined ratio (S280), and the determination result of step S280 is more than a predetermined ratio. If it is determined that the received data is a time series dataset, it may further include a step (S290), and if it is determined that the received data is not one of the numeric and date types as a result of the determination in step S250, the received data If, as a result of the judgment in step S260, it is determined that it is a non-time series data set, it is determined that it does not contain duplicate field values, the received data is determined to be a time series dataset, and as a result of the judgment in step S280, the data is determined to be more than a certain ratio. If it is determined that it is not, it is possible to determine whether the data set is a time series data set or a non-time series data set by further including a step of determining that the received data is a non-time series data set.

The method of automatically determining the type of data and generating metadata according to the second embodiment of the present invention described above can be implemented with a computer program stored in the medium according to the third embodiment of the present invention including all the same technical features. As such, it will not be described in detail to prevent redundant description, but all technical features applied to the method for automatically determining the type of data and generating metadata according to the second embodiment of the present invention described above are included in the second embodiment of the present invention. Of course, the same can be applied to the computer program stored in the medium according to the third embodiment.

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: Data type determination device using machine learning/deep learning model
10: processor
20: network interface
30: Memory
40: storage
41: computer program
50: data bus

Claims (5)

A data type determination device using a machine learning/deep learning model automatically determines the type of data - the data types include numeric, character, categorical, and date types - and In the method of generating data,
(a) First determining whether the field value of the received data is a date type;
(b) If, as a result of the determination in step (a), it is initially determined that it is not a date type, the data type determination conditions included in the metadata creation rule are applied to the field value of the received data to determine categorical and date type. A step of secondarily determining which type is one of the following;
(c) If, as a result of the determination in step (b), it is secondarily determined that it is not a category type or a date type, the field name of the received data is determined as a data type included in the metadata creation rule. Applying to the Field Mapping Table to finally determine whether it is a numeric type, character type, categorical type, or date type, and generating metadata; and
(d) updating the metadata generation rule by learning the generated metadata with a machine learning/deep learning model;
In a method of automatically determining the type of data containing and generating metadata,
After step (d) above,
(e) determining whether the data type finally determined in step (c) is a numeric type or a date type;
(f) if, as a result of the determination in step (e), it is determined that the data is of a numeric type or a date type, sorting field values of the received data to determine whether they contain duplicate field values; and
(g) if, as a result of the determination in step (f), it is determined that duplicate field values are included, calculating an interval value between two neighboring field values among the sorted field values;
(h) determining whether the ratio of the largest number of interval values among the calculated interval values between the two field values is greater than or equal to a predetermined ratio; and
(i) If it is determined as a result of step (h) that the ratio is higher than a predetermined ratio, determining that the received data is a time series dataset;
A method of automatically determining the type of data that further includes and generating metadata. According to paragraph 1,
The judgment in step (a) is based on a published programming language,
A method of automatically determining the type of data and creating metadata. According to paragraph 1,
The data type determination conditions included in the metadata creation rule of step (b) are:
A second condition is provided regarding whether the field value of the received data includes one or more of a year pattern, a month pattern, a day pattern, an hour pattern, a minute pattern, and a second pattern,
If it includes one or more patterns according to the second condition, the type of the received data is a date type,
A method of automatically determining the type of data and creating metadata. According to paragraph 1,
(f´) if, as a result of the determination in step (e), it is determined that the received data is not one of the numeric and date types, determining that the received data is a non-time series data set;
(g´) if it is determined as a result of step (f) that it does not contain duplicate field values, determining that the received data is a time series dataset; and
(i´) if, as a result of the determination in step (h), it is determined that the ratio is not more than a predetermined ratio, determining that the received data is a non-time series dataset;
A method of automatically determining the type of data that further includes and generating metadata. One or more processors;
network interface;
a memory that loads a computer program executed by the processor; and
Including storage for storing large-capacity network data and the computer program,
The computer program is operated by the one or more processors,
(A) An operation that primarily determines whether the field value of the received data is a date type;
(B) If, as a result of the judgment of the operation (A) above, it is initially determined that it is not a date type, the data type determination conditions included in the metadata creation rule are applied to the field value of the received data to determine categorical and date type. An operation to secondarily determine which type is one of the following;
(C) If, as a result of the judgment of the operation (B), it is secondarily determined that it is not a category type or a date type, the field name of the received data is determined as a data type included in the metadata creation rule. An operation that applies to the Field Mapping Table to finally determine whether it is a numeric type, character type, categorical type, or date type and generate metadata; and
(D) An operation to update the metadata creation rule by learning the generated metadata with a machine learning/deep learning model;
In a data type determination device using a machine learning/deep learning model that executes,
After operation (D) above,
(E) an operation to determine whether the data type finally determined in operation (C) above is a numeric type or a date type;
(F) If, as a result of the determination of the operation (E), it is determined that the data is of either numeric type or date type, an operation for sorting the field values of the received data to determine whether they contain duplicate field values; and
(G) If, as a result of the determination of the operation (F), it is determined that duplicate field values are included, an operation for calculating an interval value between two neighboring field values among the sorted field values;
(H) an operation to determine whether the ratio of the largest number of interval values among the calculated interval values between the two field values is more than a predetermined ratio; and
(I) an operation that determines that the received data is a time series dataset if it is determined that the (H) operation is higher than a predetermined ratio;
Data type determination device using machine learning/deep learning model that executes