KR20130011207A - Method and system for automatic classification of data - Google Patents

Abstract

PURPOSE: An automatic classification method of data and a system thereof are provided to automatically extract metadata from data and to output data by grouping the data according to a specific purpose. CONSTITUTION: A data management unit extracts a keyword from data in main database(S404). The data management unit generates and stores the meta data according to the attribute of the keyword. A data processing unit divides the input data into word. The data processing unit arranges the extracted keyword according to comparison result of the metadata(S407). [Reference numerals] (AA) Start; (BB) No; (CC) Yes; (DD) End; (S401) Data input; (S402) Generating a data unit; (S403) Comparing meta data; (S404) Is a keyword extracted?; (S405) Automatically classifying unit data; (S406) Registering unit data; (S407) Arranging a keyword; (S408) Outputting a grouping result

Description

Method and system for automatic classification of data

The present invention relates to a method and system for analyzing and standardizing input data for data quality management and automatically classifying the standardized data.

In general, data quality management refers to data management and improvement activities performed to meet the expectations of information system and database users, both inside and outside an organization or organization. Because databases are linked to each other or sometimes to databases of other organizations within the same organization, it is important to maintain consistency between the data of the organizations to which they are linked.

In other words, as all the tasks of an organization are computerized, a large amount of data is generated, and the data quality decreases due to deepening of data duplication, inconsistency of terms, and unclear meaning of terms. This deterioration of data leads to data loss and rework, which results in a loss of organizational competitiveness.

Specifically, a ManiFestation Consolidation System (MFCS) is provided to assist in the collection and presentation of cargo lists of charterers and / or forwarders on board vessels or aircraft entering and departing on Korea's behalf. For example, the data quality problem is as follows.

When entering manifests into the manifest collection system, it is not uncommon for shipping companies and forwarders to enter different names for the same cargo or to enter them using non-standard words, abbreviations, dialects, etc., especially to avoid high tariffs. For malicious purposes, the name may be different from the actual item. In this case, there is a problem that the submission and approval process of the manifest is delayed because it does not match the standard name or the cargo code of the cargo stored in the manifest collection system database.

In addition, the shipping company and the forwarder enter the manifest in the respective way for the same item, and the same items are not grouped due to differences in the notation such as typos, synonyms, spacing, and abbreviations in the manifest. The problem of waste of time and manpower arises.

To solve this problem, a data grouping method for the manifest is proposed. Conventional data grouping techniques group when the input data (mutual) is the same as the standard name of the cargo stored in the database, as shown in Table 1, and automatically classifies the grouped input data by assigning a group ID.

Group id Mutual address 0000000006 AGRICOLA NACIONAL S A C E L ALMIRANTE PASTENE 300 SANTIAGO CHILE 0000000006 AGRICOLA NACIONAL S A C E L ALMIRANTE PASTENE 300, SANTIAGO, CHILE

However, in the conventional data grouping technique, when the same mutual name having different addresses is input, the address cannot be analyzed and grouped, and there is a disadvantage in that grouping is based only on the physical form of the mutual name.

For example, according to the conventional grouping method, the same group ID is given to some mutually different words, such as group ID 0000000030 in Table 2. If the IDs are the same as the ID 0000000073, but the addresses corresponding to the IDs are different from each other, an error occurs that both are given the same group ID.

Group id Mutual address 0000000030 CELULOSA ARAU Y CONTITUCION R.U.T 93.458.000-1 FABRICA DE CELULOSA 0000000030 CELULOSA ARAU Y CONST ITUCION S.A. SANTIAGO CHILE 0000000073 DISTRIBUIDORA AUTOMOTRIZ SANTIAGO S A COQUIMBO NRO.345 SANTIAGO / CHILE 0000000073 DISTRIBUIDORA AUTOMOTRIZ SANTIAGO S A SANTA ROSA 455 SANTIAGO

If such a problem occurs continuously, as shown in Table 3, mutually different physical forms and addresses are given the same group ID, resulting in a deterioration of the reliability of the grouping technique.

Group id Mutual address 0000000055 CONSORCIO MADERERO S A MONEDA 920 OF. 708 SANTIAGO 0000000055 CONSORCIO MADERERO S A AVENIDA ANDRES BELLO 2777 OR 2004 LAS CONDES SANTIAGO CHILE 0000000055 CONSORCIO MADERERO SA AVENIDA ANDRES BELLO 2777 OR 2004 LAS CONDES SANTIAGO

In addition, as shown in Table 4, the same items are classified into different items and grouped according to the item names of the shipping company and the forwarder. If the word of CUT, CUTTED, SPOT is added to the same item of declaration (CRAB FROZEN SWIMMING) or the order of the words in the item is changed by the shipping company and the forwarder, and the item of the notification is entered, Are classified into different items and grouped into five groups as shown in Table 4.

Group ID Count New article name A0800001 One FROZEN SWIMMING CRAB A0800002 One FROZEN CUT SWIMMING CRAB A0800003 One FROZEN CUTTED SWIMMING CRAB A0800004 One FROZEN SPOT SWIMMING CRAB CUT A0800005 One CUT CARB SPOT FROZEN SWIMMING

As a result of the grouping result of Table 4, the manual and / or the manual work should be performed in order to group the same item among the items with different declared item names in consideration of typos, synonyms, dialects, abbreviations, and spacing errors. It causes the problem of wasting time.

The problem to be solved by the embodiment of the present invention is to extract a meaningful keyword as a criterion for grouping from the input data, refine and standardize it according to a predetermined policy, and then propose a method for performing grouping using the standardized data. It is.

Another problem to be solved by another embodiment of the present invention is to propose a method for continuously upgrading standardized data for purifying input data.

In order to solve the above problems, the present invention provides a metadata database for extracting a keyword for data accumulated in a main database from a data input step and the input data, and generating and storing metadata according to the attribute of the keyword. And generating the unit data by dividing the input data into word or character units, and comparing the unit data with metadata and sorting and grouping extracted keywords according to the result. Is proposed as an embodiment.

The main database may be implemented as at least one of a database management system and a flat file database including data accumulated for a certain period of time.

In the metadata database construction step, the keyword may be generated according to parts of speech and attributes of valid data capable of grasping the meaning of the words by dividing the accumulated data into word units.

The metadata database may include an attribute database for storing metadata having a typo-correction, an abbreviation / slang / verbal / non-standard-word, and a plural-singular structure information according to the attribute of the keyword and a separator database for storing the invalid data. It may include.

In addition, the grouping step may extract the keyword in which the unit data is refined by comparing the unit data with metadata.

In addition, the keyword sort may be sorted in ascending or descending order according to the alphabetical order.

In addition, in the grouping step, if the same keyword as the unit data is not extracted, the unit data may be automatically classified as a registration target of the metadata database, and registered in the metadata database through the approval of an administrator.

In addition, the grouping step may output an input error, data duplication, and related data mismatch state of the input data according to the keyword search result.

According to an embodiment of the present invention, by automatically extracting and managing metadata from data and grouping and outputting the data according to a meaning or a specific purpose, the accuracy of data classification can be improved, and the time and manpower required for data analysis can be reduced. Can be.

1 shows a system for automatic classification of data according to an embodiment of the invention.
2 illustrates a state in which a freight list is automatically classified and registered in the system according to an embodiment of the present invention.
3 illustrates a metadata generation process according to an embodiment of the present invention.
4 is a flowchart illustrating a data grouping process according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may be embodied in many different forms and is not limited to the embodiments described herein. In the drawings, the same reference numerals are used to denote the same parts throughout the specification. In addition, when any part of the specification is to "include" any component, which means that it may further include other components, except to exclude other components unless otherwise stated.

Throughout the specification, "keyword" means data (standard words) in which input data is refined according to standards and / or criteria stored in the main database, and "metadata" means attributes and structures of the "keyword" and stored Means information about the location.

1 shows a system for automatic classification of data according to an embodiment of the invention.

The automatic classification system of FIG. 1 includes an input unit 10, a data management unit 20 for extracting a keyword from data input through the input unit, and storing metadata by specifying a keyword and storing the keyword. It includes a data processing unit 30 for outputting the purified and grouped data input through.

The input unit 10 is a means for receiving data in the form of words and / or sentences from an administrator or a user of the automatic classification system.

The data management unit 20 includes a main database 21 storing work data accumulated for a predetermined period of time and data newly input through the input unit 10, extracting keywords from the work data, and generating metadata. It includes a metadata generating unit 23 and a metadata database 23 for storing the keyword and metadata.

The main database (hereinafter, the main DB) 21 stores the data coming in through the input unit 10 in real time, and the work data includes data that has passed a predetermined period of time (for example, data that has passed one year) among the input data. Store them separately.

The main DB may be implemented as a database management system (DBMS) and / or a flat file database such as Oracle, Sybase, or SQL-Server including the work data.

The metadata generator 22 extracts a keyword from the work data, stores the attribute and structure of the keyword, and generates metadata that is a standard for grouping.

Specifically, when the work data is transmitted from the main DB 21, the metadata generator 22 divides the work data into word units and checks the frequency of use of the divided words.

When the work data transmitted from the main DB 21 is divided into word units, the metadata generator 22 classifies each of the divided words into valid data having a dictionary meaning and invalid data which cannot grasp the dictionary meaning. do. At this time, the valid data is extracted as a keyword of the work data transferred from the main DB (21).

The valid data is primarily targeted to the standard words that can be understood. However, if only the meaning can be understood, typos, abbreviations, slangs, and broad terms that are not standard words can be recognized as valid data. In the example of Table 5, TUNA, which means tuna, becomes valid data as a standard word, but even if 'TUAN' is mistakenly entered, TUAN is recognized as valid data if TUNA-TUAN is pre-registered as metadata in the main database.

In the case of 'DW CANNED TUNA', DW, the first word, is registered as an abbreviation of brand 'mobilization' in the main database, so it is considered valid data. In the case of the second word CANNED, the dictionary meaning is recognized as valid data as various broad terms, and the last word TUNA is also considered valid data because it has a dictionary meaning.

However, CRAB EXTRACT-MS1197 is regarded as invalid data because '-' and '1197' are classified as special characters and numbers that do not have a dictionary meaning, and the remaining words are considered valid data. .

Type of work data case typo TUAN-> TUNA Abbreviation DW CANNED TUNA (DW-> Abbreviation for 'mobilization' brand) slang TAREFS Broad term CAN (unclear meaning of verb, verb, proper noun, etc.) Data Combination Error ABALONE EXTRACT & LAVER EXTRACT Konglish DAKGALBI (Conglish notation of chicken ribs) brand MOKWOOCHON ('Mokwoochon' brand) Spacing error SOYSAUCE-> SOY SOURCE (write two words in one word) Meaningless words FROZEN BOILED RED model name CRAB EXTRACT-MS1197 (Model name of the product)

The metadata generating unit 22 is a typo-taking with a typo, a typo, and an abbreviation-standard word with a standard word in its original notation, and a slang-slang word with a standard word with the right word. A keyword having a structure as shown in the figure is generated, and valid data of the work data are classified by parts of speech such as nouns, adjectives, and adverbs.

In this case, the keyword generated by the metadata generator 22 is classified according to the keyword of the word pair structure and the keyword for each part of speech, as shown in Table 6, and metadata having information on the attribute and structure of the keyword is generated.

Keyword division Metadata W Keyword Noun form among valid data WA keyword Adjective form among valid data WK Keyword Key keywords among noun forms of valid data PS Keyword Plural data pairs PM Keyword Typo-Staking Data Pairs PP keyword Data pairs in the form of synonyms, dialects, slang-standards T keyword Compound pairs in compound / abbreviation-standard form PT Keyword Change-data-data pairs AB keyword Data pairs in abbreviation-standard form CH Keyword Data pairs in general chemical term and standard chemical term FL Keyword Foreign language information

The keyword classification and metadata according to the keyword are not limited to Table 6, and may be added according to the type and / or attribute of the keyword.

On the other hand, invalid data has the form of special characters, numbers, characters not found in the dictionary, dialects, etc., whose meaning cannot be understood as shown in Table 7.

Invalid data type case Special Characters :; ! @ # $% ^ & * () number 1 2 3 4 5 6 7 8 9 Word not in a dictionary onitsumo dialect tater (dialect of 'potato')

The metadata database (hereinafter, referred to as metadata DB) 23 stores a keyword generated through the metadata generator 22 and metadata specified for the keyword, and an attribute database for storing metadata about the attribute of the keyword ( The attribute DB) (not shown in FIG. 1) and the separator database (hereinafter referred to as separator DB) (not shown in FIG. 1) for storing the invalid data are included.

The attribute DB stores metadata of a word pair structure such as a typo-correction, an abbreviation, a slang, a dialect, a non-standard word, a plural-singular number, a compound word-standard word, etc. according to the attribute of the keyword.

The delimiter DB stores alphabets, numbers, special characters, and spaces belonging to the invalid data.

Returning to the description of FIG. 1 again, the data processor 30 generates unit data from the data input through the input unit 10 and extracts and groups the unit data and the metadata according to a comparison result.

To this end, the data processing unit 30 compares the unit data and metadata with the data converter 31 generating unit data from the input data, extracts a keyword, and classifies the input data according to the combination of the keywords. It includes a processing unit 22.

The data converter 31 generates unit data by dividing the data input through the input unit 10 into words and / or letters, and sends the unit data to the grouping processor 32.

The grouping processor 32 searches for metadata based on the unit data, and extracts a keyword specified in the metadata.

Specifically, the grouping processor 32 determines the invalid data when the unit data is composed of only numbers by grasping the related information of the unit data. If it is composed of a combination of numbers and letters, the number is considered invalid and the metadata is searched for the remaining valid data.

Then, keywords corresponding to the unit data are extracted from metadata, and the keywords are sorted in ascending and / or descending order in alphabetical order. The group ID is then assigned to the sorted keyword combination.

In addition, when the keyword corresponding to the unit data is not searched by the grouping processor 32, the grouping processor 32 outputs the keyword in the form of an error message or a warning message. The unit data may be automatically classified as a registration target of the metadata DB 23, and the metadata DB 23 may be upgraded by registering with the metadata DB 23 through a system administrator's confirmation and / or approval process.

2 illustrates a case in which a freight list is automatically classified and registered in the system according to an embodiment of the present invention when a freight list is input to a manifest list collection system (MFCS).

In FIG. 2, the report item name is data input through the input unit 10 of the system illustrated in FIG. 1, and the automatic extraction keyword is a combination of keywords matching the report item name searched through the grouping processing unit 22. The reported item names are classified and output.

When 'FROZEN 3 SPOTED CUT CRAB' is input to the declared product name through the input unit 10, the data conversion unit 31 is divided into five unit data, that is, FROZEN, 3, SPOTED, CUT, CRAB. The divided five unit data are classified into invalid data of FROZEN, SPOTED, CRAB and 3, CUT which are valid data through the grouping processing unit 32.

Thereafter, the valid data is compared with the metadata in the metadata DB 21, and a keyword matching the valid data is extracted and sorted and grouped as 'CRAB FROZEN SPOTTED' in ascending alphabetical order. Group ID (A0800001) is assigned to the grouped keyword combination.

3 illustrates a metadata generation process according to an embodiment of the present invention.

The metadata generation process of FIG. 3 will be described in detail with reference to the embodiment of FIG. 2.

First, the declared product name for generating the metadata is inputted as 'FROZEN 3 SPOTED CUT CRAB' (S301), and the declared product name includes data stored in the main DB 21 for a predetermined period of time from among the data inputted through the input unit 10. It is targeted.

The declared item name is divided into word units such as FROZEN, 3, SPOTED, CUT, and CRAB (S302), and FROZEN, SPOTED, CUT, and CRAB, which can grasp the meaning among the words, are classified as valid data and cannot be understood. Class 3 is classified as invalid data.

Subsequently, a keyword having a word pair structure is generated by combining standard words corresponding to the valid data based on the valid data FROZEN, SPOTED, CUT, and CRAB (S303), and analyzing the attribute and structure of the keyword (S304). After analysis of the attribute and structure of the keyword, metadata is generated and stored according to the result (S305).

For example, among the valid data, FROZEN and CRAB generate keywords, but do not generate separate metadata because they are standard words, and SPOTED generates keywords for the spoofed typo-taking, that is, SPOTED-SPOTTED word-pair structure. And generate metadata including the part-of-speech and structure information.

In this case, the CUT is valid data but is not defined as a keyword as a broad term, and metadata is not generated.

4 is a flowchart illustrating a data grouping process according to an embodiment of the present invention.

The data grouping process of FIG. 4 will also be described in detail with reference to the example of FIG. 2.

First, when the declared product name is inputted as 'FROZEN 3 SPOTED CUT CRAB' through the input unit 10 (S401), the declared product name is divided into word units to generate unit data such as FROZEN, 3, SPOTED, CUT, and CRAB. (S402).

Thereafter, the unit data FROZEN, 3, SPOTED, CUT, and CRAB are compared with metadata of the metadata DB 23 (S403), and a matching keyword (standardized data) is extracted (S404).

In this case, 3 is classified as meaningless number, that is, invalid data, and keywords are not extracted, and the same keyword is not searched because CUT is excluded from the keyword target as a broad term with an unclear meaning.

If the same keyword is not extracted, the unit data is classified as a registration target of the metadata DB 23 (S405), and registered in the metadata DB 23 through the administrator's approval (S406).

The extracted keywords are sorted in order of CRAB, FROZEN, and SPOTTED in alphabetical order (S407). The keyword is classified into a combination of CRAB FROZEN SPOTTED, and a group ID 'A0800001' is assigned to the keyword combination and output.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

10: input unit 20: data management unit 21: main database
22: metadata generating unit 23: metadata database
30: data processing unit 31: data conversion unit
32: grouping processing unit

Claims (8)

Data input step;
A metadata database construction step of extracting a keyword for data accumulated in a main database among the input data and generating and storing metadata according to the attribute of the keyword;
Generating unit data by dividing the input data into word or character units; And
Comparing the unit data with metadata and sorting and grouping extracted keywords according to the result
Automatic classification method of data comprising a. The method of claim 1,
The main database is an automatic classification method of data implemented in at least one of a database management system, a flat file database including data accumulated for a certain period. The method of claim 1,
In the metadata database construction step, a keyword is generated according to parts of speech and attributes of valid data capable of identifying meaning among the words by dividing the accumulated data into word units. The method of claim 1,
The metadata database includes an attribute database for storing metadata having a typo-correction, an abbreviation / slang / verbal / nonstandard-word, and a plural-singular structure information according to the attribute of the keyword, and a separator database for storing the invalid data. How to automatically sort your data. The method of claim 1,
In the grouping step, the unit data is compared with metadata, and the automatic classification method of data for extracting a keyword in which the unit data is refined. The method of claim 1,
The keyword sorting method is an automatic classification method of data arranged in ascending or descending order according to the alphabetical order. The method of claim 1,
In the grouping step, if the same keyword as the unit data is not extracted, the unit data is automatically classified as a registration target of the metadata database, and registered in the metadata database through the approval of an administrator. The method of claim 1,
The grouping step is an automatic classification method of data for outputting an input error, data duplication, and a related data mismatch state of the input data according to the keyword search result.

Images