JP7116940B2 - Method and program for efficiently structuring and correcting open data - Google Patents

Description

The present invention relates to a technique for extracting necessary information from so-called open data scattered on a network and for which access is not restricted, and for efficiently structuring and correcting the data.

Conventionally, in order to select and collect desired information from open data, it has been necessary to manually collect and accumulate information while browsing websites one by one using a browser.

For this problem, there are the following prior art documents that describe solutions for automatically collecting desired information from open data.

Patent document 1: Automatically extract tabular or database information from websites.
Patent Document 2: Desired information is extracted from a plurality of websites using definition files or dictionary files and output in XML format.
Patent Literature 3: Extracts structured information from web data using a structured model created by machine learning.

JP 2008-102773 A Japanese Patent Application Laid-Open No. 2010-15202 JP 2016-181069 A

However, Patent Document 1 only extracts information in tabular format or database format on websites, and does not support other data formats.

In Patent Document 2, when extracting information from a website, only information prepared in advance by a definition file or a dictionary file is extracted, and data to be output is limited to XML format.

In Patent Document 3, when structuring information extracted from a website, it is necessary to prepare structured model information based on teacher data created in advance, and the structured information must be consistent and consistent. Even if optimization is necessary, corrections are not made. Therefore, although the information is structured, it is not possible to utilize the information due to reasons such as inconsistency of data. There is no problem.

In addition, in any of Patent Documents 1 to 3, the data to be extracted is all text (character) data, and desired information is extracted from other types of data, the information is structured, and further corrected. No means are provided.

The present invention was created in consideration of such circumstances, and extracts information from open data in various formats that are scattered on a network and has unrestricted access. An object of the present invention is to provide an information processing method and a program for automatically structuring data by processing such as learning and correcting the structured data.

In order to achieve the above object, the present invention provides means for collecting text data from data specified by a network address and data specified by a network address estimated in the data, and means for shaping the text data. means for extracting necessary information by automatically selecting an optimum extraction method from the text data formatted by said formatting; and a structured processing database in which each of said information is implemented in a temporary storage area and means for correcting the information structured by the structuring by means of a correction processing database implemented in a temporary storage area. is characterized by

According to the information processing method and program according to the present invention, desired information can be efficiently extracted from data of various formats scattered on a network, and structured and corrected at high speed. The information generated by the processing of (1) can be easily utilized without further processing.

It is a figure showing the whole system composition of the present invention. It is a figure which shows the hardware constitutions of this invention. 3 is a diagram showing the functions and processing configuration of the present invention; FIG. FIG. 2 is a diagram showing a flowchart showing processing steps of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention; 2 is a diagram showing an example of a directory structure, site map, or the like created in the processing steps of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention; FIG. 4 is a diagram showing an example of shaped text data created in the processing steps of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention; FIG. FIG. 3 is a diagram showing an in-memory database creation processing process of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention; It is a figure which shows the processing process of the information processing apparatus 1 concerning the information processing method and program of 2nd embodiment of this invention. It is a figure which shows the processing process of the information processing apparatus 1 based on the information processing method and program of 3rd Embodiment of this invention.

A first embodiment for carrying out the invention will be described in detail with reference to the drawings.

[First embodiment]
FIG. 1 is a diagram showing the overall configuration of a system related to an information processing method and program according to a first embodiment of the present invention.

[Overall system configuration]
In FIG. 1, an information processing apparatus 1 is connected to a plurality of content servers 2 (1 to n (n is an integer)) through a communication network N. FIG.

The communication network N may be any network capable of mutually exchanging data by electrical signals, and may be the Internet, a mobile network, or a local area network. The connection with the communication network N may be wired or wireless.

The content server 2 provides various contents (data) in response to requests from the information processing device 1 . The provided contents (data) are, for example, text files, HTML files, image files, moving image files, audio files, and the like.

Also, the content server 2 only needs to have a function of connecting to a communication network and a function of storing content (data), and does not need to have a so-called server function.

[Hardware configuration]
FIG. 2 is a diagram showing the hardware configuration of the information processing device 1 according to the first embodiment of the invention.

The information processing apparatus 1 includes a CPU (Central Processor Unit) 10 for arithmetic processing, a RAM (RANDOM ACCESS MEMORY) 11 for temporarily storing data and programs, a storage device 12 for storing data and programs, and a network connection. A communication interface 13 is included. The information processing device 1 is also connected to an input device 14 and an output device 15 via a bus or the like.

Each component of the information processing apparatus 1 may have a plurality of components, and may be distributed physically or virtually. Further, they may be connected to each other by wire, or may be configured to be connected wirelessly.

The CPU 10 reads programs and the like from the RAM 11 and the storage device 12 and executes various commands to control the entire information processing apparatus 1 and realize the functions of each section described later.

The RAM 11 is a storage medium for temporarily storing application data being activated and input data, and may include a cache memory that the CPU 10 can access at a higher speed. An in-memory database, which will be described later, is a database mounted on the RAM 11, and enables faster and real-time data retrieval and collation than a database mounted on a hard disk or the like.

The storage device 12 stores a program that executes the functions of the present invention, structured data accumulated by a structured data holding unit (to be described later), dictionary data, and the like. It should be noted that the storage device 12 is not limited to a single medium and read/write method, and may be, for example, a magnetic hard disk, an optical disk, or a solid state drive.

The communication interface 13 is an interface for enabling connection with the content server 2 via the communication network N, and the communication method may be wired or wireless.

The input device 14 is implemented by, for example, a mouse, keyboard, touch panel, or the like, and is used for input operations. The output device 15 is implemented by, for example, a display, and is used to confirm the output of characters, images, sounds, and the like.

[Function and processing configuration]
FIG. 3 is a block diagram showing functions and processes of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention. Each part of this configuration represents a unit that executes functions and processes realized by software, for example, a group of instructions, methods, classes, or libraries in a programming language that execute each function and process. can be thought of as representing a set of

The information processing device 1 includes a data acquisition unit 20 , a directory structure extraction unit 21 , a data cleansing unit 22 , a data structure unit 23 , a structured data holding unit 24 and an in-memory database 25 .

The data acquisition unit 20 receives a network address as an input, and acquires the markup language or the like that constitutes the web page at the address destination as text data.

The directory structure extraction unit 21 comprehends network addresses derived from the acquired text data, and constructs a directory structure.

The data cleansing unit 22 performs shaping such as removing extra characters in order to facilitate processing of the text data.

The data structure unit 23 extracts necessary information from the text data shaped by the data cleansing unit, and structures the extracted information by adding a tag (additional information).

The structured data storage unit 24 stores the structured text data. It also stores various data to be loaded into the in-memory database 25 and used for matching or integration processing, which will be described later.

The in-memory database 25 is a database including data tables such as dictionary data and structured existing data temporarily stored in the RAM 11 in order to perform matching or integration processing, which will be described later.

FIG. 4 is a flow chart showing processing steps of the information processing apparatus 1 according to the information processing method and program according to the first embodiment of the present invention.

Step S1 (URL reading) accesses the location of the top page of the website (for example, "http://www.abc.com/") using a network address such as an input URL (Uniform Resource Locator). do. The URL to be input may be manually input each time, or may be input in order from the top of the list of URLs. In addition, the URLs to be listed may be automatically extracted from the history information of the browser, or a keyword search by a search engine may be used to automatically extract the URL from the top of the search results. good.

In step S2 (text data acquisition), HTML (Hypertext data) that constitutes a file (for example, ".htm" or ".html" file, hereinafter referred to as "web page") that can be displayed by a browser located on the top page. A markup language such as Text Markup Language is acquired as text data.

Step S3 (directory structure extraction) analyzes HTML tags (for example, <A> to </A>) included in the text data, and extracts links (addresses such as URLs) derived from the top page. , select a directory in which one or more web pages exist, and construct a directory structure or a site map, etc. centering on the top page, which is the root directory (FIG. 5). At this time, create the following directory table.

[Example] Directory table root directory|directory|read flag abc. com |company |1
abc. com |contact|0
abc. com |service|0
abc. com | fbook.com | jp/abc|0

As described above, the "read flag" attribute of the directory table is input with a value different from the initial value such as "1" (zero in the example) when the processing up to step S15 is performed for all the web pages existing in the directory. , moves to the next directory, and repeats the process from step S1 until values are input to all directories. The directory is not limited to the so-called folder of the directory system, and may indicate the external site of the link destination as in the above examples.

Step S4 (data type discrimination) discriminates the data type when there is an audio or moving image file or a document file under each directory constituting the directory structure. In the following, the flow when there is no audio or video file or document file (step S5: NO) will be described, and the processing when there is (step S5: YES) will be described in the second embodiment and the third embodiment described later. Described in form.

Step S6 (data cleansing) performs a parsing process for shaping the text data obtained in the text data obtaining step S2, which is so-called raw data, in order to prepare input data for subsequent structuring processing. Specifically, HTML tags and the like and meaningless blank symbols and the like are removed, leaving only meaningful combinations of words and sentences, and their order is rearranged (FIG. 6).

In step S7 (sentence division into morphemes), sentences forming the parsed text data are separated into morphemes (minimum units of meaningful expression elements).

Step S8 (extracting parts of speech) divides the morphemes into parts of speech, and extracts parts of speech that can be combined into words.

In step S9 (automatic generation of part-of-speech combination patterns), chunking between parts of speech is performed in order to automatically target new words that combine parts of speech and words that are not registered in the dictionary data or words that are not registered in the dictionary data. (segmentation or combination processing of sentence constituent elements) is automatically performed, parts of speech are combined, patterns with a high probability of frequent occurrence among the resulting patterns are recognized as terms, and subjected to structuring processing. Then, a search is made to see if the term exists in the in-memory database, and if it does not exist, it is added to the screening database in step S27 of FIG. 7 (step S10 NO). After that, whether or not the term is registered in the in-memory database (step S29 in FIG. 7) will be explained in the later-described in-memory database creation processing step. If the term exists in the in-memory database, go to step S11 (step S10 YES).

Not only the above-mentioned terms by combined parts of speech, but also the above-mentioned parts of speech that are not combined, parts of speech that are likely to occur frequently are recognized as terms and subjected to structuring processing. Then, the in-memory database is searched to see if it exists, and if it does not exist, it is added to the selection database (step S10 NO). After that, whether or not the term is registered in the in-memory database (step S29 in FIG. 7) will be explained in the later-described in-memory database creation processing step. If the term exists in the in-memory database, go to step S11 (step S10 YES).

In step S11 (matching with in-memory database), each of the parts of speech (including those recognized as the terms in step S9) and the in-memory database (after being added to the screening database, the in-memory It performs high-speed matching processing of attribute-value pairs that constitute an existing data table (for example, data table 1 below) loaded into a database (including terms registered in a database). Here, regarding the matching of the in-memory database, in order to deal with not only exact matches but also similarities, based on a statistical model or a machine learning model based on the appearance frequency, among the collected information contents It judges whether it is used as a general term or as a highly important term in , and extracts only the highly important terms (keyword extraction). The model is generated based on the number of occurrences of each word in the context and the calculation of the distance after replacing each word with a vector representation (semantic similarity, relationship, etc. are derived). For example, when extracting feature data described in a job offer web page, terms such as "benefits" and "social insurance" appear frequently because they are used as additional information about treatment in job offer information. Therefore, it is not necessary to exclude "welfare" and the like because it is suitable for the information collection theme of extracting treatment information. However, it is not suitable for the information collection theme of extracting job function (skill) information as a requirement for recruitment, so it is possible to exclude "benefits" etc. by using the extraction method based on the above model. can.

[Example] Data table 1
Company name ｜ Phone number ｜ Product ABC Co., Ltd. xxx-xxxx ｜ Ringo

If the network address indicating the root directory of the display table is confirmed to belong to ABC Corporation, the part of speech to be matched is existing data, for example, "company name: ABC Inc." data set and matching is performed. For example, the noun "fruit" is determined to match "apple" in data table 1 above. This determination may be made using dictionary data in which categories and elements belonging to them are registered, or may be determined by semantic analysis of "fruit" and the context before and after "fruit".

In step S12 (tag information addition), an attribute corresponding to the value of the data table determined to match the part of speech is tagged as additional information of the noun, and is input to the data table as a new data set. Taking the data table 1 as an example and visualizing it, the following data table 2 is obtained.

[Example] Data table 2
｜Company name ｜Phone number ｜Products Existing｜ABC Co., Ltd.｜xxx-xxxx｜Apples New｜ABC Co., Ltd.｜ ｜Fruit

Note that there are times when a part of speech that is presumed to be a new attribute or value that does not match the attribute or value of the data set is detected. In that case, a regular expression of the data expression pattern and a model for determining the similarity of the data attribute degree are used. This model is a model for determining whether or not language data are identical based on machine learning such as the K-means method, LSTM (Long Short-Term Memory), and CBOW (Countinuous Bag-of-Words) method. to perform attribute analysis. For example, in the text data of FIG. 6, the expression "company overview" is determined to be a frequent and meaningful word on company-related web pages. can grasp. Then, it becomes possible to grasp that the values ("processed sales", "at home", etc.) that are data subordinate to the corporate profile and are not in the dataset are values corresponding to the corporate profile, which is an attribute, and the dataset are entered as new attributes and values in the data table 3 below. Also, if "processed sales" and "at home" are determined to be attribute values of "industry" and "company culture" by dictionary data and attribute analysis, for example, input as shown in data table 4 below. can be

[Example] Data table 3
｜Company name ｜Phone number｜Products｜Company profile

[Example] Data table 4
｜Company Name ｜Phone Number｜Products｜Industry｜Corporate Culture New｜ABC Co.｜ ｜Fruits｜Processing and Sales｜At Home

By adding the tag information as described above, the attribute of each information can be determined, and the visibility, workability and usability of the data are improved. However, it is conceivable that there are attributes or values that are considered to overlap between data sets, and that consistency as information is not maintained.

In step S13 (integration (name identification)), the new data structured by adding the tag information and the existing data are integrated by name identification. Said existing data is stored in the structured data holding unit 24 and is loaded from the structured data holding unit 24 into the in-memory database 25 for efficient matching and keyed by one or more attributes and values of the new data. , the attributes and values of corresponding existing data are compared and integrated, and the integrated data is stored in the structured data holding unit 24 . Two methods of determining attributes to be compared can be used: a method of manually setting a group of attributes to be compared in advance and an automatic clustering method. When setting manually, select the attributes to be compared (eg, "company overview" comparison item; company name, telephone number, product). In either method, attributes to be compared are selected, and comparison and identity determination are performed.

For example, when "telephone number" is integrated, various description forms such as "main telephone number" and "telephone" appear dispersedly as data attributes to be compared. In order to identify these description forms as "telephone number", a regular expression of data expression patterns and a model for discriminating the similarity of data attribute degrees are used. This model is a combination of models for judging whether language data using machine learning such as the K-means method, LSTM, and CBOW method are the same terms. A similar method is used to compare and integrate the values corresponding to the attributes to be compared. Even if the identity of attributes is confirmed, the identity of values may not be confirmed. In that case, if the identity of multiple attributes is confirmed, or if the identity of the attribute that is considered to be the primary key (holding a value that can be an index) is confirmed by the data content, new data and existing data It is allowed for one attribute to have multiple values if it is determined that they are the same as a set.

A specific example of the integration (name identification) is shown below. The data sets of the data table 5 and the data table 6 become the data set shown in the data table 7 or the data table 8 by the processing of integration. Although the data table 5 and the data table 6 are represented as separate data tables for convenience, they may be data sets of the same data table.

[Example] Data table 5
｜Company name ｜Phone number ｜Product Existing｜ABC Co., Ltd.｜xxx-xxxx｜Ringo [Example] Data table 6
｜Company name ｜Representative phone number ｜Product New｜ABC Corporation｜ ｜Fruit

[Example] Data table 7
｜Company name ｜Phone number ｜Product integration｜ABC Co., Ltd.｜xxx-xxxx｜Fruit

[Example] Data table 8
｜Company name ｜Phone number ｜Product integration｜ABC Co., Ltd.｜xxx-xxxx｜Fruits, apples

Conventionally, if duplicates of attributes and values existed in the dataset, they were visually eliminated, but the above integration (name identification) can automatically solve this problem and improve consistency. The data in which the is maintained can reduce problems in subsequent utilization of the data. For example, since there are no duplicate telephone numbers or e-mail addresses in the data set that holds company information, it is possible to expect effects such as not inducing system errors or human errors in BtoB marketing.

For example, if the theme of information collection is job recruitment, etc., from the viewpoint of personal information protection, it is possible to carry out a process of replacing the person's name with anonymity at the same time as executing the integration (name identification). For example, it is conceivable to write a person's name using alphabets, numerical values, or a combination thereof, but it is sufficient that a certain person cannot be specified, and the anonymization method is not limited to this.

A step S14 (structured data storage) stores the data table holding the data set integrated by the integration (name identification) in the structured data storage unit 24. FIG. From the standpoint of strong security, it is possible to automatically encrypt the data table or the data contained in the data table when saving the data. Any encryption method may be used, but it is not necessary to use only one method, and a plurality of methods may be used. Encrypted and saved data is automatically decrypted when read (when needed). Also, a function of duplicating and compressing encrypted data and the like and automatically creating a backup may be added.

A step S15 decides whether or not the web page is the last web page in the directory to which the web page from which the text data is acquired belongs. If it is not the last web page, branch to NO and return to the process of step S2. If it is the last web page, branch to YES and proceed to the next step S16.

A step S16 determines whether the directory is the last directory. If it is not the last directory, it branches to NO and returns to the process of step S2. If it is the last directory, branch to YES and terminate the process.

FIG. 7 is a flow chart showing an in-memory database creation processing process of the information processing apparatus 1 according to the information processing method and program of the first embodiment of the present invention. Note that the in-memory database is not a set of training data generated from a fixed amount of data, but a set of new words, terms, and phrases used simultaneously with them. Moreover, although it is possible to create in advance, it is not necessary to create in advance.

The in-memory database creation processing steps are the same as steps S20 to S26 and steps S1 to S3 and steps S6 to S9 of the processing steps of the information processing apparatus 1 described above. These two processing steps may be executed separately in parallel, or the processing steps of the information processing apparatus 1 may be executed in place of the in-memory database creation processing steps.

Hereinafter, the explanation from step S20 to step S26 will be omitted, and the subsequent processing steps will be explained.

Step S27 (registration in the selection database) is a step of registering candidate terms and the like to be registered in the in-memory database in the selection database. It is assumed that the sorting database is temporarily stored in the RAM 11 like the in-memory database.

There are two patterns registered in the screening database. This is the case where the pattern registered after the process of step S26 (automatic generation of the part-of-speech combination pattern) immediately above and the processing process step S10 of the information processing apparatus 1 in FIG. 4 branch to NO.

Step S28 (selection of optimum method) selects a method to be used when constructing each data table stored in the in-memory database 25 using or utilizing the terms registered in the screening database. Methods to be selected include, for example, machine learning methods, statistical methods, and deep learning methods, and an ensemble method that selects the most suitable method for improving accuracy from these methods on a case-by-case basis is used. .

Using machine learning techniques as an example, text data is broken down into words using morphological analysis technology, which is a natural language processing machine learning technique, and the number and pattern of occurrences of terms and data in the entire data to be analyzed is calculated. are counted and analyzed, and dictionary data showing words and their frequency of occurrence and an attribute discrimination model are generated.

Also, taking a statistical method as an example, semantic analysis is performed based on the appearance frequency of words in a group of texts forming each web page. For example, if one of the web pages that make up a recruitment website has the expression "job title: engineer position recruitment", the expression "job title" is a frequent and meaningful phrase on recruitment-related web pages. Therefore, it can be understood that the job title is attribute information for structuring. Then, it becomes possible to grasp that the data subordinate to the job offer name is the value (in this case, "Engineer position recruitment") corresponding to the job offer name, which is an attribute, and is learned as one pattern of attributes and values.

In methods using deep learning techniques, for example, the CBOW technique is used to grasp the bias of content information on a word-by-word basis in the input layer into which the data to be analyzed is input, and the main contents of the content are identified by group-of-word units. comprehend with In addition, for word-by-word recognition, in order to follow new words, coined words, technical terms, and industry terms, we use dedicated dictionary data created in advance and automatically chunk between words to achieve frequent occurrences. Automatically add terms that are likely to be used to dictionary data. For example, "virtual currency" and "currency" are usually separate words, but when the word "virtual currency" appears frequently, the word "virtual currency" can be recognized as a single term.

Step S29 (Register in in-memory database) registers in the in-memory database the dictionary data, models, patterns, new words, etc. selected or generated by the optimum method, and ends the process. However, in step S27, when the pattern registered in the selection database is a case where the processing process step S10 of the information processing apparatus 1 in FIG. The process transitions to step S<b>11 of the processing process of the processing apparatus 1 . From the standpoint of strong security, it is possible to automatically encrypt dictionary data, models, patterns, new words, etc., or data contained therein, when performing the registration. Any encryption method may be used, but it is not necessary to use only one method, and a plurality of methods may be used. Encrypted and registered data is automatically decrypted when it is read (when needed). Also, a function of duplicating and compressing encrypted data and the like and automatically creating a backup may be added.

The in-memory database constructed as described above is used for matching in the process step S11 of the information processing apparatus 1 in FIG. By using the in-memory database, it is possible to perform high-speed matching of data according to a theme and advanced structuring corresponding to language fads and the like.

[Second embodiment]
The second embodiment is an example in which data to be structured is included in a document file.

The configurations of the second embodiment and the third embodiment, which will be described later, are the same as those of the first embodiment (the overall system configuration, hardware configuration, functions, and processing configuration), so they are omitted.

FIG. 8 is a flow chart showing processing steps of the information processing apparatus 1 according to the information processing method and program of the second embodiment of the present invention. Note that steps up to step S30 are the same as steps S1 to S4 of the processing steps (FIG. 4) of the information processing apparatus 1 of the first embodiment, so description thereof will be omitted and subsequent processing steps will be described.

In step S30 (document type determination), when a document file is detected in step S4 (data type determination) of FIG. The document type is determined mainly by the file extension as to whether or not it can be the target of If the document type is any of document, spreadsheet, presentation, PDF, and XML, proceed to step S32 (step S31 YES), otherwise proceed to step S35 (text structuring). (step S31 NO).

A step S32 detects whether data in a table format exists in the data of the document file of document, spreadsheet, presentation, PDF or XML type. If the data in the table format is confirmed, the process proceeds to the next step S33 (YES), otherwise, the process proceeds to step S35 (structuring of normal text) (NO).

In step S33 (extraction of table, etc.), data in table format is extracted from the document file while it is in table format. The data is typically extracted in a format such as Data Tables 9 or 10 below.

[Example] Data table 9
Item number｜Company name｜Capital｜
1 | ABC Co., Ltd. | 1 million yen |

[Example] Data table 10
Item No. 1
Company name | ABC Co., Ltd. Capital | 1 million yen

Step S34 (structuring of tables, etc.) comprehends data dependencies (attributes and values), detects and sets data that can be used as indexes, and organizes data for each row or column to create a data set. , to complete the structuring.

There are two methods of structuring. (1) a method using an in-memory database; and (2) a method using the statistical method described in the first embodiment.

In the method (1), first, each of the data entered in the first row or first column of the data in the table format and the attribute column or attribute row of each data table constituting the in-memory database Match each data. Then, either the first row or the first column, whichever has the greater number of occurrences of matching data, is set as the attribute row or the attribute column, respectively.

When the attribute row is set, the data in the table format are grouped as a data set for each row, and when the attribute column is set, the data are grouped as a data set for each column, thereby structuring the data.

Whether the data is organized by rows or by columns, the data is structured in a format such as the following example.
[Example] Data table 11
Item number: 1, Company name: Company A, Capital: 1 million yen Item number: 2, Company name: Company B, Capital: 100 million yen ・・・・・・・・・・・・

The values belonging to the above data table 11 "item number" are a series of numbers and each value is unique by analyzing the appearance pattern and character type using the machine learning method described in the first embodiment. I understand. Therefore, set the attribute "item number" and its value as an index. If a value to be set as an index is not detected, it is possible to add an attribute (for example, "item number") and consecutive numerical values starting from 1 at the beginning of the data set.

In method (2), semantic analysis is performed based on the appearance frequency of words in the text group that constitutes the document file, as in the case of the first embodiment.

For example, if the document file is IR-related material for a company, it is determined that words such as "company name" and "capital" appear frequently and are related to the material. Therefore, for example, in the above data table 9, it can be determined that the first row is a set of attributes. Also, considering the meaning of the word "item number", the value corresponding to the attribute "item number" may not be the "company name" or "capital" aligned to the right, but the numbers aligned downward. It can be confirmed that there is an attribute in the first row also from the fact that the probability is high.

After determining that there is a data set for each row with the attribute "item number" at the beginning, the data is summarized for each row in the same manner as in method (1), and the same format as the data table 11 Structure the data with The same applies to the case where it is determined that the first column is a set of attributes.

In step S35 (text structuring), when the data in the table format organized row by row or column by column no longer exists or when the data in the table format did not exist from the beginning, the data in the table format Structuring general text parts other than The structuring of the text portion is performed by the same method (steps S6 to S12) as in the first embodiment. Note that if the text portion does not exist, text structuring processing is not performed.

When the structuring of the text portion is completed, the process transitions to step S13 (integration (name identification)) in FIG. 4 according to the first embodiment, and the subsequent processes are the same as in the first embodiment.

[Third embodiment]
The third embodiment is an example in which data to be structured is included in an audio or video file.

FIG. 9 is a flow chart showing processing steps of the information processing apparatus 1 according to the information processing method and program of the third embodiment of the present invention. Note that steps up to step S40 are the same as steps S1 to S4 of the processing steps (FIG. 4) of the information processing apparatus 1 of the first embodiment, so the description thereof will be omitted and the subsequent processing steps will be described.

Step S40 (determination of audio/video type) is performed when a document file is detected in step S4 (data type determination) of FIG. The type of the audio or video file is determined, mainly by the file extension, whether it can be subjected to further processing. If the type of object to be processed cannot be detected, the process is aborted.

In step S41 (converting speech into text), speech is converted into text by the speech recognition module in the case of an audio file, and the speech portion in the case of a moving image file.

When the audio of the audio file or the audio part of the video file has been converted to text, the process transitions to step 23 (data cleansing) in FIG. 4 according to the first embodiment, and the subsequent processing is the same as in the first embodiment. is.

Although the embodiments of the present invention have been described above, the above-described embodiments of the present invention are merely enumerations of typical embodiments, and are not limited to the above-described embodiments. Further, the effects described in the above-described embodiments of the present invention are merely enumerations of the most suitable effects resulting from the present invention, and the effects of the present invention are not limited to the effects described in the above-described embodiments of the present invention. Not limited.

1 information processing device 2 content server 10 CPU
11 RAM
12 storage device 13 communication interface 14 input device 15 output device 20 data acquisition unit 21 directory structure extraction unit 22 data cleansing unit 23 data structure unit 24 structured data holding unit 25 in-memory database

Claims (7)

an information processing device collecting text data from data specified by a network address and data specified by a network address estimated in the data;
formatting the text data;
a step of extracting necessary information by automatically selecting an optimum extraction method from the text data formatted by the shaping;
structuring each of the information by adding additional information using a structuring processing database implemented in a temporary storage area;
and a step of correcting the information structured by the structuring by using a correction processing database mounted in a temporary storage area. a step in which the information processing device determines the data type of the data specified by the network address and the data specified by the network address estimated in the data;
collecting text data from the data by a method appropriate for the data type;
shaping the data collected by said collecting;
a step of extracting necessary information by automatically selecting an optimum extraction method from the data shaped by the shaping;
structuring information by adding additional information to each of the information extracted by said extraction using a database for structuring processing implemented in a temporary storage area;
and a step of correcting the information structured by the structuring by using a correction processing database mounted in a temporary storage area. 3. The information processing method according to claim 2, wherein said data type is either audio, moving image or document file. 2. From claim 1, wherein, in said structuring step , if a new word/phrase that does not exist in said structuring processing database exists in said information, a step of registering said word/phrase in said structuring processing database is executed. The information processing method according to claim 3 . If there is data about a person in the data held in the database for structuring processing or the database for correction processing implemented in the temporary storage area, a step of anonymizing the name of the person at the same time as integrating the data. 5. The information processing method according to claim 1, wherein: The data held in the structuring processing database and correction processing database implemented in the temporary storage area are automatically encrypted, and decrypted only when data matching, data integration, and data correction are performed. 6. The information processing method according to any one of claims 1 to 5, characterized in that a step to be executed is executed. A program for executing the information processing method according to any one of claims 1 to 6 on a computer.

Images