TWI453608B - System and method for managing a large number of multiple data - Google Patents

Description

System and method for managing a large number of multiple data screening management

The invention relates to a system for mass multivariate data screening management and a method service system thereof, in particular to a function of simultaneously crawling multiple data sources, combined with a point of interest search engine, a large amount of data classification, a large amount of data management, and a large amount of data. Compared with a number of software technologies, and a large number of data extraction modules to query the database, you can meet the large number of store information or geographic location query requests, and automatically report the results to customers (E-mail or newsletter). A query service system that can reduce a large amount of manpower and improve the correctness of the data.

Before the launch of the present invention, it is still necessary to manually search for popular websites and provide the website information or geographical attractions that the customers need, and the search speed is quite slow, and the store is confirmed after the store information is found. Whether the address and phone number are correct (or are in the process of continuing to operate), and the store data has not been used to compare the search technology. The improvement of the correct rate is quite difficult. The data that encounters the error can only rely on the manual to do error analysis, and then improve the data. The shortcomings of the correct rate cannot be improved, and after the query program is finished, the query result cannot be sent to the client immediately. It is still necessary to rely on the manpower to organize the data and then forward it.

It can be seen that there are still many places where the above-mentioned methods of use can be improved. It is not a good design and needs to be improved.

After searching for Taiwan's public patents, there are two patent cases similar to the present invention. The articles are: "Local Search Service Use and Provision Method and Its Program Products, Directory Service Platform and Architecture" and "Data Acquisition Device, Data Acquisition System and Access Data Method" patent case. The "Local Search Service Use and Provision Method and Its Program Products, Directory Service Platform and Architecture" patent case is mainly a method of providing local search service, which is executed by a directory service platform, which stores most local search engines. Data, and when receiving a list download request from a terminal device, selecting at least one of the local search engine data to form a service list for returning to the terminal device, the service list containing each selected local search The local search engine option corresponding to the engine can be connected to the corresponding local search engine when each local search engine option is selected, so as to automatically provide local search engine data. The patents in this article are not focused on searching for local service materials. Instead, they can choose to enter "keywords" to search for information published on major websites. The search method proposed by the present invention can read information published on a large number of web pages at one time. And set the original database to match, accurately find out that the database does not contain data, if there is any change data can enter the manual review, so "local search service use and supply methods and its program products, directory service platform and architecture" The inventive concepts proposed by the invention are greatly different.

The content of the patent acquisition device, the data acquisition system and the method for obtaining data is mainly through the communication unit and GPS to obtain data. The present invention only needs to acquire data through a Point of Interest (POI) search engine module, and calculates its latitude and longitude data through the location, without additional hardware facilities; and "data acquisition device, data acquisition system, and data acquisition method" "This patent, from the 20th of the case, observes the entire process, The correct rate of store business hours is that the user stays at the point of interest for a sufficient period of time to judge whether the time data is correct. The disadvantage is that the user will find that the store has no business when he arrives at the destination, but the concept of the invention can be By comparing the data stored in the prior database with the data obtained on the Internet, if there is a difference, it will enter the manual review stage, thereby improving its correct rate.

In addition to using a large number of data comparison modules and Enterprise Application Integration (EAI) data matching modules to improve the accuracy rate, the present invention combines the search engine modules of the points of interest for crawling web pages, and self-developed. A large number of data classification modules and a large number of data management modules can start a series of multi-variable data screening management processes after crawling webpage data; first input multiple global resource locators for crawling websites (Uniform Resource Locator) To query data, large-scale data classification and management, data comparison, rapid manual review, data return and induction, customer application, mass data extraction, billing, and transmission of query data results to customers.

In view of the shortcomings derived from the above-mentioned methods, the inventors of the present invention have improved and innovated, and after years of painstaking research, they decided to adopt the current advanced software technology to develop the multi-data search and comparison system for the interest points of the telecommunication business. Finally, we successfully developed and completed the system and method of a large number of multi-dimensional data screening management.

The object of the present invention is to provide a system and method for screening and managing a large number of multiple data, and to establish a system capable of solving "a large number of diverse data sources but causing multiple data repetitions", that is, a telecommunication service point of interest service system that can automate operations. And automatically classify and manage data, analyze customer's inquiry request form, and can query a large number of popular stores and geographic information materials on the Internet in a short time, and use the functions of a large number of data extraction modules, according to the customer's The need to extract data from the database store can get accurate results in a short period of time.

The second object of the present invention is to integrate a large number of data comparison modules and an enterprise application system to integrate data comparison modules, and propose a large number of data source manual review interfaces and a large number of data extraction manual review interfaces, and compare with the original database. Data difference; after manually storing the audit results into the database, the old store information and geographic information in the database will be automatically updated to improve the correctness of the data.

A large number of multi-dimensional data screening management systems and methods for achieving the above-mentioned objects are based on the ability to simultaneously crawl multiple sources of data, combined with web search engines, large-scale data classification, large-scale data comparison, and the like, and matching A large number of data extraction modules to query the database to meet the large number of store information or geographic location query requests made by customers, and automatically report the results to customers (E-mail or SMS), which is a query service system that can reduce a lot of manpower. And to improve the correctness of the information. These include: a. The point of interest search engine module, the main function is to perform scheduled work, use the point of interest search engine system to capture store information, and classify it into index POI WEB DB or index point of interest data. Library (Index POI DB), and the index point of interest web database as the store information to be reviewed, and the index point of interest database as the store information to be classified; b. a large number of data classification module, the main function is to accept the point of interest search The instructions transmitted by the engine module, according to the captured store data content or geographic information, if new data is introduced, will be actively classified into the store basic information (store name, store phone, store address) or store value added Information (business hours, product prices, transportation methods, store profiles, number of rooms, etc.), for integrated planning of the data, and with the reviewed and classified store information, back to the point of interest host database Ability, and inform a large amount of data to have new data in the module, you can start this module to process data; c. A large number of data comparison modules, mainly In order to accept the instructions transmitted by a large number of data classification modules, the crawling network data of the execution network is stored in the index interest point webpage database and the index interest point database for comparison operations, thereby finding that the interest point database does not contain Store information or differential store information, and send the found store data path to a large number of data source manual review interface for audit action; d. Unique Identification (Unique Identification) definition module, the main function is to accept a large number of data source host The data transmitted by the database, and the collated data is written into the database, and a specific label field specified is a unique label, and the format is ASCII encoding, and the new specific label field is set. Move to the point of interest host database management, and notify a large number of data management modules; e. A large number of data management modules, the main function is to accept the instructions passed by the unique label definition module, to read data under the specified database And classify, and plan the scheduled schedule to transfer the store data or geographic information to the POI search engine database, and is a standard format acceptable to the system; f. Enterprise application system integration data comparison module, the main function is the execution data ratio For the production operation, the instructions transmitted by a large number of management modules are generated to generate corresponding data, and the data source is the database of the point of interest search engine and the host database of the point of interest, and the generated data path is sent to the large amount of data extraction. The manual review interface performs the auditing action, and informs a large number of data management module that the data processing operation has been completed. The enterprise application system integration data comparison module is the core function of this patent; g. a large number of data extraction modules, which will be standardized The store information is read from the point of interest host database, and the billing module is notified, and the short message is notified to the customer. In this process begun applications filed parts; H billing module executed based costing and inform the client, and the end of the calculation result sent to the client track line and associated personnel.

The invention can simultaneously process multi-party data sources, analyze various store formats or geographic information transmitted by different webpages, and can query a large amount of materials in a short time, and integrate the data matching modules by using the enterprise application system developed by ourselves. Compare the original store data in the host database with a large number of sources to get more accurate results. And combined with the charging module, the customer is informed that the case has been processed, and this inquiry will be charged. The cost, after the query result output standard format, for example: (xml, pdf), will automatically be returned to the customer according to the EMAIL address left by the customer. The main function is based on the object orientation (Object Oriented) concept, using the Chinese search engine FAST ESP to search the content of the web pages of each website, and using Microsoft VB.Net language as a programming tool, using MS SQL Server 2008 as data. The library, combined with the transformation and encapsulation technology of MS SQL Server Integration Services, uses HTTP and TCP/IP as the communication medium between the system and other interface systems, and integrates the above overall concept to develop the system.

Overall, the system combines a telecom service inquiry system, a point of interest search engine system, a large number of data classification and management modules, a large number of data comparison modules, a large number of data collection modules, a charging module, and a telecom service value-added service. A number of functions make it a single system, and it has the functions of graphics and automation, which greatly enhances the convenience of telecom business maintenance management.

In order to explain in detail the structure and features of the present invention, the following preferred examples are illustrated and illustrated with the following description:

Please refer to FIG. 1 and FIG. 2 first, which is a method for inputting a webpage seed by an operator of a plurality of multi-data screening management systems and methods thereof, and the operator must first join a large number of crawling webpage seed services. Enter any keyword in the keyword field of "Seed Collection" and save the URL according to the specified function. After that, collect a large amount of store information of the URL.

The necessary fields for the basic information of the store are: store name, store address, store association Network phone; other major value-added data fields (non-essential) are: business hours, consumer price, MRT station (check out the MRT station near the store), product features (why will someone push), traffic information, room The number of the number of rooms (the number of rooms in the accommodation), the location of the location (the relative location of the inquiry), the type of the attraction, the service item, the driving route, etc., the basic information and the value-added information of the store will be handled by the operator. Set in the point of interest search engine module 1.

Please refer to FIG. 3, which is a system architecture diagram of a system and a method for filtering and managing a large number of multiple data according to the present invention, including relevant components of the customer application inquiry (2, 21), and related components of the operator matching (3, 31, 32, 33) and system components (1, 11-19) of the present invention. The system components of the present invention include:

a. The point of interest search engine module 1 performs a scheduled work, uses the point of interest search engine system to crawl the store data, classifies it into the index database 31, and uses the index point of interest web database as the store to be audited. The information, and the index point of interest database as the store information to be classified, the operation includes: (a) when the user operates incorrectly or the input parameters are invalid, an error message is generated; (b) the network is disconnected, Or when the database fails to respond, an alarm message is generated; (c) the search scope includes the contents of the websites covered by the major websites and the seed list; (d) the scheduled schedule checking function is provided, and if the job processing requires other When the function module is coordinated, a processing message is generated and The instruction is sent to the relevant function module; (e) the content of the database (some fields of the basic information of the shop) can be imported into the search engine, that is, the basic information of the point of interest must be resident in the search engine index database 31. And scan the website list (Seed List) whether the website contains the relevant information of the relevant store in the index database 31, and output related reports; (f) display the specified message on the screen according to the requirements of other function modules , or update the specific screen display; (g) according to the requirements of other functional modules, display its error message on the screen, print it on the report, and record it in the system event log database; (h) according to the operator The requirements can be used to query the system's event records, call records, and which list of sites is currently being processed, and can generate reports at any time; (i) fully automated according to the operator's needs The adult job industry is started, and the direct operating system processes a Uniform Resource Locator.

b. The mass data classification module 11 receives the new data from the designated database and notifies the large data comparison module 12 to start other modules to process the file, and the operation includes: (a) if new data is introduced If the specified host IP does not exist, or the specified directory is invalid, an error message is generated on the screen; (b) an error message is generated on the screen when the receiving or updating data process fails or is incorrect; (c) receiving or updating the data When finished, check if the received data is complete. If it is not complete, it will generate an error message. (d) When receiving other function module messages, display the message directly on the screen; (e) When the receiving index database 31 is completed, The data will be classified according to its characteristics and imported into the relevant information tables of the POI search engine database 32; (f) The manual review interface 13 of a large number of sources can be received and updated, and a large amount of correct data reviewed will be followed. POI search engine to the storage database 32 of the data table; (G) daily using SQL Server Integration Services provide timing scheduling function, and can receive the output data to the main point of interest Database 33.

c. A large number of data comparison module 12, which performs the online crawling to store data stored in the index database 31 for comparison operations, and the operations include: (a) the main function is to perform network crawling to store data storage. The index database 31 performs the database comparison operation, and only needs to compare the information of the "basic data" in the index point of interest database database and the index point of interest database, and then set the data as information to be reviewed; (b) The difference information or the store information not included in the index database 31 can be viewed through the audit screen, and sent to a large number of sources of manual review interface 13 for audit reference; (c) A large number of sources The manual review interface 13 can support multiple users at the same time, avoiding the fact that a certain data is locked by the competition at the same time, resulting in unrecognizable data. After entering the review interface, the user who presses the archive button can be recorded. The login ID, and the current archive time, are then written to the temporary table of interest points; (d) When the review screen displays the data difference, the difference data will be changed to a red font display, and each field is provided. The function that can be edited, if there is a difference in the data results of the two parties, the editor-based information of the interest point webpage database is mainly used, and the index database 31 (requires the value-added content field) is written back.

d. The unique tag definition module 14 writes the defined data to the point of interest host database 33, and its operations include: (a) accepting a large amount of data source host database 3, and will The sorted data is written to the point of interest host database 33, and the specified specific label field is a unique label, and the format is similar to the ASCII encoding mode, and the new specific label field is stored in the point of interest host database 33 management. (b) In the process of writing, if a data format error occurs, or any abnormal error occurs, the error data is written to the record.

e. A large number of data management modules 15 are data extracted from the point of interest host database 33 and need to be managed, and determine the characteristics of the input data to determine which type of point of interest host database 33 is classified. In the form, its operation includes: (a) managing all points of interest data in the host database 33 of the point of interest based on the specific tab field; (b) receiving data from a large number of sources of the host database 3, The characteristics determine which type of data table is stored in the point of interest host database 33; (c) can receive and update a large number of data extraction manual review interface 17, a large amount of correct data reviewed, will be stored to the point of interest The data table of the host database 33; (d) generating an alarm message when the network is disconnected, or when the database fails to respond; (e) daily using SQL erver Integration Services to provide a scheduled scheduling function, which can receive And output the data to the database 32 of the point of interest search engine.

f. The enterprise application system integration data comparison module 16 performs the data comparison generation operation. The data source is the point of interest search engine database 32 and the point of interest host database 33, and the operations thereof include: (a) the generated The data path is sent to a large amount of data extraction manual review interface 17 for review actions; (b) through the enterprise application system integration data comparison module 16 to check the data returned by the large number of sources of manual review interface 13, whether the store phone number exists, if If it does not exist, delete it. Otherwise, if it exists, skip to the next step; (c) then read the POI database table and check if the number exists in the table. If it does not exist, it is marked as new to be reviewed. Information; if it exists, compare it with the content of the form, If it is different, it will be marked as the difference data to be reviewed; (d) According to the query conditions, the information of the main point of the interest point and the FAST UGC source data to be reviewed/reviewed can be inquired.

The mass data extraction module 18 reads the standardized store data from the point of interest host database 33 and notifies the billing module 19, the operation of which includes: (a) extracting the interest according to the application file 21 filed by the client. Point to the demand data of the host database 33, and output the query result to the standard format, for example: (xml, pdf, xls); (b) automatically send a short message to notify the customer 2 that the application has been processed; (c) if After processing, the billing module 19 is automatically notified.

h. Billing module 19, which performs cost calculation and informs customer 2, whose operation includes: (a) pricing based on the number of documents queried by the case number; (b) if there is a large amount of information required for inquiry, the price is additionally Calculate; (c) automatically bring out the customer 2 ledger number and transfer it to the accounting system; (d) automatically send the billing notice to notify the customer 2.

The steps of the system and method for completing a large number of multivariate data screening management according to the present invention are as follows:

a. The operator searches the engine module 1 GUI interface from the point of interest, and inputs the keyword or global resource locator of the website to be searched (Uniform Resource Locator) (The format is as shown in Figure 1), and the data is processed in the prescribed format.

b. After the GUI interface is operated, another GUI interface of the search engine module 1 can be accessed to the point of interest, and the contents of each global resource locator (the format of the Uniform Resource Locator) can be edited and deleted (the format is as shown in FIG. 2).

c. The department will perform scheduled scheduling operations, use the point of interest search engine module 1 to capture the store data, classify it into the index database 31, and use the index point of interest web database as the pending store information, and the index interest The point database is used as the store information to be classified, and the data is sent to the mass data classification module 11.

d. The mass data classification module 11 will actively classify the basic data belonging to the store or the value-added data of the store according to the characteristics of the data, integrate the data, and transmit and receive the data to the host database 33 of the point of interest.

e. A large number of data comparison modules 12 need to compare the information of the "basic data" in the interest point web database and the point of interest database: (a) first compare the full number of the phone number FullTel field, and check the points of interest The table in the database, whether the POI_ENABLE field of the phone number is 0, which means that it is in use, and a phone number may have many pen history records. If there is a piece of data in the phone number history record, it is continued. The following inspection process; if there is no data in use, jump directly from the following inspection procedures, and discard the data, on behalf of the phone has been disassembled, the content on the network does not need to trust; (b) again is the installation Address device_address field, the level of detail to a few floors, if the address is the same, then continue to compare the store name information; (c) then compare the store name POI_name field, and than its store The similarity of the names to determine whether it is the same store; (d) If the information retrieved on the Internet is the same, the phone number, the installed address, and the store name are the same, and no value-added data exists, and the pen is directly discarded; The information searched on the Internet has the same phone number, installed address, and store name, and there is a value-added data, and the information needs to be imported into the data area to be audited; (e) if the information retrieved on the Internet, its phone number, The installed addresses are the same, but the store name is different, directly import the data to the data area to be audited, and filter the name of the store that was retrieved from the database of the interest point database to filter out irrelevant content such as E-MAIL or preferential information; (f) If the information retrieved on the Internet is the same, the telephone number and the store name are the same, but the installed address is different. In principle, the information of the interest point database is the main one. If there is value-added data, the data needs to be imported to be reviewed. In the data area, if there is no value-added information, directly discard the pen; (g) if the information retrieved on the Internet, the phone number does not exist in the database of interest points, ie The information to be reviewed data area.

f. After a large amount of data comparison module 12 generates the data to be reviewed, a large number of data source manual review interface 13 will be used for the review action, and after the review is completed, the interest point search engine database 32 is returned.

g. The unique tag definition module 14 will receive each piece of data plus a particular tag field and then transfer the data to a plurality of data management modules 15 for management.

h. A large number of data management modules 15 based on the data characteristics, active classification of relevant data, integrated planning of data, and transmission and reception of data to the point of interest search engine database 32.

i. The enterprise application system integration data comparison module 16 checks the data returned by the manual review interface 13 of the large data source through the enterprise application system integration data comparison module 16 whether the store phone number exists and the generated data will be generated. The route is sent to a large amount of data extraction manual review interface 17 for audit action.

j. A large amount of data extraction manual review interface 17 will return the reviewed data to the point of interest host database 33.

k. Customer 2 fills in the application document 21 and fills in the information in the prescribed format to start processing the application.

After the application file 21 is completed, it is sent to the mass data extraction module 18, and the large data extraction module 18 will extract the demand data from the point of interest host database 33 and then transmit it to the billing module 19.

m. At the end of the processing of the billing module 19, a report data and a charging brief will be generated, and the report will be sent to the customer 2 by using E-MAIL.

Please refer to FIG. 4 , which is a data flow sequence diagram of a system and a method for filtering and managing a large number of multiple data according to the present invention, which illustrates a sequence of calls between modules, sorted according to call time, from top to bottom, from left to right. .

The top and bottom of the diagram are the modules and customer personnel that make up the system. The indicators can be divided into three types: solid black triangle head indicator, dotted triangle head indicator, and solid black triangle head indicator. The solid black triangle head indicator represents the solid line. The call module and the start module execute the command; the dotted triangle header indicator represents the response message to the original caller; the solid black triangle header indicator represents the transfer of the data to the accepting end.

From the top of the diagram you can see the following modules:

1. Point of interest search engine module 1.

2. A large number of data classification modules 11.

3. A large amount of data is compared to module 12.

4. Unique tag definition module 14.

5. A large number of data management modules 15.

6. The enterprise application system integrates the data comparison module 16.

7. A large number of data extraction modules 18.

8. Billing module 19.

In addition to these modules, there are operators and databases, and the role of the client personnel. The order is to input the global resource locator (Uniform Resource Locator) from the leftmost operation terminal, and the point of interest search engine module. 1 starts to call a large number of data classification modules 11, and receives a large number of data classification modules 11 to return a message to the point of interest search engine module 1, requesting the point of interest search engine module 1 to download the data to a specified directory, and hand over the data to a large number of The data classification module 11 processes, after receiving a large number of data classification modules 11, it will first verify whether the data format is correct. If there is an error, the data will be rejected. If there is no major error in the data, after the data is classified, the data will continue to be specified. In the directory, the data processing module 12 continues to process. Assuming that the general data is processed by the large amount of data matching module 12, a large number of sources of manual review interface 13 will continue to be launched, and the searched and classified reviewed data will be written into the point of interest search engine database 32, and additional The supplementary information waits until all the data is stored in the interest point search engine database 32, and the large data classification module 11 sends a message to the large data management module 15 to inform the review processing that the processing is completed, and the scheduled scheduling operation can be started. A large number of data source host database 3, will call the unique tag definition module 14. The unique tag definition module 14 will return a message to the large data source host database 3 to generate a specific tag field, and the large data source host database 3 sends the new data to the mass data management module 15. The plurality of data management modules 15 receive the data of the point of interest search engine database 32 and the plurality of data source host database 3, and then sort the data and send it to the enterprise application system integration data comparison module 16. Assuming that the general data is processed by the enterprise application system integration data matching module 16, a large amount of data extraction manual review interface 17 is continued, and the integrated and classified review data is written into the point of interest host database 33, and With additional supplementary information, a large number of data management modules 15 will send a message to the large data classification module 11, informing that the review operation is completed, and the scheduled scheduling operation can be started. If the client 2 starts to deliver the application file 21, the bulk data extraction module 18 starts to call the mass data management module 15, and the large number of data management modules 15 return the message to the large data extraction module 18, and a large number of data extraction modules. 18Send the newsletter to inform the customer 2 that the application case 21 has been processed. After the processing is completed, it will be sent directly to the billing module 19, and the price will be calculated based on the number of documents queried by the case number, and the customer will be notified of the automatic billing information. . After the message is sent, the successful message will be called and returned to the process billing module 19, and the application file 21 can be closed.

The system and method for managing a large amount of multivariate data screening provided by the present invention have the following advantages when compared with other conventional technologies:

1. The device number can be input through the enterprise application system to integrate the data comparison module to query the basic data, and the enterprise application system integrates the data comparison module to calculate the latitude and longitude data by using the address.

2. Combining the high-performance full-text search technology, it can quickly and correctly compare the data that is not available in the host data database in a short time. The efficiency is hundreds of times of manual query.

3. A large number of data extraction modules can automatically generate various types of reports for the query results, and automatically send a short message to inform the customer that the application has begun to process the application.

4. The billing module will send the results report to the customer and related personnel, and send a billing newsletter.

5. The point of interest search engine module is a function that is lacking in the general inquiry system. It can search related store information through the input keywords.

6. The invention can reduce the burden of a large number of manpower inquiries and reduce the lack of human factors

The detailed description of the preferred embodiments of the present invention is intended to be limited to the scope of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

To sum up, this case is not only innovative in terms of technical thinking, but also has many of the above-mentioned functions that are not in the traditional methods of the past. It has fully complied with the statutory invention patent requirements of novelty and progressiveness, and applied for it according to law. Approved this invention patent application, in order to invent invention, to the sense of virtue.

1‧‧‧Point of Interest Search Engine Module

11‧‧‧A large number of data classification modules

12‧‧‧Many data comparison modules

13‧‧‧Many sources of manual review interface

14‧‧‧Unique label definition module

15‧‧‧Many data management modules

16‧‧‧Enterprise Application System Integration Data Comparison Module

17‧‧‧Many data extraction manual review interface

18‧‧‧Many data extraction modules

19‧‧‧ Billing Module

2‧‧‧Customer

21‧‧‧ Application documents

3‧‧‧Many sources of host database

31‧‧‧ Index database

32‧‧‧Point of Interest Search Engine Database

33‧‧‧Point of Interest Host Database

FIG. 1 is a method for inputting a webpage seed by an operator of a system and a method for filtering and managing a large amount of multiple data according to the present invention; Figure 2 shows the way in which the operator of the large-scale multivariate data screening management system and its method manages the seed of the webpage; Figure 3 shows the system architecture diagram of the system and method of the multivariate data screening management; and Figure 4 shows the large amount of multivariate data. A data flow sequence diagram of the system and method for screening management.

1‧‧‧Point of Interest Search Engine Module

11‧‧‧A large number of data classification modules

12‧‧‧Many data comparison modules

13‧‧‧Many sources of manual review interface

14‧‧‧Unique label definition module

15‧‧‧Many data management modules

16‧‧‧Enterprise Application System Integration Data Comparison Module

17‧‧‧Many data extraction manual review interface

18‧‧‧Many data extraction modules

19‧‧‧ Billing Module

2‧‧‧Customer

21‧‧‧ Application documents

3‧‧‧Many sources of host database

31‧‧‧Catalogue Database

32‧‧‧Point of Interest Search Engine Database

33‧‧‧Point of Interest Host Database

Claims (11)

A large-scale multi-dimensional data screening management system, which combines database full-text search, data file format conversion, communication, and system state detection technology to achieve a fully automated query service system, which includes: a. a point of interest search engine module, It is used to perform scheduled scheduling operations, using a point of interest search engine system to capture store information, categorizing it into an index point of interest web database or an index point of interest database, and treating the index point of interest web database as a Review the store information, and use the index point of interest database as the store information to be classified; b. a large number of data classification module, which is used to accept the instructions transmitted by the point of interest search engine module, according to the captured store Data content or geographic information, if new information is introduced, it will actively classify the basic information of the store or the value-added information of the store, and make an integrated plan for the data, and have the information of the store that has been reviewed and classified, and transmitted back to Interested in the ability of the host database, and informed the large amount of data to have new data in the module to start the large amount of data comparison mode a group to process data; c. a large number of data matching modules, which are used to accept instructions transmitted by the mass data classification module, and store the store data crawled on the network into the index point of interest webpage database Comparing with the index point of interest database, searching for the store information or the differential store data not included in the index point of interest database, and sending the found data path to the large number of sources The manual review interface performs the audit action; d. The unique tag definition module is used to accept the data transmitted by the host source database of a large number of sources, and write the collated data to the host database of the point of interest to specify A specific label field is a unique label, and the format is ASCII encoding. The new specific label column is placed in the host database of the point of interest for management, and a large number of data management modules are notified; e. a large number of data management modules, It is used to accept the instructions passed by the unique tag definition module, read the data and classify it under the specified database, and plan the scheduled schedule to transfer the store data or geographic information to a point of interest search engine database. And the standard format acceptable to the system; f. Enterprise application system integration data comparison module, mainly to perform data comparison to generate operations, will accept a large number of management module to transmit instructions to generate corresponding data, the source of which is The point of interest searches the engine database and the point of interest host database, and sends the generated data to a large number of data extraction labor The core interface performs the auditing action, and notifies the data management module that the data processing operation has been completed; g. The mass data extraction module reads the standardized store data from the host database of the point of interest and notifies the charging module. The group, and send a newsletter to inform the customer that the application has been processed; and, h. the billing module, is to calculate the execution fee and inform the customer, and send the calculated result to the customer and related processing personnel; Among them, the system of multi-variable data screening management is used to simultaneously process multi-party data sources, analyze store information or geographic information transmitted from web pages, query a large amount of data in a short time, and use the enterprise application system to integrate data comparison. The module is compared with the original store information in the large number of management modules to obtain accurate results, and combined with the billing module, the customer is informed that the sent case is already being processed, and the query will be charged. The fee is automatically returned to the customer according to the E-MAIL address left by the customer after the standard format of the query result is output. A multi-variable data screening management method includes the following steps: a. A point of interest search engine module checks whether there is a new global resource addresser, which performs a scheduled job, and uses a point of interest search engine system to crawl The store information is classified into an index point of interest webpage database or an index point of interest database, and the index point of interest webpage database is used as the store information to be audited, and the index point of interest database is used as the store information to be classified; b. Download the new or changed store information to a designated directory, and start a large data classification module. According to the captured store information, the classified basic information or the store value added data is The data is used for integrated planning, and the reviewed and classified store information is returned to a point of interest host database; c. The large data classification module sends the file of the store data classification to a large amount Data comparison module review, which is performed by customers who want to crawl on the Internet to request specific industry store information and index interest. Point database to compare operations, and then find the store information or differential store data not included in the database of interest points, and send the found store data path to the large data source manual review interface for audit action, and use A large number of sources manually review the data back to a point of interest search engine database, and receive and return data to the point of interest host database; d. A unique identification definition module to confirm each piece of data plus a specific identification field Bit, a specific identification field specified by the system is a unique label, the format is ASCII encoding, the new specific identification field is stored in the host database of the point of interest, and a large data management module is started to process the received data. Information; e. The mass data management module receives and returns data to the point of interest search engine database, reads and classifies the data under the designated database, and plans a scheduled schedule to transmit the store data or geographic information to The point of interest search engine database is a standard format acceptable to the system and can be sent to an enterprise application system integration. Material comparison module review, using a large amount of data extraction manual review interface to transfer data back to the point of interest host database; f. A large number of data extraction module is to standardize the store data according to customer needs to extract the point of interest host The data of the database, and inform the customer that the inquiry case has been accepted and is already being processed; g. after the large number of data extraction modules are finished, the charging module is notified; h. After the billing module has calculated the cost, the customer and related personnel will be notified of the processing result and the cost related information, and the report will be sent to the customer. For example, the plurality of multivariate data screening management systems described in claim 1 wherein the feature of the point of interest search engine module comprises: a. generating an error message when the user operates incorrectly or the input parameter is invalid; b. A warning message is generated when the network is disconnected, or when the database fails to respond; c. The search scope includes the contents of the websites covered by the major websites and the seed list; d. provides a scheduled scheduling function, if the operation is performed In the process, other function modules are required to cooperate, and a processing message is generated, and an instruction is sent to the relevant function module; e. The content of the database (the field of the basic information of the shop) can be imported into the search engine, that is, the point of interest. The basic data must be resident in the search engine's index database, and scan the website list to see if there is relevant information about the relevant stores in the index database, and output relevant reports; f. According to the requirements of other functional modules, display Specify the message on the screen or update the specific screen display; g. Display the error message on the screen and print it on the screen according to the requirements of other function modules. On, and recorded in the system event log database;. H according to the needs of the operator, the query system event logs, call record Recorded, which website list data is currently being processed, and reports are generated; and i. According to the operator's needs, the global resource addresser that activates the direct operating system will be fully automated. For example, the plurality of multivariate data screening management systems described in claim 1 wherein the features of the mass data classification module include: a. if new data is introduced, the specified host address does not exist, or the specified directory is invalid. When an error message is generated on the screen; b. When receiving or updating the data processing failure or error, an error message is generated on the screen; c. When receiving or updating the data, check whether the received data is complete, if not complete, generate Error message; d. When receiving other function module messages, display their messages directly on the screen; e. When receiving the index database, the data will be classified according to its characteristics and imported into the point of interest search engine data. The relevant data forms of the library; f. receiving and updating a large number of sources of manual review interface, a large amount of correct data reviewed, will be stored in the data table of the search engine data of the interest point; and g. daily use timing Scheduling function, receiving and outputting data to the host database of the point of interest. For example, a large number of multi-dimensional data screening management systems described in claim 1 of the patent application, wherein the characteristics of the large-scale data matching module include: a. The main function is to perform online comparison and retrieval of the store data into the index point of interest database database and the index point of interest database for comparison operations, only need to compare the interest point web database and the point of interest database in the basic Information that is inconsistent, and then set the information as information to be reviewed; b. View the difference information or the store information not included in the database through the audit screen, and send it to the manual review interface of a large number of sources for audit reference; c. The review interface supports multiple users at the same time, avoiding the fact that a certain data is locked by the competition at the same time, resulting in the unrecognizable data. After entering the review interface, the user login status of the archive button is recorded, and the archive time at that time. And then write to the temporary table of interest points; and d. When the review screen displays the data difference, the difference data will be changed to red font display, and the editing function will be provided. If the data results of the two parties are different, the last is The edited point of interest webpage database information is mainly used, and the database is written back. For example, the large number of multivariate data screening management systems described in claim 1 wherein the unique tag definition module includes: a. will receive data from a large number of source host databases, and will organize the data. Write to the point of interest host database, and specify a specific label field as a unique label, the format is ASCII encoding, store this new specific label field into the point of interest host database management; b. During the writing process, A data format error, or any abnormal error, is written to the record. A large number of multivariate data screening management systems as described in item 1 of the patent application scope. The characteristics of the mass data management module include: a. managing all the points of interest data in the database according to the specific label field; b. receiving the data of the host data database of a large number of data sources, and determining according to the characteristics of the data Which type of form is stored in the host database of the point of interest; c. Receive and update a large amount of data to extract the manual review interface, and a large amount of correct data that has been reviewed will be stored in the data form of the host database of the point of interest; d. Generate an alarm message when the network is disconnected, or when the database fails to respond; and e. daily use the timing scheduling function to receive and output data to the database of the point of interest search engine. For example, a large number of multivariate data screening management systems described in claim 1 of the patent scope, wherein the characteristics of the enterprise application system integration data matching module include: a. sending the generated data path to a large amount of data extraction manual review interface for auditing action b. Through the enterprise application system integration check the data returned by the manual review interface of a large number of data sources, whether the store phone number exists, if it does not exist, delete it; otherwise, if it exists, skip to the next step; c. read the Interest point database form, and check whether the number exists in the form, if it does not exist, it is marked as new data to be reviewed; if it exists, it is compared with the content of the form, if it is different, it is marked as the difference to be reviewed Information; d. According to the query conditions, the information of the main points of the interest points and the files to be reviewed and reviewed Nuclear data. For example, the plurality of multivariate data screening management systems described in claim 1 wherein the features of the mass data extraction module include: a. extracting the demand data of the host database of the point of interest according to the application file submitted by the client, and The result of the query is output in a standard format; b. The automatic short message informs the customer that the application has been processed; c. If processed, the billing module is automatically notified. For example, the plurality of multivariate data screening management systems described in claim 1 wherein the billing module includes: a. pricing based on the number of data queried by the case number; b. if a large number of demand queries are encountered Data, the price is calculated separately; c. automatically bring out the customer's account number and transmit it to the accounting system; d. automatically send a fee to notify the customer. For example, a large number of multi-dimensional data screening management systems described in claim 1 of the patent scope, wherein each module uses TCP/IP or Socket as a communication medium.