CN117312351A - Database processing system based on multi-source heterogeneous data - Google Patents
Database processing system based on multi-source heterogeneous data Download PDFInfo
- Publication number
- CN117312351A CN117312351A CN202311617142.8A CN202311617142A CN117312351A CN 117312351 A CN117312351 A CN 117312351A CN 202311617142 A CN202311617142 A CN 202311617142A CN 117312351 A CN117312351 A CN 117312351A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- information
- sub
- processing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 claims abstract description 14
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000005192 partition Methods 0.000 claims abstract description 5
- 230000008676 import Effects 0.000 claims description 6
- 238000009825 accumulation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention is applicable to the field of data processing systems, and provides a database processing system based on multi-source heterogeneous data, which comprises a basic database and an automatic database processing system connected with the basic database; the database classification module carries out partition processing on the basic database and forms a sub-database; the data instruction receiving unit is used for receiving operation instruction information, converting the operation instruction information into data information and transmitting the data information to the sub-database processing module; the sub-database processing module receives the instruction information transmitted by the data instruction receiving unit and processes the data in the sub-data, so that the sub-database processing module can process the data information in the basic database, and the processing process comprises data importing/adding, data deleting and data modifying; and the application also sets an auxiliary database and an auxiliary guiding module, and provides selectable items to assist operators in adding data information into the database.
Description
Technical Field
The invention relates to the field of data processing systems, in particular to a database processing system based on multi-source heterogeneous data.
Background
The existing databases are various in types, but are different in size, and are used for storing useful information required by a subsequent processing system, such as keyword information, data operation parameters and the like, and the processing modes of data information in most databases are single, such as some simpler database processing modes which can be automatically executed, such as data query (information screening based on the content stored in the database), data extraction and the like.
However, the creation of databases from scratch often requires an accumulation of certain processing frequencies, such as databases of some search classes (the creation of which is dependent on the frequency of the various keyword searches), and thus the accumulation of frequencies of keyword searches is not as labor intensive for database operators.
In summary, it is clear that the prior art has inconvenience and defects in practical use, so that improvement is needed.
Disclosure of Invention
In view of the above-mentioned drawbacks, an object of the present invention is to provide a database processing system based on multi-source heterogeneous data, which can process data information in a base database through a sub-database processing module, wherein the processing process includes importing/adding data, deleting data, and modifying data; and the application also sets an auxiliary database and an auxiliary guiding module, and provides selectable items to assist operators in adding data information into the database.
In order to achieve the above object, the present invention provides a database processing system based on multi-source heterogeneous data, including a base database and a database automatic processing system connected with the base database and used for maintaining and processing data in the database; the database automatic processing system includes: the database classification module is used for carrying out partition processing on the basic database and forming a sub-database, butting a processing port with the corresponding sub-database in a manner of establishing a sub-account, and operating the data in the sub-database; the data instruction receiving unit receives the operation instruction information, converts the operation instruction information into data information and transmits the data information to the sub-database processing module; the sub-database processing module is used for receiving the instruction information transmitted by the data instruction receiving unit and processing the data in the sub-database, and comprises a full-text indexing module and a data acquisition module.
According to the database processing system based on the multi-source heterogeneous data, the processing instructions of the sub-database processing modules comprise: importing/adding data, importing/adding needed data information from the external environment into a basic database; deleting data, and deleting partial/all data information in the database; modifying data, namely modifying partial/all data information in the database; after each operation is finished, prompting yes/no finishing operation, and if not prompting error reasons.
According to the database processing system based on the multi-source heterogeneous data, the data sources of the importing/adding data operation comprise historical processing data and real-time data subjected to preliminary data processing; in the data importing/adding retrieval process, the full text index module preferentially searches information fields processed by historical data, if the searching result is no, real-time data searching is started, and keyword information is imported through the data acquisition module.
The database processing system based on the multi-source heterogeneous data according to the invention further comprises: the auxiliary database is connected with the basic database and is used for temporarily storing information and deleting the internal information after the account is switched; the auxiliary guiding module is used for picking information fields related to the data through the Internet after the operator imports/adds the data, and forming selectable options; when the operator selects one of the selectable entries, the remaining related entries are stored into the auxiliary database.
According to the database processing system based on the multi-source heterogeneous data, the selectable items are selected by an operator in the form of a list.
According to the database processing system based on the multi-source heterogeneous data, the sequence of the list is arranged according to the size of a base C, wherein the base C is calculated according to the following formula: c=b/B 0 X epsilon; wherein B is the number of days from the occurrence of the event to the current time, B 0 Epsilon is the number of clicks of the keyword information event, which is the time base.
The database processing system based on the multi-source heterogeneous data comprises the following components 0 Obtained with the length of the duration of the keyword information event, classified as follows: the duration is 0-3 months, B 0 3; the duration is 3-6 months, B 0 Is 2; the duration is more than 6 months, B 0 1.
The invention provides a database processing system based on multi-source heterogeneous data, which comprises a basic database and a database automatic processing system which is connected with the basic database and is used for maintaining and processing data in the database; the database automatic processing system includes: the database classification module is used for carrying out partition processing on the basic database and forming a sub-database, butting a processing port with the corresponding sub-database in a manner of establishing a sub-account, and operating the data in the sub-database; the data instruction receiving unit receives the operation instruction information, converts the operation instruction information into data information and transmits the data information to the sub-database processing module; the sub-database processing module is used for receiving the instruction information transmitted by the data instruction receiving unit and processing the data in the sub-database, and comprises a full-text indexing module and a data acquisition module. The invention can process the data information in the basic database through the sub-database processing module, and the processing process comprises importing/adding data, deleting data and modifying data; and the application also sets an auxiliary database and an auxiliary guiding module, and provides selectable items to assist operators in adding data information into the database.
Drawings
FIG. 1 is a diagram of the processing of a base database by a sub-database processing module;
FIG. 2 is a data source flow chart of the database import/add data operation of the present invention.
Detailed Description
For the purpose of making apparent the objects, technical solutions and advantages of the present invention, the present invention will be further described in detail with reference to the accompanying drawings and examples, it being understood that the specific examples described herein are for illustration only and are not intended to limit the present invention.
Referring to fig. 1, the invention provides a database processing system based on multi-source heterogeneous data, which comprises a basic database and a database automatic processing system connected with the basic database, wherein the database automatic processing system is used for maintaining and processing data in the database; the management platform of the basic database is suitable for a technical model selection framework of a big data platform, is based on Hbase, hdfs and other storage systems in a Hadoop ecological environment, and has the characteristics of easy expansion and easy maintenance.
The database automatic processing system includes:
the database classification module is used for carrying out partition processing on the basic database and forming a sub-database, butting a processing port with the corresponding sub-database in a manner of establishing a sub-account, and operating data in the sub-database (the butted sub-database), wherein when an account is logged in, referring to fig. 1, the account information is uploaded to a login platform, if the login is successful, the user character identification is used for entering the corresponding sub-database, and if the normal login cannot be completed, the login information is prompted to be wrong, and the login is carried out again;
the data instruction receiving unit receives operation instruction information (sent by an operator), converts the operation instruction information into data information and transmits the data information to the sub-database processing module; the front end development framework of the data instruction receiving unit uses technologies such as Vue.JS, vue Router, axios, ant-design-Vue and the like; the design purpose is to truly change the interaction mode of the user and the document.
The sub-database processing module receives the instruction information transmitted by the data instruction receiving unit and processes the data in the sub-database, and includes a full text indexing module and a data acquisition module (specific functions will be described in detail below). And the processing instructions of the sub-database processing module include:
importing/adding data, importing/adding needed data information from the external environment into a basic database;
deleting data, and deleting partial/all data information in the database;
modifying data, namely modifying partial/all data information in the database;
after each item of operation is finished, prompting yes/no operation, and prompting the completion of the operation if the operation is finished; otherwise, the error cause is indicated (shown in fig. 1). Further, the data sources of the import/add data operation include historical processing data and preliminary data processing real-time data (shown in fig. 2).
In the data importing/adding process, the full text index module preferentially searches information fields processed by historical data, if the searching result is no, real-time data searching is started, and keyword information is imported through the data acquisition module.
Preferably, the invention further comprises an auxiliary database and an auxiliary guiding module; the auxiliary database is connected with the basic database and is used for temporarily storing information and deleting internal information after the account is switched, and the working mode of the auxiliary database is similar to that of a memory RAM processing unit in a computer.
The auxiliary guiding module is used for picking information fields related to the data through the Internet after the operator imports/adds the data, and forming selectable options; when an operator selects one of the selectable entries, the rest related entries are stored in the auxiliary database, and by temporarily storing the related entries in the auxiliary database, when the operator wants to re-reference the content of the related entries (related entries corresponding to the keywords), the related entries can be directly called from the auxiliary database, so that the indexing process can be greatly reduced. Preferably, the selectable items are selected by an operator in a list form and sequentially displayed in a top-down order, wherein the order of the list is arranged according to the size of a base number C (a preliminary data processing process), and the base number C is calculated according to the following formula:
C=B/B 0 x epsilon; wherein B is the number of days from the occurrence of the event to the current time, B 0 For the time base, ε is the number of keyword information event clicks (introduced from the Internet of things). The larger the value of the base number C, the closer the corresponding index information is to the upper position of the list, and the higher the recommended degree of the index information is further indicated. And B is 0 Obtained with the length of the duration of the keyword information event, classified as follows:
the duration is 0-3 months, B 0 3, this setting indicates that the data information is continuously low in heat, so the above processing formula is combined to reduce the recommended degree;
the duration is 3-6 months, B 0 2, this setting indicates that the heat of the data information is relatively high, so that the recommended degree is relatively high in combination with the above processing formula;
the duration is more than 6 months, B 0 For 1, this setting indicates that the heat of the data information is continuously increasing, so that the recommended degree is higher in combination with the above processing formula.
Of course, the present invention is capable of other various embodiments and its several details are capable of modification and variation in light of the present invention, as will be apparent to those skilled in the art, without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (7)
1. A database processing system based on multi-source heterogeneous data is characterized by comprising a basic database and a database automatic processing system which is connected with the basic database and is used for maintaining and processing data in the database;
the database automatic processing system includes:
the database classification module is used for carrying out partition processing on the basic database and forming a sub-database, butting a processing port with the corresponding sub-database in a manner of establishing a sub-account, and operating the data in the sub-database;
the data instruction receiving unit receives the operation instruction information, converts the operation instruction information into data information and transmits the data information to the sub-database processing module;
the sub-database processing module is used for receiving the instruction information transmitted by the data instruction receiving unit and processing the data in the sub-database, and comprises a full-text indexing module and a data acquisition module.
2. The multi-source heterogeneous data based database processing system of claim 1, wherein the processing instructions of the sub-database processing module comprise:
importing/adding data, importing/adding needed data information from the external environment into a basic database;
deleting data, and deleting partial/all data information in the database;
modifying data, namely modifying partial/all data information in the database;
after each operation is finished, prompting yes/no finishing operation, and if not prompting error reasons.
3. The multi-source heterogeneous data based database processing system of claim 2 wherein the data sources of the import/add data operations include historical process data and preliminary data processed real-time data;
in the data importing/adding retrieval process, the full text index module preferentially searches information fields processed by historical data, if the searching result is no, real-time data searching is started, and keyword information is imported through the data acquisition module.
4. The multi-source heterogeneous data based database processing system of claim 3, further comprising:
the auxiliary database is connected with the basic database and is used for temporarily storing information and deleting the internal information after the account is switched;
the auxiliary guiding module is used for picking information fields related to the data through the Internet after the operator imports/adds the data, and forming selectable options; when the operator selects one of the selectable entries, the remaining related entries are stored into the auxiliary database.
5. The multi-source heterogeneous data based database processing system of claim 4 wherein the selectable items are presented to the operator in the form of a list.
6. The multi-source heterogeneous data based database processing system of claim 5 wherein the order of the list is arranged according to the size of radix C, wherein radix C is calculated according to the following formula:
C=B/B 0 x epsilon; wherein B is the number of days from the occurrence of the event to the current time, B 0 Epsilon is the number of clicks of the keyword information event, which is the time base.
7. The multi-source heterogeneous data based database processing system of claim 6 wherein B 0 Obtained with the length of the duration of the keyword information event, classified as follows:
the duration is 0-3 months, B 0 3;
the duration is 3-6 months, B 0 Is 2;
the duration is more than 6 months, B 0 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311617142.8A CN117312351B (en) | 2023-11-30 | 2023-11-30 | Database processing system based on multi-source heterogeneous data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311617142.8A CN117312351B (en) | 2023-11-30 | 2023-11-30 | Database processing system based on multi-source heterogeneous data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117312351A true CN117312351A (en) | 2023-12-29 |
| CN117312351B CN117312351B (en) | 2024-02-13 |
Family
ID=89285227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311617142.8A Active CN117312351B (en) | 2023-11-30 | 2023-11-30 | Database processing system based on multi-source heterogeneous data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117312351B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20030065663A (en) * | 2002-01-30 | 2003-08-09 | 삼성전자주식회사 | system for management of history data in EMS and method thereof |
| US20080288522A1 (en) * | 2007-01-26 | 2008-11-20 | Herbert Dennis Hunt | Creating and storing a data field alteration datum using an analytic platform |
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN106599253A (en) * | 2016-12-21 | 2017-04-26 | 济南浪潮高新科技投资发展有限公司 | Method for achieving distributed computation by using NoSQL database |
| CN109033387A (en) * | 2018-07-26 | 2018-12-18 | 广州大学 | A kind of Internet of Things search system, method and storage medium merging multi-source data |
| CN115587084A (en) * | 2022-08-31 | 2023-01-10 | 中国人民解放军陆军边海防学院 | Comprehensive management system and method for geographic information data |
| CN116028653A (en) * | 2023-03-29 | 2023-04-28 | 鹏城实验室 | Method and system for constructing map by visually configuring multi-source heterogeneous data |
-
2023
- 2023-11-30 CN CN202311617142.8A patent/CN117312351B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20030065663A (en) * | 2002-01-30 | 2003-08-09 | 삼성전자주식회사 | system for management of history data in EMS and method thereof |
| US20080288522A1 (en) * | 2007-01-26 | 2008-11-20 | Herbert Dennis Hunt | Creating and storing a data field alteration datum using an analytic platform |
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN106599253A (en) * | 2016-12-21 | 2017-04-26 | 济南浪潮高新科技投资发展有限公司 | Method for achieving distributed computation by using NoSQL database |
| CN109033387A (en) * | 2018-07-26 | 2018-12-18 | 广州大学 | A kind of Internet of Things search system, method and storage medium merging multi-source data |
| CN115587084A (en) * | 2022-08-31 | 2023-01-10 | 中国人民解放军陆军边海防学院 | Comprehensive management system and method for geographic information data |
| CN116028653A (en) * | 2023-03-29 | 2023-04-28 | 鹏城实验室 | Method and system for constructing map by visually configuring multi-source heterogeneous data |
Non-Patent Citations (2)
| Title |
|---|
| 蒋春曦;谢庆胜;李毅军;李罡风;: "安徽省防震减灾基础地理数据库设计与实现", 华北地震科学, no. 03 * |
| 蔡维: "基于云平台的核安全异构数据库系统设计与实现", 工程科技Ⅱ辑;信息科技 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117312351B (en) | 2024-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240901B (en) | Performance analysis method, performance analysis device, storage medium, and electronic apparatus | |
| US7673234B2 (en) | Knowledge management using text classification | |
| US8706748B2 (en) | Methods for enhancing digital search query techniques based on task-oriented user activity | |
| US20090157653A1 (en) | Methods for enhancing digital search results based on task-oriented user activity | |
| US20220365957A1 (en) | Log parsing method and device, server and storage medium | |
| US20090157729A1 (en) | Methods for generating search engine index enhanced with task-related metadata | |
| CN103164449A (en) | Search result showing method and search result showing device | |
| CN102027471A (en) | Improved search engine | |
| WO2007085187A1 (en) | Method of data retrieval, method of generating index files and search engine | |
| CN111522905A (en) | Document searching method and device based on database | |
| CN102393858A (en) | Meta search engine system based on client side real time aggregation | |
| CN113190687B (en) | Knowledge graph determining method and device, computer equipment and storage medium | |
| CN111309868B (en) | Knowledge graph construction and retrieval method and device | |
| CN106777343A (en) | increment distributed index system and method | |
| CN103942268A (en) | Method and device for combining search and application and application interface | |
| CN101082936A (en) | Data enquiring system and method | |
| CN101661469A (en) | System and method for indexing and retrieving keywords of academic documents | |
| CN102063454A (en) | Method and equipment combining search and application | |
| TWI284821B (en) | Patent family analysis system and method | |
| US10235455B2 (en) | Semantic search system interface and method | |
| CN117312351B (en) | Database processing system based on multi-source heterogeneous data | |
| CN111984673A (en) | Fuzzy retrieval method and device for tree structure of power grid electric energy metering system | |
| CN112395365B (en) | Knowledge graph batch offline query solution | |
| CN111079036B (en) | Field type searching method | |
| CN110321351A (en) | A kind of vendor name method for normalizing based on fuzzy matching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |