CN113392116A - Distributed data processing system and method - Google Patents
Distributed data processing system and method Download PDFInfo
- Publication number
- CN113392116A CN113392116A CN202110940252.2A CN202110940252A CN113392116A CN 113392116 A CN113392116 A CN 113392116A CN 202110940252 A CN202110940252 A CN 202110940252A CN 113392116 A CN113392116 A CN 113392116A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- equipment
- data access
- access equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a distributed data processing system and a data processing method, and belongs to the technical field of data query. The system comprises a plurality of data access devices and data exchange devices, wherein the data exchange devices and the data access devices are in a star topology structure, and all departments are in communication connection with the data exchange devices through the corresponding data access devices. The method comprises the following steps: 1, equipment registration: the data access equipment sends a registration message to the data exchange equipment, wherein the registration message comprises a data access equipment identifier; the data exchange equipment judges whether the data access equipment is legal equipment, if so, the data access equipment information is stored, and if not, the access is forbidden; the data exchange device responds to the registration success message. And 2, data asset notification and data asset query, wherein the data asset query comprises a data asset query method I and a data asset query method II. The invention can ensure the safety of data and ensure the real-time performance of the data under the condition of larger data volume.
Description
Technical Field
The invention relates to a distributed data processing system and a distributed data processing method, and belongs to the technical field of data query.
Background
The current situation of informatization of governments and enterprises is that each government and enterprise generally designs and constructs information processing systems belonging to the application, flow and data of the organization and business itself, and the information processing systems which are independent, heterogeneous and cover the content of each business, the system design and construction periods are different, the business modes are different, and the informatization construction lacks effective overall planning and is repeatedly constructed; the system is lack of a uniform design standard, most systems are developed by different manufacturers on different platforms by using different languages, information interaction and sharing are difficult, and a large amount of information islands and flow islands exist. Therefore, data sharing is difficult to realize among all the current business systems, the cooperation among all the functional departments of the government/enterprise and the promotion of the working efficiency are seriously restricted, and the efficient data cooperative processing on the basis of not changing the original system becomes a problem which needs to be solved on an information-based construction road.
There are also some relevant scenario processing schemes in the market today:
a. using the synchronization tool of the database itself: for example, Oracle golden Gate can realize automatic synchronization between Oracle data to achieve the purpose of data sharing;
b. establishing a central database to realize data sharing and exchange: all department or business system data are uniformly stored in the central database, so that the data sharing effect is achieved.
The current scheme requires all department data to be stored on a central database node, and the scheme has the following problems: 1. data are stored in a centralized manner, so that potential safety hazards of the data are greatly increased; 2. the data is stored in multiple copies, so that the storage resources are wasted; 3. data can be shared and checked only after the business part is synchronized to the central database, and the real-time performance is difficult to guarantee under the condition of large data volume.
Disclosure of Invention
The invention aims to overcome the problems in the prior art and provide a distributed data processing method which can ensure the safety of data and ensure the real-time performance of the data under the condition of large data volume.
The distributed data processing system comprises a plurality of data access devices and data exchange devices, wherein the data exchange devices and the data access devices are in star topology structures, and all departments are in communication connection with the data exchange devices through the corresponding data access devices.
The distributed data processing method can conveniently access all distributed independent databases.
The invention discloses a distributed data processing method, which comprises the following steps:
s1: registering equipment;
s2: the method comprises the following steps of data asset notification and data asset query, wherein the data asset query comprises a data asset query method I and a data asset query method II.
Further, the device registration specifically includes the following steps:
s201, the data access equipment sends a registration message to the data exchange equipment, wherein the registration message comprises a data access equipment identifier;
s202, the data exchange equipment judges whether the data access equipment is legal equipment, if so, the data access equipment information is stored, and if not, the access is forbidden;
s203, the data exchange device responds to the registration success message.
Further, the data asset notification in step S2 specifically includes the following steps:
s301, the data access equipment at the edge reports the data asset information of the department to the data exchange equipment at the center, wherein the data asset information comprises a database identifier, a database table name and a table definition; the data asset information is generated by the data access equipment of the edge or pushed to the data access equipment of the edge by other equipment;
s302, the data exchange equipment inquires the equipment information stored in the step S202 and notifies the received data asset information to all other data access equipment;
and S303, storing data asset information after the other data access equipment receives the message, wherein the data asset information comprises a data access equipment identifier, a database table name and a database column name.
Further, the first data query method in step S2 specifically includes the following steps:
s401, after receiving a data query request of a user, the data access equipment judges whether the data is the data of a local department, and if the data is the data of the local department, the data access equipment enters S402; if the data of other departments is queried, the method goes to S403;
s402, directly returning the query result of the home department data to the user to complete the query;
s403, the data access device sends a query message to the data exchange device, wherein the message contains data asset information to be queried;
s404, the data exchange equipment sends the query message to the corresponding data access equipment according to the data asset information to be queried;
s405, after the corresponding data access equipment inquires the data, returning an inquiry result to the data exchange equipment;
s406, after receiving the query results of all the data access devices, the data exchange device processes the query results according to the query message;
s407, the data exchange device returns the processing result to the data access device initiating the query, and the data access device returns the result to the terminal user.
Further, the second data query method includes the following steps:
s501, after receiving a data query request of a user, the data access equipment judges whether the data access equipment queries data of a home department, and if the data access equipment queries data of the home department, the data access equipment enters S502; if the inquired data is data of other departments, the step S503 is entered;
s502, directly returning the query result of the home department data to the user to finish the query;
s503, the data access equipment initiating the inquiry directly sends inquiry information to the corresponding data access equipment according to the inquired information;
s504, after receiving the data inquired by the inquired data access equipment, returning an inquiry result to the data access equipment initiating the inquiry;
and S505, after receiving the query results of all the data access devices receiving the query, the data access device initiating the query processes the query results according to the query message and returns the query results to the terminal user.
Further, the criterion for determining whether the data access device is a legal device in step S202 includes the device identifier and the IP address information.
Further, the data asset information includes an access device identifier, a database list name, and a database list name.
Furthermore, the number of the corresponding data access devices is one or a plurality of data access devices.
Further, the data exchange device processes according to the query message in a manner including sorting or calculation.
The invention has the beneficial effects that: (1) the data island is effectively connected under the condition that an original data storage system is not changed, the original safety of data is guaranteed, and data collaborative comprehensive processing is efficiently carried out.
(2) Data does not need to be stored in an additional unified mode, a large number of storage resources are saved, and repeated infrastructure construction is avoided.
(3) The data does not need large-scale synchronization, and the real-time performance of data processing is higher.
Drawings
FIG. 1 is a block diagram of a distributed data processing system of the present invention;
FIG. 2 is a flow chart of device registration in the present invention;
FIG. 3 is a flow chart of a data asset advertisement of the present invention;
FIG. 4 is a flowchart of a first data query method according to the present invention;
FIG. 5 is a flowchart of a second data query method according to the present invention.
Detailed Description
The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic views illustrating only the basic structure of the present invention in a schematic manner, and thus show only the constitution related to the present invention.
As shown in fig. 1, a distributed data processing system according to the present invention includes a plurality of data access devices and data exchange devices, where the data exchange devices and the data access devices are in a star topology structure, and each department is in communication connection with the data exchange devices through corresponding data access devices.
The distributed data processing method can conveniently access all distributed independent databases.
The invention discloses a distributed data processing method, which comprises the following steps:
s1: registering equipment;
s2: the method comprises the following steps of data asset notification and data asset query, wherein the data asset query comprises a data asset query method I and a data asset query method II. The data asset information includes an access device identification, a database table name, and a database column name.
As shown in fig. 2, the device registration specifically includes the following steps:
s201, the data access equipment sends a registration message to the data exchange equipment, wherein the registration message comprises a data access equipment identifier;
s202, the data exchange equipment judges whether the data access equipment is legal equipment, if so, the data access equipment information is stored, and if not, the access is forbidden; in step S202, the criterion for determining whether the data access device is a legal device includes the device identifier and the IP address information.
S203, the data exchange device responds to the registration success message.
As shown in fig. 3, the data asset notification in step S2 specifically includes the following steps:
s301, the data access equipment at the edge reports the data asset information of the department to the data exchange equipment at the center, wherein the data asset information comprises a database identifier, a database table name and a table definition; the data asset information is generated by the data access equipment of the edge or pushed to the data access equipment of the edge by other equipment;
s302, the data exchange equipment inquires the equipment information stored in the step S202 and notifies the received data asset information to all other data access equipment;
s303, other data access equipment stores data asset information after receiving the message, wherein the data asset information comprises a data access equipment identifier, a database table name and a database list name;
s303', the data exchange device may also directly save the data asset information after querying the save information in step S202.
As shown in fig. 4, a first data query method in step S2 specifically includes the following steps:
s401, after receiving a data query request of a user, the data access equipment judges whether the data is the data of a local department, and if the data is the data of the local department, the data access equipment enters S402; if the data of other departments is queried, the method goes to S403;
s402, directly returning the query result of the home department data to the user to complete the query;
s403, the data access device sends a query message to the data exchange device, wherein the message contains data asset information to be queried;
s404, the data exchange equipment sends the query message to the corresponding data access equipment according to the data asset information to be queried; one or a plurality of corresponding data access devices;
s405, after the corresponding data access equipment inquires the data, returning an inquiry result to the data exchange equipment;
s406, after receiving the query results of all the data access devices, the data exchange device processes the query results according to the query message;
s407, the data exchange device returns the processing result to the data access device initiating the query, and the data access device returns the result to the terminal user. The way in which the data switching device processes according to the query message includes sorting or calculation.
As shown in fig. 5, the second data query method includes the following steps:
s501, after receiving a data query request of a user, the data access equipment judges whether the data access equipment queries data of a home department, and if the data access equipment queries data of the home department, the data access equipment enters S502; if the inquired data is data of other departments, the step S503 is entered; the data access equipment A receives a data query request of a user;
s502, directly returning the query result of the home department data to the user to finish the query;
s503, the data access equipment initiating the inquiry directly sends inquiry information to the corresponding data access equipment according to the inquired information; one or a plurality of corresponding data access devices; accessing the corresponding data access equipment B to the access equipment A according to the inquired information;
s504, after receiving the data inquired by the inquired data access equipment, returning an inquiry result to the data access equipment initiating the inquiry; the data access equipment B returns a query result after querying the data;
and S505, after receiving the query results of all the data access devices receiving the query, the data access device initiating the query processes the query results according to the query message and returns the query results to the terminal user. The data exchange equipment A processes according to the query message, and the processing mode of the data exchange equipment A according to the query message comprises sequencing or calculation.
In light of the foregoing description of the preferred embodiment of the present invention, many modifications and variations will be apparent to those skilled in the art without departing from the spirit and scope of the invention. The technical scope of the present invention is not limited to the content of the specification, and must be determined according to the scope of the claims.
Claims (10)
1. A distributed data processing system, characterized by: the system comprises a plurality of data access devices and data exchange devices, wherein the data exchange devices and the data access devices are in a star topology structure, and all departments are in communication connection with the data exchange devices through the corresponding data access devices.
2. A distributed data processing method, comprising the steps of:
s1: registering equipment;
s2: the method comprises the following steps of data asset notification and data asset query, wherein the data asset query comprises a data asset query method I and a data asset query method II.
3. The distributed data processing method according to claim 2, wherein the device registration specifically includes the steps of:
s201, the data access equipment sends a registration message to the data exchange equipment, wherein the registration message comprises a data access equipment identifier;
s202, the data exchange equipment judges whether the data access equipment is legal equipment, if so, the data access equipment information is stored, and if not, the access is forbidden;
s203, the data exchange device responds to the registration success message.
4. The distributed data processing method of claim 2, wherein: the data asset notification in step S2 specifically includes the following steps:
s301, the data access equipment at the edge reports the data asset information of the department to the data exchange equipment at the center, wherein the data asset information comprises a database identifier, a database table name and a table definition; the data asset information is generated by the data access equipment of the edge or pushed to the data access equipment of the edge by other equipment;
s302, the data exchange equipment inquires the equipment information stored in the step S202 and notifies the received data asset information to all other data access equipment;
and S303, storing data asset information after the other data access equipment receives the message, wherein the data asset information comprises a data access equipment identifier, a database table name and a database column name.
5. The distributed data processing method according to claim 2, wherein the first data query method in step S2 specifically includes the following steps:
s401, after receiving a data query request of a user, the data access equipment judges whether the data is the data of a local department, and if the data is the data of the local department, the data access equipment enters S402; if the data of other departments is queried, the method goes to S403;
s402, directly returning the query result of the home department data to the user to complete the query;
s403, the data access device sends a query message to the data exchange device, wherein the message contains data asset information to be queried;
s404, the data exchange equipment sends the query message to the corresponding data access equipment according to the data asset information to be queried;
s405, after the corresponding data access equipment inquires the data, returning an inquiry result to the data exchange equipment;
s406, after receiving the query results of all the data access devices, the data exchange device processes the query results according to the query message;
s407, the data exchange device returns the processing result to the data access device initiating the query, and the data access device returns the result to the terminal user.
6. The distributed data processing method of claim 2, wherein: the second data query method comprises the following steps:
s501, after receiving a data query request of a user, the data access equipment judges whether the data access equipment queries data of a home department, and if the data access equipment queries data of the home department, the data access equipment enters S502; if the inquired data is data of other departments, the step S503 is entered;
s502, directly returning the query result of the home department data to the user to finish the query;
s503, the data access equipment initiating the inquiry directly sends inquiry information to the corresponding data access equipment according to the inquired information;
s504, after receiving the data inquired by the inquired data access equipment, returning an inquiry result to the data access equipment initiating the inquiry;
and S505, after receiving the query results of all the data access devices receiving the query, the data access device initiating the query processes the query results according to the query message and returns the query results to the terminal user.
7. A distributed data processing method according to claim 3, characterized in that: in step S202, the criterion for determining whether the data access device is a legal device includes the device identifier and the IP address information.
8. The distributed data processing method of claim 4, wherein: the data asset information includes an access device identification, a database table name, and a database column name.
9. The distributed data processing method according to claim 5 or 6, characterized in that: the number of the corresponding data access devices is one or a plurality of data access devices.
10. The distributed data processing method of claim 5, wherein: the way in which the data switching device processes according to the query message includes sorting or calculation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940252.2A CN113392116A (en) | 2021-08-17 | 2021-08-17 | Distributed data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940252.2A CN113392116A (en) | 2021-08-17 | 2021-08-17 | Distributed data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113392116A true CN113392116A (en) | 2021-09-14 |
Family
ID=77622872
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110940252.2A Pending CN113392116A (en) | 2021-08-17 | 2021-08-17 | Distributed data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392116A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104142957A (en) * | 2013-05-10 | 2014-11-12 | 上海联影医疗科技有限公司 | Method and system for regional medical treatment-orientated data sharing |
| CN105681474A (en) * | 2016-03-31 | 2016-06-15 | 浪潮通信信息系统有限公司 | System architecture for supporting upper layer applications based on enterprise-level big data platform |
| CN110059109A (en) * | 2019-03-12 | 2019-07-26 | 平安普惠企业管理有限公司 | Device, method and the storage medium of data query |
| US20200334605A1 (en) * | 2018-11-21 | 2020-10-22 | Beijing Yutian Technology Co. Ltd | Emergency resource sharing and exchange system |
-
2021
- 2021-08-17 CN CN202110940252.2A patent/CN113392116A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104142957A (en) * | 2013-05-10 | 2014-11-12 | 上海联影医疗科技有限公司 | Method and system for regional medical treatment-orientated data sharing |
| CN105681474A (en) * | 2016-03-31 | 2016-06-15 | 浪潮通信信息系统有限公司 | System architecture for supporting upper layer applications based on enterprise-level big data platform |
| US20200334605A1 (en) * | 2018-11-21 | 2020-10-22 | Beijing Yutian Technology Co. Ltd | Emergency resource sharing and exchange system |
| CN110059109A (en) * | 2019-03-12 | 2019-07-26 | 平安普惠企业管理有限公司 | Device, method and the storage medium of data query |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1815469B (en) | Database connection resource management method | |
| CN110209719B (en) | System and method for unified access of multiple databases based on micro-service architecture | |
| CN105930446B (en) | A kind of telecom client label generating method based on Hadoop distributed computing technology | |
| CN102201010A (en) | Distributed database system without sharing structure and realizing method thereof | |
| CN1437355A (en) | Management method of multicast users in mobile network | |
| CN109327535B (en) | Database access method, system, middleware equipment and medium | |
| CN101771723A (en) | Data synchronization method | |
| CN111597160A (en) | Distributed database system, distributed data processing method and device | |
| CN112367354B (en) | Cloud edge resource map intelligent scheduling system and scheduling method thereof | |
| WO2019233061A1 (en) | Routing resource control method and system for layer 3 switch | |
| CN102447734A (en) | Cloud service method for taxation cloud computing network billing IM (Instant Messaging) online customer system | |
| CN103336782A (en) | Relational distributed database system | |
| EP3937502A1 (en) | Method, apparatus and device for pushing video stream, and storage medium | |
| CN100396002C (en) | System and method for identifying authority using relative inquire | |
| CN110311896A (en) | Airborne platform service register center system | |
| CN102737061B (en) | Distributed ticket query management system and method | |
| CN101778131A (en) | Data synchronization system | |
| CN100484014C (en) | Distributed cluster service management system and service management method in intelligent network | |
| CN101789963A (en) | Data synchronization system | |
| CN103646210A (en) | Method and device for calling system function | |
| CN104580421A (en) | Server and user group information pushing method adopting same | |
| CN108681588A (en) | A kind of interface accesses real-time statistical method and system | |
| CN113392116A (en) | Distributed data processing system and method | |
| CN101924841B (en) | Building and control method of a kind of call center and data center's distributed frame | |
| CN114978890A (en) | Port mapping system and mapping method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210914 |