CN112860674A - Data sharing method and system - Google Patents
Data sharing method and system Download PDFInfo
- Publication number
- CN112860674A CN112860674A CN202110128331.3A CN202110128331A CN112860674A CN 112860674 A CN112860674 A CN 112860674A CN 202110128331 A CN202110128331 A CN 202110128331A CN 112860674 A CN112860674 A CN 112860674A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- processing module
- standard format
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6209—Protecting access to data via a platform, e.g. using keys or access control rules to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Abstract
The invention relates to a data sharing method and a system, wherein the method comprises the following steps: the data import module receives original data; a data cleaning unit of the data processing module performs data filtering processing on the original data to obtain filtered data; a data conversion unit of the data processing module performs data format conversion processing according to the filtered data to obtain standard format data; a data cutting unit of the data processing module determines a field to which the data belongs according to the standard format data, and cuts the standard format data according to the field to obtain cut data; the cropping data has a location ID; the location ID comprises a database ID, a data table ID and a field ID; the data inserting unit of the data processing module inserts the cutting data into the corresponding field according to the position ID; and the data query module extracts data according to the received query request and feeds back an extraction result.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a data sharing method and system.
Background
With the rapid development and application of information technology, data has become a fundamental resource, and big data increasingly has important influence on global production, circulation, distribution, consumption activities, economic operation mechanisms and social life styles.
The traditional information system construction is of a chimney type, each field, industry, mechanism and department can store data respectively, but a cross data sharing management mechanism is lacked, and most enterprise system construction has a data island problem.
In the big data era, the realization of digital transformation and the realization of data interconnection and intercommunication are of great significance. At the present stage, through the construction of a related big data platform, data collection and data management are already completed, however, the value of big data is far from being fully mined, some companies or organization business systems relate to core data and data security requirements, the sharing and use of data meet a great safety technical bottleneck, data change and data operation under the premise of data security assurance are not realized, users with data analysis capability often do not have exact data with high value, and units and organizations with mass data do not have data analysis mining capability.
Disclosure of Invention
The invention aims to provide a data sharing method and a data sharing system aiming at the defects of the prior art, and solves the problems that each field, industry, mechanism and department at the present stage lacks a data sharing management mechanism, a data island exists, and no safety guarantee exists when a user uses data.
To achieve the above object, in a first aspect, the present invention provides a data sharing method, including:
the data import module receives original data;
a data cleaning unit of the data processing module performs data filtering processing on the original data to obtain filtered data;
a data conversion unit of the data processing module performs data format conversion processing according to the filtered data to obtain standard format data;
a data cutting unit of the data processing module determines a field to which the standard format data belongs according to the standard format data, and cuts the standard format data according to the field to obtain cut data; the cropping data has a location ID; the location ID comprises a database ID, a data table ID and a field ID;
a data inserting unit of the data processing module inserts the cutting data into a corresponding field according to the position ID;
and the data query module extracts data according to the received query request and feeds back an extraction result.
Preferably, after the data clipping unit of the data processing module determines the field to which the standard format data belongs according to the standard format data, and clips the standard format data according to the field to obtain clipped data, the data sharing method further includes:
the data clipping unit sends the clipping data to a data buffering unit of the data processing module;
and the data buffer unit queues the cutting data and sends the cutting data to the data insertion unit according to a queuing sequence.
Preferably, the original data has a user ID; after the data import module receives the original data, the data sharing method further includes:
a data verification unit of the data processing module determines a corresponding private key according to the user ID, decrypts the original data according to the private key, and performs comparison processing according to the decrypted data and preset plaintext data;
and when the comparison result is inconsistent, the data verification unit generates first alarm information.
Preferably, the raw data comprises time data; after the data import module receives the original data, the data sharing method further includes:
a data verification unit of the data processing module calculates time difference data according to the time data and the current time data;
comparing the time difference data with a preset time difference;
and when the time difference data is larger than the preset time difference, generating second alarm information.
In a second aspect, the present invention provides a data sharing system based on the above data sharing method, the data sharing system including:
the data import module is used for receiving original data;
the data processing module is used for filtering, converting, cutting and inserting the original processing;
the data cleaning unit is used for carrying out data filtering processing on the original data to obtain filtered data;
the data conversion unit is used for carrying out data format conversion processing according to the filtering data to obtain standard format data;
the data cutting unit is used for determining the field to which the standard format data belongs according to the standard format data and cutting the standard format data according to the field to obtain cut data;
the data inserting unit is used for inserting the cutting data into a corresponding field according to the position ID;
and the data query module is used for extracting data according to the received query request and feeding back an extraction result.
Preferably, the data processing module further comprises a data buffering unit;
and the data buffer unit is used for queuing the cutting data and sending the cutting data to the data insertion unit according to a queuing sequence.
Preferably, the original data has a user ID; the data processing module also comprises a data checking unit;
the data verification unit is used for determining a corresponding private key according to the user ID, decrypting the original data according to the private key and comparing the decrypted data with preset plaintext data;
and when the comparison results are inconsistent, the data verification unit generates alarm information.
The data sharing method and the data sharing system provided by the embodiment of the invention establish a data sharing management mechanism, realize sharing of the same system or program components under a multi-user environment, ensure the isolation of data among users and ensure the safety of the data in the using process.
Drawings
Fig. 1 is a schematic structural diagram of a data sharing system according to an embodiment of the present invention;
fig. 2 is a flowchart of a data sharing method according to an embodiment of the present invention.
Detailed Description
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
The data sharing method and the data sharing system provided by the invention establish a data sharing management mechanism, realize sharing of the same system or program components under a multi-user environment, not only can ensure the isolation of data among users, but also can ensure the safety of the data in the using process.
Fig. 1 is a schematic structural diagram of a data sharing system according to an embodiment of the present invention, and the data sharing system of the present invention is described in detail below with reference to fig. 1.
As shown in fig. 1, the data sharing system includes: the system comprises a data import module 1, a data processing module 2 and a data query module 3. The data processing module 2 includes a data checking unit 21, a data cleaning unit 22, a data converting unit 23, a data clipping unit 24, a data buffering unit 25 and a data inserting unit 26.
The data importing module 1 is used for receiving original data, and the original data has a user ID. And the data processing module 2 is used for carrying out filtering, conversion, clipping and insertion processing on the original processing. And the data verification unit 21 is configured to determine a corresponding private key according to the user ID of the original data, decrypt the original data according to the private key, and compare the decrypted data with preset plaintext data. When the comparison results are inconsistent, the data verification unit 21 generates alarm information. And the data cleaning unit 22 is used for performing data filtering processing on the original data to obtain filtered data. And the data conversion unit 23 is configured to perform data format conversion processing according to the filtered data to obtain standard format data. And the data clipping unit 24 is configured to determine the field to which the standard format data belongs according to the standard format data, and perform clipping processing on the standard format data according to the field to obtain clipped data. The data buffer unit 25 is configured to queue the clip data and send the clip data to the data insertion unit 26 according to the queue sequence. And a data inserting unit 26 for inserting the cropping data into the corresponding field according to the position ID. And the data query module 3 is used for extracting data according to the received query request and feeding back an extraction result.
Fig. 2 is a flowchart of a data sharing method according to an embodiment of the present invention, and as shown in fig. 2, the data sharing method according to the present invention includes the following steps.
101, a data import module receives original data;
specifically, the original data has a user ID and time data. The user ID may be understood as an account number for logging into the data sharing system.
In an optimal scheme, different levels of user IDs are set for reducing misoperation of users and controlling the users, and permission limitation is performed on the users at different levels to different degrees.
In the embodiment of the present invention, the original data may be a data file imported according to an import operation of a user, or may be data synchronized with an external database in real time. To reduce the latency of the data sharing system of the present invention relative to the external database, each operation on the external database is synchronized to the data sharing system in the form of a piece of synchronization data. And the data sharing system updates corresponding data according to the synchronous data.
102, a data cleaning unit of a data processing module performs data filtering processing on original data to obtain filtered data;
specifically, the data filtering process may include one or a combination of several of a comparison operation, a range operation, a null value matching, a character matching, and a logic matching. The data filtering processing can remove redundant data, and the pressure of subsequent data processing is reduced.
In a specific example, step S1 receives the original data, and determines the corresponding data receiving user according to the subscription information stored therein. Step S2, extracting data features of the original data, and determining whether the data features satisfy the data filtering conditions of the corresponding data receiving users according to the data filtering conditions of each data receiving user. The method specifically comprises the following steps: and matching the data content type identification of the original data with a data content type identification set corresponding to a data filtering condition pre-configured by each data receiving user, wherein if the data content type identification of the original data is matched with the data content type identification set, the data filtering condition is met. Wherein the extracted data characteristics of the original data at least comprise data content type identification. In addition, before the original data is filtered, a dynamic link library is configured for each data receiving user in advance, and the dynamic link library at least comprises a data filtering function. And step S3, if the data filtering condition is met, filtering the corresponding original data to obtain filtered data, and forwarding the filtered data to the corresponding data receiving user. The method specifically comprises the following steps: and if the original data meet the data filtering condition, calling a data filtering function in a dynamic link library corresponding to the corresponding data receiving user to filter the original data.
103, the data conversion unit of the data processing module performs data format conversion processing according to the filtered data to obtain standard format data;
specifically, the data conversion unit identifies a data format of the filtered data and converts the data format of the filtered data into a standard format. The standard format is understood to be a data format that can be recognized by a database.
104, determining the field of the data processing module according to the standard format data by a data cutting unit of the data processing module, and cutting the standard format data according to the field to obtain cut data;
specifically, the clip data has a position ID. The location ID includes a database ID, a data table ID, and a field ID. The standard format data has a header, the data insertion unit determines the field to which the standard format data belongs according to the header, and the standard format data is cut according to the field, namely key data in the standard format data is intercepted.
In a specific scheme, in order to make the processing of data sharing data more stable, the resource cost is saved. And after the data clipping unit of the data processing module determines the field to which the data belongs according to the standard format data and clips the standard format data according to the field to obtain the clipped data, the data clipping unit sends the clipped data to the data buffering unit of the data processing module. The data buffer unit converts synchronous push into asynchronous indirect push by using a message queue, and the middle part of the data buffer unit receives instantaneous flow peak at one end through a queue and smoothly pushes out the message at the other end. The method realizes the queuing processing of the cutting data and sends the cutting data to the data insertion unit according to the queuing sequence
In another specific scheme, the data buffer unit is an answering unit, and comprises a question bank generating module, a question bank pushing module and a question picture generating module. The question bank generating module is used for generating questions and answers and preventing the questions from being answered by the machine. And the question bank pushing module is used for pushing the questions in advance before answering the questions. The question requested by the user each time is ensured to be unique, and the purpose is to prevent answering cheating. And the theme picture generation module is used for generating the theme into a picture format and adding some interference factors in the picture. Also to prevent the machine from answering questions directly.
It is also noted here that since the network is crowded when answering questions, we should push pictures of the questions ahead of time and warm up the pictures, otherwise when the user really requests the questions, the pictures may be loaded slowly, thereby affecting the answering experience.
That is, when the answer submitted by the user is compared with the answer corresponding to the topic, if the answer passes the next step, the next step is continued, otherwise, the answer fails.
105, inserting the cutting data into a corresponding field by a data inserting unit of the data processing module according to the position ID;
specifically, the data insertion unit inserts each cropping data into a corresponding field for storage. The distributed storage mode has the characteristics of high reliability, high performance, high expansibility and the like, ensures the data security and the flexibility of a data architecture from the surface of a data surface, and also supports the rapid transverse expansion of a cluster, thereby meeting the requirements of storage and query performance.
And 106, the data query module extracts data according to the received query request and feeds back an extraction result.
Specifically, the user may input a query request to the data sharing system to extract the desired information.
In a preferred embodiment, the stored data is protected against manipulation in order to ensure data security. Each user is assigned a user ID with each user ID having a set of provisioned public and private keys. And the user encrypts the query request through the public key in the process of querying the data.
After the data import module receives the original data, the data verification unit of the data processing module determines a corresponding private key according to the user ID, decrypts the original data according to the private key, and performs comparison processing according to the decrypted data and preset plaintext data. And when the comparison result is inconsistent, the query request is falsified, and the data verification unit generates first alarm information and rejects the query request.
In order to secure the security of the key, a new key is generated when a predetermined event such as a new user being added to the group or a user being removed from the group occurs or when a certain time has elapsed. Furthermore, the key may also be generated such that the lifetime of the session key is at least one user login period. That is, the key may be generated when the user of the first person logs in, and the key may be discarded when all users log out.
In another preferred embodiment, in order to prevent repeated calling of information, crawling of data, and the like, after the data import module receives the original data, the data verification unit of the data processing module calculates time difference data according to the time data and the current time data. And comparing the time difference data with a preset time difference. And when the time difference data is larger than the preset time difference, generating second alarm information.
The data sharing method and the data sharing system establish a data sharing management mechanism, realize sharing of the same system or program components under a multi-user environment, and not only can ensure data isolation among users, but also can ensure safety in the data use process.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied in hardware, a software module executed by a processor, or a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (7)
1. A data sharing method, characterized in that the data sharing method comprises:
the data import module receives original data;
a data cleaning unit of the data processing module performs data filtering processing on the original data to obtain filtered data;
a data conversion unit of the data processing module performs data format conversion processing according to the filtered data to obtain standard format data;
a data cutting unit of the data processing module determines a field to which the standard format data belongs according to the standard format data, and cuts the standard format data according to the field to obtain cut data; the cropping data has a location ID; the location ID comprises a database ID, a data table ID and a field ID;
a data inserting unit of the data processing module inserts the cutting data into a corresponding field according to the position ID;
and the data query module extracts data according to the received query request and feeds back an extraction result.
2. The data sharing method according to claim 1, wherein after the data clipping unit of the data processing module determines the field to which the standard format data belongs according to the standard format data, and clips the standard format data according to the field to obtain clipped data, the data sharing method further comprises:
the data clipping unit sends the clipping data to a data buffering unit of the data processing module;
and the data buffer unit queues the cutting data and sends the cutting data to the data insertion unit according to a queuing sequence.
3. The data sharing method according to claim 1, wherein the original data has a user ID; after the data import module receives the original data, the data sharing method further includes:
a data verification unit of the data processing module determines a corresponding private key according to the user ID, decrypts the original data according to the private key, and performs comparison processing according to the decrypted data and preset plaintext data;
and when the comparison result is inconsistent, the data verification unit generates first alarm information.
4. The data sharing method according to claim 1, wherein the original data includes time data; after the data import module receives the original data, the data sharing method further includes:
a data verification unit of the data processing module calculates time difference data according to the time data and the current time data;
comparing the time difference data with a preset time difference;
and when the time difference data is larger than the preset time difference, generating second alarm information.
5. The data sharing system according to the data sharing method of claim 1, wherein the sharing system includes:
the data import module is used for receiving original data;
the data processing module is used for filtering, converting, cutting and inserting the original processing;
the data cleaning unit is used for carrying out data filtering processing on the original data to obtain filtered data;
the data conversion unit is used for carrying out data format conversion processing according to the filtering data to obtain standard format data;
the data cutting unit is used for determining the field to which the standard format data belongs according to the standard format data and cutting the standard format data according to the field to obtain cut data;
the data inserting unit is used for inserting the cutting data into a corresponding field according to the position ID;
and the data query module is used for extracting data according to the received query request and feeding back an extraction result.
6. The data sharing system according to claim 1, wherein the data processing module further comprises a data buffering unit;
and the data buffer unit is used for queuing the cutting data and sending the cutting data to the data insertion unit according to a queuing sequence.
7. The data sharing system according to claim 1, wherein the original data has a user ID; the data processing module also comprises a data checking unit;
the data verification unit is used for determining a corresponding private key according to the user ID, decrypting the original data according to the private key and comparing the decrypted data with preset plaintext data;
and when the comparison results are inconsistent, the data verification unit generates alarm information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110128331.3A CN112860674A (en) | 2021-01-29 | 2021-01-29 | Data sharing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110128331.3A CN112860674A (en) | 2021-01-29 | 2021-01-29 | Data sharing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112860674A true CN112860674A (en) | 2021-05-28 |
Family
ID=75987019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110128331.3A Pending CN112860674A (en) | 2021-01-29 | 2021-01-29 | Data sharing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860674A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704811A (en) * | 2021-07-16 | 2021-11-26 | 杭州医康慧联科技股份有限公司 | Data value management method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177133A (en) * | 2019-12-24 | 2020-05-19 | 集奥聚合(北京)人工智能科技有限公司 | Processing insertion method for multivariate data |
-
2021
- 2021-01-29 CN CN202110128331.3A patent/CN112860674A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177133A (en) * | 2019-12-24 | 2020-05-19 | 集奥聚合(北京)人工智能科技有限公司 | Processing insertion method for multivariate data |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704811A (en) * | 2021-07-16 | 2021-11-26 | 杭州医康慧联科技股份有限公司 | Data value management method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111556120B (en) | Data processing method and device based on block chain, storage medium and equipment | |
| US8813243B2 (en) | Reducing a size of a security-related data object stored on a token | |
| US6823391B1 (en) | Routing client requests to back-end servers | |
| US10263778B1 (en) | Synchronizable hardware security module | |
| CN107295069A (en) | Data back up method, device, storage medium and server | |
| US20220391893A1 (en) | Secure electronic messaging guaranteeing integrity and non-repudation | |
| CN108764902B (en) | Method, node and blockchain system for storing data | |
| AU2020203467A1 (en) | Email verification method | |
| CN102281141A (en) | Document permission management method, apparatus and system | |
| CN112653556A (en) | TOKEN-based micro-service security authentication method, device and storage medium | |
| CN101175089A (en) | Data transmission method between server and client terminal based on HTTP protocol and .NET structure | |
| CN113568568B (en) | Hardware encryption method, system and device based on distributed storage | |
| CN112860674A (en) | Data sharing method and system | |
| JP6534478B1 (en) | File sharing system and method | |
| CN114519197A (en) | Data storage architecture and method based on block chain and cloud service | |
| US8826026B2 (en) | Systems and methods for tracking electronic files in computer networks using electronic signatures | |
| KR102191111B1 (en) | System and method of providing anonymity message service using block chain | |
| CN111327680A (en) | Authentication data synchronization method, device, system, computer equipment and storage medium | |
| CN110737705A (en) | Data acquisition system for forwarding behaviors | |
| CN112751841A (en) | Encryption communication method and system based on block chain technology | |
| CN110098931A (en) | Data transmission method based on trusted " government and enterprises' connection connects " platform | |
| CN205080542U (en) | Data storage device | |
| CN106971119A (en) | The key data in database safe read-write authentication method of trusted identity | |
| CN112187467B (en) | Method and system for realizing encryption algorithm integrating multiple signatures | |
| WO2024066974A1 (en) | Blockchain-based data processing method, device, and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |