CN115186003A - Data management method and system based on cloud computing - Google Patents
Data management method and system based on cloud computing Download PDFInfo
- Publication number
- CN115186003A CN115186003A CN202211113349.7A CN202211113349A CN115186003A CN 115186003 A CN115186003 A CN 115186003A CN 202211113349 A CN202211113349 A CN 202211113349A CN 115186003 A CN115186003 A CN 115186003A
- Authority
- CN
- China
- Prior art keywords
- data
- management
- node
- unit
- searching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013523 data management Methods 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000007726 management method Methods 0.000 claims abstract description 162
- 238000012790 confirmation Methods 0.000 claims description 27
- 230000005540 biological transmission Effects 0.000 description 19
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data management method and a system thereof based on cloud computing, wherein the data management method based on the cloud computing specifically comprises the following steps: acquiring a data management request; searching management data according to the data management request; classifying the management data in response to the completion of searching the management data; determining a management node according to the classified management data; and managing the classified management data according to the management node. According to the method and the device, different management data can be classified, stable data and active data are distinguished, the data needing to be managed are finely divided, different management nodes can be selected for different types of data to be managed, and the management nodes are determined more finely.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data management method and system based on cloud computing.
Background
Cloud computing is a novel computing mode following distributed computing, peer-to-peer computing and grid computing, and distributes tasks on a data center composed of a large number of computing resources and storage resources, so that a user can access the data center through a PC (personal computer) and mobile terminal equipment to obtain corresponding computing resources, storage resources and the like as required. However, in the data management process, data is often directly managed without considering the type of the data, and the simplified management manner makes the data too rough in the subsequent data management process, and thus the data cannot be finely managed.
Therefore, how to provide a management method for refining data is a problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The application provides a data management method based on cloud computing, which specifically comprises the following steps: acquiring a data management request; searching management data according to the data management request; in response to the completion of the search of the management data, classifying the management data; determining a management node according to the classified management data; and managing the classified management data according to the management node.
As above, the data management request includes the address of the data requested to be managed and the specific operation requested to be managed.
As above, according to the data management request, searching for management data specifically includes the following sub-steps: confirming whether the data management request directly hits the data; and if the data management request does not directly hit the data, searching the management data from the storage area.
As described above, the method further includes backing up data in the storage area in advance, and searching for management data from the storage area if the data is not directly hit.
As above, the classifying the management data in response to completing the search of the management data specifically includes the following sub-steps: confirming the frequency of the management data being searched; the classification of the management data is performed according to the frequency of being searched.
As above, the classification of the management data is, specifically, to divide the management data into the stable data and the positive data.
As described above, the determining the management node according to the classified management data specifically includes the following sub-steps: determining a reference node; and determining a management node for managing the active data according to the reference node.
A data management system based on cloud computing specifically comprises: the system comprises an acquisition unit, a search unit, a classification unit, a node confirmation unit and a management unit; an acquisition unit configured to acquire a data management request;
the searching unit is used for searching the management data according to the data management request; the classification unit is used for responding to the completion of the search of the management data and classifying the management data; a node confirmation unit, configured to determine a management node according to the classified management data; and the management unit is used for managing the classified management data according to the management node.
As above, the lookup unit specifically includes the following sub-modules: a hit confirmation module and a search module; the hit confirmation module is used for confirming whether the management request directly hits the data; if the data is hit, the management data is directly obtained, and if the data is not hit, the search module searches the management data from the storage area.
As above, the classification unit specifically includes the following sub-modules: a frequency confirmation module and a classification module; the frequency confirmation module is used for confirming the searched frequency of the management data; and the classification module is used for classifying the management data according to the searched frequency.
The application has the following beneficial effects:
(1) The method and the device can classify different management data, distinguish stable data from active data, finely divide the data to be managed, and provide a better basis for subsequent fine management data.
(2) According to the method and the device, different management nodes are selected according to the stable data and the active data, and the different types of data can be managed according to the different management nodes, so that the determination of the management nodes is more refined, and the management nodes can be managed more carefully in the subsequent management of the different types of data.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a flowchart of a cloud computing-based data management method provided according to an embodiment of the present application;
fig. 2 is an internal structural diagram of a cloud computing-based data management system provided according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Example one
As shown in fig. 1, a data management method based on cloud computing provided by the present application specifically includes the following steps:
step S110: acquiring a data management request;
the client sends a data management request to the system, wherein the data management request comprises an address of data requiring management and specific operation requiring management.
Specific operations for requesting management include operations on data such as requesting storage of data, requesting deletion of data, and the like.
Step S120: and searching management data according to the data management request.
Specifically, data in the system is distributed in a data area, wherein since the data management request carries an address of data requesting management, data is searched according to the carried address, so as to determine data needing to be managed, and the data needing to be managed is defined as management data.
In order to ensure the rigor of searching the management data, the present embodiment performs a more rigorous confirmation on the search of the data, where the searching the management data specifically includes the following sub-steps:
step S1201: it is confirmed whether the data management request directly hits the data.
And determining whether the data management request directly hits the data, specifically whether the data can be directly searched according to the address information carried in the request.
If the data can be directly found when the address carried by the management request is used for searching the data in the data area of the system, the data is directly hit, and the data is directly obtained.
If the data is not found in the data area of the system according to the address carried by the management request, the data cannot be directly hit, and step S1202 is executed.
The method in the prior art can be referred to for searching data according to the address of the data, and details are not described herein.
Step S1202: the management data is searched from the storage area.
The system performs data backup in advance, specifically, data distributed in the data area are backed up according to addresses, and mapping relationships exist between the data stored in the storage area and the addresses in the data area, where the mapping relationships are specifically such that the data in the storage area can be found according to the addresses in the data area.
When the data cannot be found in the data area, the data is directly read in the storage area.
Step S130: and in response to finishing the search of the management data, classifying the management data.
After the designated time, the designated time is divided into a plurality of time periods, the number of times of management of the management data is confirmed in the plurality of time periods, and the management data is classified according to the number of times of management.
The step S130 specifically includes the following sub-steps:
step S1301: the frequency with which the management data is looked up is confirmed.
The frequency H at which the management data is searched is specifically represented as:
wherein
Which indicates the number of times the management data is looked up at the time t during a time period,
indicating the number of seeks received by the management data within a time period,
indicating whether the management data receives a new search request within a time period (specifically, a request for managing the data is received within a time period, so that the data is searched according to the management request),
indicating that a new lookup request has been received,
indicating that a new lookup request has not been received,
the number of search cycles experienced by the management data is represented, and C represents the total number of times the management data is searched within the total time period.
Step S1302: the classification of the management data is performed according to the frequency of being searched.
Specifically, when the frequency H with which the management data is searched is greater than a specified threshold, the management is classified as positive data, and when the frequency H with which the management data is searched is less than the specified threshold, the management is classified as stable data.
The active data indicates that the number of times the management data is accessed is slightly larger, and the client has a higher requirement for the stability of the data, so that the system provides more attention to the data, which is embodied in the management of the data in the subsequent system.
The stabilizing data indicates that the management data needs to be managed less frequently and that the data is more stable.
The embodiment classifies the management data by confirming the frequency of the management data being searched, and can provide a good basis for subsequent management optimization.
Step S140: and determining the management node according to the classified management data.
In the embodiment, appropriate nodes are selected for the active data and the stable data to manage the active data and the stable data, and particularly, the operations of storing, transmitting, reading and the like of the data by the client can be satisfied through the appropriate nodes, and different levels of management are performed according to different nodes, so that how to select the appropriate nodes is very important when data management is performed.
The step S140 specifically includes the following sub-steps:
step S1401: a reference node is determined.
The selection standard of the reference node is that a node with relatively high disk reading and writing speed and relatively short network transmission delay is selected as the reference node. Specifically, the disk read-write speeds and the network transmission delays of the plurality of nodes may be sorted in a descending order, so that the node with the fastest disk read-write speed and the shortest network transmission delay may be selected from the plurality of nodes as the reference node.
Step S1402: and determining a management node for managing the active data according to the reference node.
Since the active data is data for which the customer demands more, it is required to be managed more strictly, and therefore, it is required to select a node for which the condition is more strict.
A more stringent node is one that selects a node that manages aggressive data by comparing the capabilities of data transmissions between nodes. Wherein based on reference nodes
With other nodes
The nodes are selected according to the transmission capacity, so that the selected nodes are expected to have faster and better transmission capacity, and therefore, the data storage, transmission and reading can be completed with faster and better reaction capacity.
In particular, transmission capacity thereof
The concrete expression is as follows:
wherein
Representing reference nodes
With other arbitrary nodes
The transmission bandwidth of (a) is set,
representing other arbitrary nodes
The number of times of abnormity occurring in a specified historical time period refers to the number of times of response failure or non-response occurring in the historical use process of the node,
representing other arbitrary nodes
The number of times used within a specified historical period of time, use may refer to the node being specified by the system to perform an operation such as data storage,
representing reference nodes
With other arbitrary nodes
The transmission distance of (2).
Transmission capability when computed with reference node
When the value is larger than a specified threshold value, the arbitrary node is illustrated
Has better transmission capability, the node is connected with the node
And if the node is used as the node for managing the active data, the transmission capability of the node is poor, and the node cannot be used as the node for managing the active data.
By the method, the nodes with the designated number are selected as the nodes for managing the active data, and the method has better reaction capability when the data is stored, read and the like.
Step S1403: and determining a management node for managing the stable data according to the reference node.
The node for managing stable data is still selected according to the transmission capability, but because the stable data is data with low requirement on the client, in the calculation transmission capability, the selection method of the node for managing active data can be referred to according to the benchmark node
With other nodes
To directly determine the transmission bandwidth ofTransmission capability.
In which the transmission capacity is
The concrete expression is as follows:
wherein
Representing reference nodes
With other arbitrary nodes
The transmission bandwidth of (a) is set,
representing other arbitrary nodes
The number of times an anomaly occurred within a specified historical period of time,
representing other arbitrary nodes
The number of times used within a specified historical period of time.
The steps S1402 and S1403 may be executed simultaneously without being sequentially executed.
The different management data are classified, the stable data and the active data are distinguished, the management nodes are further selected according to the stable data and the active data, different nodes can be selected for different types of data, in the selection process of the management nodes, the management nodes are directly assigned in the past, the nodes for managing the stable data and the active data are confirmed by considering the transmission capacity among the nodes, the determination of the management nodes is more refined, and more perfect management can be provided in the subsequent data management process.
Step S150: and managing the classified management data according to the management node.
According to the specific operation of request management carried in the acquired request management request, the management node respectively manages the stable data and the active data which need to be managed. Specifically, stable data is managed by a node that manages stable data, and active data is managed by a node that manages active data.
Example two
As shown in fig. 2, the data management system based on cloud computing provided by the present application specifically includes an obtaining unit 210, a searching unit 220, a classifying unit 230, a node confirming unit 240, and a managing unit 250.
Wherein the obtaining unit 210 is used for obtaining the data management request.
The searching unit 220 is connected to the obtaining unit 210, and is configured to search for the management data according to the data management request.
The searching unit 220 specifically includes the following sub-modules: a hit confirmation module and a search module;
the hit confirmation module is used for confirming whether the management request directly hits the data;
if the data is hit, the management data is directly obtained, and if the data is not hit, the search module searches the management data from the storage area.
The classification unit 230 is connected to the search unit 220, and is configured to perform classification of the management data in response to completion of search of the management data.
The classification unit specifically comprises the following sub-modules: frequency confirmation module and classification module.
And the frequency confirmation module is used for confirming the searched frequency of the management data.
The classification module is connected with the frequency confirmation module and is used for classifying the management data according to the searched frequency.
The node confirmation unit 240 is connected to the classification unit 230, and is configured to determine a management node according to the classified management data.
The node confirmation unit specifically comprises the following sub-modules: the system comprises a reference node confirmation module, a first management node confirmation module and a second management node confirmation module.
The reference node confirmation module is used for determining a reference node;
the first management node confirmation module is connected with the reference node and used for confirming a management node for managing the positive data according to the reference node.
And the second management node confirmation module is connected with the reference node and used for determining a management node for managing the stable data according to the reference node.
The management unit 250 is connected to the node confirmation unit 240, and is configured to manage the classified management data according to the management node.
The application has the following beneficial effects:
(1) According to the method and the device, different management data can be classified, stable data and active data are distinguished, data needing to be managed are finely divided, and a good basis is provided for subsequent fine management data.
(1) According to the method and the device, different management nodes are selected according to the stable data and the positive data, and different types of data can be managed according to different management nodes, so that the management nodes are determined more finely, and the management can be managed more carefully in the process of managing different types of data subsequently.
Although the present application has been described with reference to examples, which are intended to be illustrative only and not to be limiting of the application, changes, additions and/or deletions may be made to the embodiments without departing from the scope of the application.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data management method based on cloud computing is characterized by comprising the following steps:
acquiring a data management request;
searching management data according to the data management request;
classifying the management data in response to the completion of searching the management data;
determining a management node according to the classified management data;
and managing the classified management data according to the management node.
2. The cloud-computing-based data management method of claim 1, wherein the data management request includes an address of data requested for management and an operation requested for management.
3. The cloud-computing-based data management method according to claim 1, wherein searching for management data according to a data management request comprises the following sub-steps:
confirming whether the data management request directly hits the data;
and if the data management request does not directly hit the data, searching the management data from the storage area.
4. The cloud-computing-based data management method according to claim 3, further comprising backing up data in the storage area in advance, and searching for management data from the storage area if the data is not directly hit.
5. The cloud-computing-based data management method according to claim 1, wherein the classification of the management data is performed in response to completion of the search of the management data, and specifically includes the following substeps:
confirming the frequency of the management data being searched;
the classification of the management data is performed according to the frequency of being searched.
6. The cloud-computing-based data management method of claim 5, wherein the management data is classified into stable data and active data.
7. The cloud-computing-based data management method of claim 6, wherein the determining of the management node according to the classified management data specifically comprises the sub-steps of:
determining a reference node;
and determining a management node for managing the active data according to the reference node.
8. A data management system based on cloud computing is characterized by specifically comprising: the system comprises an acquisition unit, a search unit, a classification unit, a node confirmation unit and a management unit;
an acquisition unit configured to acquire a data management request;
the searching unit is used for searching the management data according to the data management request;
the classification unit is used for responding to the completion of the search of the management data and classifying the management data;
a node confirmation unit, configured to determine a management node according to the classified management data;
and the management unit is used for managing the classified management data according to the management node.
9. The cloud-computing-based data management system of claim 8, wherein the lookup unit specifically comprises the following sub-modules: a hit confirmation module and a search module;
the hit confirmation module is used for confirming whether the management request directly hits the data;
if the data is not hit, the searching module searches the management data from the storage area.
10. The cloud-computing-based data management system of claim 8, wherein the classification unit specifically comprises the following sub-modules: a frequency confirmation module and a classification module;
the frequency confirmation module is used for confirming the searched frequency of the management data;
and the classification module is used for classifying the management data according to the searched frequency.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211113349.7A CN115186003A (en) | 2022-09-14 | 2022-09-14 | Data management method and system based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211113349.7A CN115186003A (en) | 2022-09-14 | 2022-09-14 | Data management method and system based on cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115186003A true CN115186003A (en) | 2022-10-14 |
Family
ID=83524207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211113349.7A Pending CN115186003A (en) | 2022-09-14 | 2022-09-14 | Data management method and system based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115186003A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104142953A (en) * | 2013-05-10 | 2014-11-12 | 中国电信股份有限公司 | Metadata dynamic management method and system |
| CN109388658A (en) * | 2018-09-21 | 2019-02-26 | 麒麟合盛网络技术股份有限公司 | A kind of data determination method and device |
| CN113900597A (en) * | 2021-11-30 | 2022-01-07 | 深圳市安信达存储技术有限公司 | Data storage method, system, equipment and storage medium |
| CN114398397A (en) * | 2021-12-02 | 2022-04-26 | 阿里巴巴（中国）有限公司 | Data processing method, device, storage medium and system |
-
2022
- 2022-09-14 CN CN202211113349.7A patent/CN115186003A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104142953A (en) * | 2013-05-10 | 2014-11-12 | 中国电信股份有限公司 | Metadata dynamic management method and system |
| CN109388658A (en) * | 2018-09-21 | 2019-02-26 | 麒麟合盛网络技术股份有限公司 | A kind of data determination method and device |
| CN113900597A (en) * | 2021-11-30 | 2022-01-07 | 深圳市安信达存储技术有限公司 | Data storage method, system, equipment and storage medium |
| CN114398397A (en) * | 2021-12-02 | 2022-04-26 | 阿里巴巴（中国）有限公司 | Data processing method, device, storage medium and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220335034A1 (en) | Multi-master architectures for distributed databases | |
| US7007144B2 (en) | Method, apparatus, and computer readable medium for managing back-up | |
| KR20120102664A (en) | Allocating storage memory based on future use estimates | |
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| CN111831618A (en) | Data writing method, data reading method, device, equipment and storage medium | |
| US11366821B2 (en) | Epsilon-closure for frequent pattern analysis | |
| US7895247B2 (en) | Tracking space usage in a database | |
| CN110381136B (en) | Data reading method, terminal, server and storage medium | |
| CN116244085A (en) | Kubernetes cluster container group scheduling method, device and medium | |
| CN115408546A (en) | Time sequence data management method, device, equipment and storage medium | |
| WO2021056243A1 (en) | Method and device for managing data in partition table, management node, and storage medium | |
| CN109165207B (en) | Drinking water mass data storage management method and system based on Hadoop | |
| CN115186003A (en) | Data management method and system based on cloud computing | |
| CN112711564B (en) | Merging processing method and related equipment | |
| TWI665568B (en) | Method and device for clustering data stream | |
| US10067678B1 (en) | Probabilistic eviction of partial aggregation results from constrained results storage | |
| CN115905049A (en) | Hot and cold memory graded migration method based on Itanium memory | |
| CN110837428A (en) | Storage device management method and device | |
| CN116029891A (en) | Graph data storage, access and processing methods, training methods, equipment and media | |
| US11797919B2 (en) | Document-based distributed inventory system with rebalancing | |
| US20210117233A1 (en) | Service band management system | |
| CN111858150A (en) | System and method for management of large-scale data backup | |
| CN115510292B (en) | Distributed storage system tree search management method, device, equipment and medium | |
| US20240086727A1 (en) | Automatically Building Efficient Machine Learning Model Training Environments | |
| US20240119091A1 (en) | Storing graph data in a key-value store |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221014 |
|
| RJ01 | Rejection of invention patent application after publication |