CN112579611B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112579611B CN112579611B CN202011600572.5A CN202011600572A CN112579611B CN 112579611 B CN112579611 B CN 112579611B CN 202011600572 A CN202011600572 A CN 202011600572A CN 112579611 B CN112579611 B CN 112579611B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- stored
- storing
- acquired
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000000034 method Methods 0.000 claims description 28
- 238000013500 data storage Methods 0.000 claims description 19
- 238000013075 data extraction Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Abstract
The application provides a data processing device and a data processing method, wherein the data processing method comprises the steps of extracting the same data from acquired initial data to serve as first data, and taking other data except the first data as second data; establishing a first database based on the first data, and storing the first data to the first database; establishing a second database storing a data snapshot of the first data based on the first data stored in the first database; a third database associated with the second database is established based on the second data, and the second data is stored to the third database. Specifically, the data processing method stores the same data in the acquired initial data into the public first database to form the public first data, and the public first data is shared by the third databases of all branches in a data snapshot mode.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, a computing device, and a computer readable storage medium.
Background
The SVN (subversion) library file storage system of the prior art often involves the storage of a very large number of loose files when performing the storage of project files (e.g., game development projects). Multiple branches may exist for bulk files developed based on the same platform. And these loose files all require memory space. This results in a large library of files, which takes up a lot of storage space.
Disclosure of Invention
In view of the foregoing, embodiments of the present application provide a data processing method and apparatus, a computing device, and a computer readable storage medium, so as to solve the technical drawbacks in the prior art.
According to a first aspect of an embodiment of the present application, there is provided a data processing method, including:
extracting the same data from the acquired initial data as first data, and taking other data except the first data as second data;
establishing a first database based on the first data, and storing the first data to the first database;
establishing a second database storing a data snapshot of the first data based on the first data stored in the first database;
a third database associated with the second database is established based on the second data, and the second data is stored to the third database.
Optionally, the establishing a third database associated with the second database based on the second data, and storing the second data to the third database, includes:
at least one third database associated with the second data is established based on the second data, and a data snapshot of the second data and the first data is stored to the third database.
Optionally, after the storing the second data in the third database, the method further includes:
receiving a data storage request, wherein the data storage request carries data to be stored;
determining whether the data to be stored matches the first data,
if yes, the process is finished,
and if not, storing the data to be stored as the second data to the third database.
Optionally, after the storing the second data in the third database, the method further includes:
receiving a data acquisition request, wherein the data acquisition request carries data to be acquired;
judging whether the data to be acquired is matched with the first data,
if so, acquiring first data matched with the data to be acquired from the first database as the data to be acquired,
If not, second data matched with the data to be acquired is acquired from the third database and is used as the data to be acquired.
Optionally, the determining whether the data to be stored matches the first data includes:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
Optionally, the storing the data to be stored as the second data in the third database includes:
and under the condition that the third database reaches the preset storage condition, creating at least one second database for storing the data snapshot of the first data, and at least one third database associated with the created at least one second database, and storing the data to be stored and the data snapshot of the first data into the created at least one third database.
Optionally, after the storing the second data in the third database, the method further includes:
and receiving a deleting instruction aiming at the third database, and deleting the third database and the second data stored in the third database based on the deleting instruction.
Optionally, after the storing the second data in the third database, the method further includes:
and receiving an update instruction aiming at the first database, and updating the first data stored in the first database based on update data carried in the update instruction.
Optionally, after updating the first data stored in the first database based on the update data carried in the update instruction, the method further includes:
and establishing a second database for storing the data snapshot of the updated first data based on the updated first data in the first database.
According to a second aspect of embodiments of the present application, there is provided a data processing apparatus comprising:
a data extraction module configured to extract the same data from the acquired initial data as first data, and other data than the first data as second data;
a first database creation module configured to create a first database based on the first data and store the first data to the first database;
a second database creation module configured to create a second database storing a data snapshot of the first data based on the first data stored in the first database;
A third database creation module configured to create a third database associated with the second database based on the second data and store the second data to the third database.
Optionally, the third database creation module is further configured to:
at least one third database associated with the second data is established based on the second data, and a data snapshot of the second data and the first data is stored to the third database.
Optionally, the apparatus further comprises:
a data storage module configured to:
receiving a data storage request, wherein the data storage request carries data to be stored;
determining whether the data to be stored matches the first data,
if yes, the process is finished,
and if not, storing the data to be stored as the second data to the third database.
Optionally, the apparatus further comprises:
a data acquisition module configured to:
receiving a data acquisition request, wherein the data acquisition request carries data to be acquired;
judging whether the data to be acquired is matched with the first data,
if so, acquiring first data matched with the data to be acquired from the first database as the data to be acquired,
If not, second data matched with the data to be acquired is acquired from the third database and is used as the data to be acquired.
Optionally, the data storage module is further configured to:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
Optionally, the data acquisition module is further configured to:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
Optionally, the third database creation module is further configured to:
and under the condition that the third database reaches the preset storage condition, creating at least one second database for storing the data snapshot of the first data, and at least one third database associated with the created at least one second database, and storing the data to be stored and the data snapshot of the first data into the created at least one third database.
Optionally, the apparatus further comprises:
and a database deleting module configured to receive a deleting instruction for the third database, and delete the third database and the second database stored in the third database based on the deleting instruction.
Optionally, the apparatus further comprises:
and the database updating module is configured to receive an updating instruction aiming at the first database and update the first data stored in the first database based on updating data carried in the updating instruction.
Optionally, the apparatus further comprises:
and the database establishing module is configured to establish second data for storing the data snapshot of the updated first data based on the updated first data in the first database.
According to a third aspect of embodiments of the present application, there is provided a computing device comprising a memory, a processor and computer instructions stored on the memory and executable on the processor, the processor implementing the steps of the data processing method when executing the instructions.
According to a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the data processing method.
The embodiment of the application provides a data processing device and a data processing method, wherein the data processing method comprises the steps of extracting the same data from acquired initial data to serve as first data, and taking other data except the first data as second data; establishing a first database based on the first data, and storing the first data to the first database; establishing a second database storing a data snapshot of the first data based on the first data stored in the first database; a third database associated with the second database is established based on the second data, and the second data is stored to the third database. Specifically, the data processing method stores the same data in the acquired initial data into the public first database to form the public first data, and the public first data is shared by the third databases of all branches in a data snapshot mode.
Drawings
FIG. 1 is a block diagram of a computing device provided by an embodiment of the present application;
FIG. 2 is a flow chart of the data processing method provided in an embodiment of the present application;
FIG. 3 is a specific process flow diagram of the data processing method provided in an embodiment of the present application;
fig. 4 is a schematic structural diagram of the data processing apparatus according to the embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is, however, susceptible of embodiment in many other ways than those herein described and similar generalizations can be made by those skilled in the art without departing from the spirit of the application and the application is therefore not limited to the specific embodiments disclosed below.
The terminology used in one or more embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the application. As used in this application in one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present application refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present application. The word "if" as used herein may be interpreted as "responsive to a determination" depending on the context.
First, terms related to one or more embodiments of the present invention will be explained.
SVN: the method is abbreviated as subversion, is an open source version control system, and is used for developing the same project by a plurality of persons in short by adopting efficient management of a branch management system, so that shared resources are realized, and final centralized management is realized.
In the present application, a data processing method and apparatus, a computing device, and a computer-readable storage medium are provided, and detailed descriptions are provided in the following embodiments.
FIG. 1 illustrates a block diagram of a computing device 100, according to an embodiment of the present application. The components of the computing device 100 include, but are not limited to, a memory 110 and a processor 120. Processor 120 is coupled to memory 110 via bus 130 and database 150 is used to store data.
Computing device 100 also includes access device 140, access device 140 enabling computing device 100 to communicate via one or more networks 160. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 140 may include one or more of any type of network interface, wired or wireless (e.g., a Network Interface Card (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present application, the above-described components of computing device 100, as well as other components not shown in FIG. 1, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device shown in FIG. 1 is for exemplary purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as desired.
Computing device 100 may be any type of stationary or mobile computing device including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 100 may also be a mobile or stationary server.
Wherein the processor 120 may perform the steps of the data processing method shown in fig. 2. Fig. 2 shows a flow chart of a data processing method according to an embodiment of the present application, including the following steps.
Step 202: the same data is extracted from the acquired initial data as first data, and other data except the first data are taken as second data.
The data processing method is applied to an SVN library file storage system, and bulk files developed by the same platform are stored in a reasonable mode, so that the storage space of the file storage system is saved; and the data includes, but is not limited to, pictures, videos, log files, and the like.
Specifically, the SVN library file storage system has multiple branches, in practical application, initial data is first obtained from the branches of multiple loose files in the SVN library file storage system, then the same data is extracted from the obtained initial data as first data, and other data except the first data is used as second data.
For example, the acquired initial data includes a, b, c, d, a, b, c, e, a, b, c, then the same data is extracted from the initial data: a. b, c as first data, and other data d, e than the first data a, b, c as second data.
Step 204: a first database is established based on the first data and the first data is stored to the first database.
Wherein the first database, the second database and the third database can be understood as virtual disk.
Specifically, after the first data is determined, a first database is built based on the first data, and the first data is stored to the first database.
Along the above example, a first database is established based on the first data a, b, c, and the first data a, b, c is stored to the first database.
In practical applications, the first database is a database storing the same data obtained from branches of each loose file, which may be also understood as a common database, that is, the same data of branches of each loose file is stored in the first database.
Step 206: based on the first data stored in the first database, a second database storing a data snapshot of the first data is established.
Specifically, at least one second database associated with the first database is established based on the first database, wherein a data snapshot of the first data is stored in the second database.
Along the above example, the first data is a, b, c, then the second database stores snapshots of a, b, c.
Step 208: a third database associated with the second database is established based on the second data, and the second data is stored to the third database.
Specifically, a third database associated with the second database is established based on the second data, and the second data is stored in the third database.
Along with the above example, a third database associated with each second database is established based on the second data d, e, and the second databases d, e are stored to the third database.
In practical application, the third database is at least one and is used for storing different data acquired from branches of each scattered file.
In a specific implementation, the establishing a third database associated with the second database based on the second data, and storing the second data to the third database includes:
at least one third database associated with the second data is established based on the second data, and a data snapshot of the second data and the first data is stored to the third database.
In practical application, in order to facilitate subsequent application, the data snapshot of the first data can be stored in each third database, and in the subsequent specific application, the first data can be directly identified based on the data snapshot of the first data in the third database, so that the data snapshot of the first data does not need to be acquired from the second database, the data processing time is saved, and the data processing efficiency is improved.
According to the data processing method, the same data in the acquired initial data are stored in the public first databases to form the public first data, the third databases of all branches are shared in a data snapshot mode, and the storage space can be greatly saved by adopting the data storage mode.
Referring to fig. 3, fig. 3 shows a specific process flow chart of the data processing method according to an embodiment of the present application, which specifically includes the following steps.
Step 302: raw data is obtained from a plurality of scattered file branches of the SVN library file storage system.
Step 304: the same data a, b and c are extracted from the original data, and the data a, b and c are stored in the newly built virtual disk file A.
Step 306: and a virtual disk file B is newly built based on the virtual disk file A, and the data snapshots of the data a, B and c are respectively stored into the virtual disk file B.
Step 308: and a virtual disk file C is newly built based on the virtual disk file A, and the data snapshots of the data a, b and C are respectively stored into the virtual disk file C.
Step 310: and creating a virtual disk file D based on the virtual disk file B, and storing other data D except the data a, B and c and the data snapshot of the data a, B and c in the original data to the virtual disk file D.
Step 312: and creating a virtual disk file E based on the virtual disk file C, and storing other data E except the data a, b and C and data snapshots of the data a, b and C in the original data to the virtual disk file E.
Specifically, virtual disk file a corresponds to the first database of the above embodiment, virtual disk files B and C correspond to the second database of the above embodiment, and virtual disk files D and E correspond to the third database of the above embodiment.
By the method, the virtual disk files B and C are small, and the occupied space is small because no substantial content is put, but common data a, B and C can be found through the virtual disk files B and C; the size of the virtual disk file D is the size of the stored data D, and the public data a, b, c can be found, and the size of the virtual disk file E is the size of the stored data E, and the public data a, b, c can be found. The memory space occupation of each branch can be made smaller. In practical application, if the data d or the data E is not needed, the virtual disk file E can be directly deleted, and the data need to be screened and deleted when the data is deleted unlike the traditional data storage mode, so that the data processing efficiency is low; if the virtual disk file E needs to be re-established in the subsequent practical application, the virtual disk file C is directly based on the virtual disk file E, and the public data a, b and C can be found from the newly-established virtual disk file E, so that the user experience is improved.
In another embodiment of the present application, after the storing the second data in the third database, the method further includes:
Receiving a data storage request, wherein the data storage request carries data to be stored;
determining whether the data to be stored matches the first data,
if yes, the process is finished,
and if not, storing the data to be stored as the second data to the third database.
In practical application, after the database storage structures of the first database, the second database and the third database are established, when new data are stored, the data can be quickly and reasonably stored based on the first database, the second database and the third database.
If the data to be stored comprises b, c and f, judging whether the b, c and f are matched with the first data a, b and c, if so, not storing the data to be stored, and if not, storing the data to be stored as the second data to a third database.
That is, the data b and c to be stored can be determined to be matched with the data b and c in the first data through judgment, and then the data b and c to be stored are not stored repeatedly at the moment because the first data b and c are already stored in the public first database; meanwhile, the fact that the data f to be stored is not matched with the first data a, b and c can be determined through judgment, and in order to ensure the integrity of the data, the data f to be stored is stored in a corresponding third database. In practical applications, there may be multiple third databases, and when the data f to be stored is stored, the data f to be stored may be stored in the corresponding third database according to practical applications, for example, the data f to be stored is stored in the third database with smaller data storage amount, or the data f to be stored is stored in the third database with the same data attribute (such as a picture format, an audio format, etc.), and so on.
In another embodiment of the present application, after the storing the second data in the third database, the method further includes:
receiving a data acquisition request, wherein the data acquisition request carries data to be acquired;
judging whether the data to be acquired is matched with the first data,
if so, acquiring first data matched with the data to be acquired from the first database as the data to be acquired,
if not, second data matched with the data to be acquired is acquired from the third database and is used as the data to be acquired.
Specifically, after the data is reasonably stored in the first database or the third database, the SVN library file storage system may further receive a data acquisition request, so as to acquire the required data from the first database and/or the third database.
Along the above example, if the data to be acquired includes b, c, d, whether b, c, d are matched with the first data a, b, c is firstly judged, if yes, the first data matched with the data to be acquired is acquired from the first database as the data to be acquired, and if not, the second data matched with the data to be acquired is acquired from the third database as the data to be acquired.
Namely, the data b and c to be stored can be determined to be matched with the data b and c in the first data through judgment, and then the first data b and c matched with the data to be obtained are obtained from the first database and are used as the data b and c to be obtained; meanwhile, the fact that the data d to be stored are not matched with the first data a, b and c and are matched with the second data d stored in the third database can be determined through judgment, and the second data d matched with the data to be obtained are obtained from the third database and serve as the data d to be obtained. In practical application, the data is stored in the above manner, when the data is acquired later, the public part of the data to be acquired can be acquired from the public first database, the other part of the data to be acquired is acquired from the third database, and then the data extracted from the first database and the third database are combined to form the data to be acquired required in practical application. By the data acquisition mode, the situation that each data needs to be acquired from different databases is avoided, the searching time of the data during data acquisition is greatly saved, and the data acquisition efficiency is improved.
In a specific implementation, the determining whether the data to be stored matches the first data includes:
Obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
In practical application, when judging whether the data to be stored is matched with the first data, the data snapshot of the first data is obtained from the second database or the third database, the data to be stored is matched with the data snapshot of the first data, the waste of data processing time when the total first data is required to be obtained from the first database for sequential comparison is avoided, the data snapshot with smaller data occupation space is obtained, and the data matching efficiency is greatly improved.
In another embodiment of the present disclosure, the storing the data to be stored as the second data in the third database includes:
and under the condition that the third database reaches the preset storage condition, creating at least one second database for storing the data snapshot of the first data, and at least one third database associated with the created at least one second database, and storing the data to be stored and the data snapshot of the first data into the created at least one third database.
The preset storage condition may be set according to an actual application, which is not limited in this application, for example, the preset storage condition is that the storage amount of the third database is greater than a preset storage threshold, or attribute information of the data to be stored is not matched with data attribute information stored in the third database.
In practical applications, the data to be stored is stored in the third databases, and the third databases may be full, or the data attribute stored in each third database is different, for example, the third databases can only store the data in the picture format. Then, in order to ensure that the data to be stored can be stored and avoid loss, whether the data is full or the attribute information of the data to be stored is not matched with the data attribute stored in the current third data, at least one second database storing the data snapshot of the first data and at least one third database associated with the newly-built at least one second database are newly built, and the data to be stored and the data snapshot of the first data are stored in the newly-built at least one third database.
In another embodiment of the present disclosure, after the storing the second data in the third database, the method further includes:
And receiving a deleting instruction aiming at the third database, and deleting the third database and the second data stored in the third database based on the deleting instruction.
In practical application, since the non-public data is stored in each third database, in order to save the storage space of the file storage system, the third databases and the second data stored in the third databases can be deleted directly under the condition that the non-public data is not used, and when the third databases are needed to be used again later, the third databases can be newly built.
By adopting the mode, the third database can be deleted to vacate the storage space when the third database is idle, and the third database can be established as required when the third database is used, so that the user experience is good.
In another embodiment of the present disclosure, after the storing the second data in the third database, the method further includes:
and receiving an update instruction aiming at the first database, and updating the first data stored in the first database based on update data carried in the update instruction.
In practical application, since the first database stores public data, the public data may be updated according to actual project requirements, so as to meet specific applications.
In the above example, if the update data is 1, 2, or 3, the first data a, b, or c in the first database is updated to 1, 2, or 3.
In addition, since the second database stores the data snapshot of the first data, in the case of updating the first data, in order to ensure synchronization between the data snapshot of the first data in the second database and the first data in the first database, the data snapshot of the first database in the second database needs to be updated. The specific implementation mode is as follows:
after updating the first data stored in the first database based on the update data carried in the update instruction, the method further includes:
and establishing a second database for storing the data snapshot of the updated first data based on the updated first data in the first database.
In practical application, if the first data in the first database is updated, the original second database is deleted, after the first data in the first database is updated, the second database associated with the first database is re-established based on the updated first data, so that the situation that data update is disordered, data repetition is generated or data loss is caused when the data snapshot in the second database is updated directly based on the updated first data is avoided.
In practical application, the data processing method is applied to an SVN library file storage system, and firstly, a packing VHD (virtual disk) is adopted, and data of each branch is stored in a mode that other virtual disks (used for storing difference data) are connected under the virtual disks. The common portion of each branch is extracted into the base elements and these base elements are combined as needed into the stored portion of the virtual disk. Because of the similarity of the data of each branch, which is developed on the same windows, multiple branches of data may share a backbone (i.e., the first database). By adopting the storage mode, a great amount of storage space can be saved.
Specifically, the scheme of the application is that the common parts of all branches are extracted and stored by using a snapshot method. I.e. a backbone (first database storing common data) and a branch (third database storing difference data) and using a snapshot, when a certain branch lacks which element of the common part, it is directly taken from the snapshot. In this way, for the operation of the public part, no artificial extraction is needed, for example, the a library backbone a can be directly fetched and placed in the virtual disk X, then a difference Y disk (inherited from the virtual disk X, where Y is the same as X, and has no difference and negligible size) is generated, and then in the Y disk, the a library backbone a is directly switched to the branch b by using the svn instruction switch (switching branch instruction), so that the Y disk size is the difference size between the a library backbone a and the branch b.
For example: the original data comprise A1, A2, A3 and A4, and the original A1, A2, A3 and A4 can be copied through a snapshot function; the common element library is then extracted and stored A1, A2, A3, A4. And elements are extracted from the common element library for combination at the time of respective branch data storage.
If the data of a certain branch is A1, A2, A5 and A6, the system automatically extracts A1 and A2 from the public element library. Thus, only the A5 and the A6 are required to be stored separately, the other part of the data A1 and the A2 are directly extracted and combined from the public element library, the elements (A1 and A2) extracted from the public element library are finally combined with the data (A5 and A6) stored separately, and the final data can be obtained.
Corresponding to the above method embodiments, the present application further provides an embodiment of a data processing apparatus, and fig. 4 shows a schematic structural diagram of the data processing apparatus according to one embodiment of the present application. As shown in fig. 4, the apparatus includes:
a data extraction module 402 configured to extract the same data from the acquired initial data as first data, and other data than the first data as second data;
a first database creation module 404 configured to create a first database based on the first data and store the first data to the first database;
A second database creation module 406 configured to create a second database storing a data snapshot of the first data based on the first data stored in the first database;
a third database creation module 408 configured to create a third database associated with the second database based on the second data and store the second data to the third database.
Optionally, the third database creation module 408 is further configured to:
at least one third database associated with the second data is established based on the second data, and a data snapshot of the second data and the first data is stored to the third database.
Optionally, the apparatus further comprises:
a data storage module configured to:
receiving a data storage request, wherein the data storage request carries data to be stored;
determining whether the data to be stored matches the first data,
if yes, the process is finished,
and if not, storing the data to be stored as the second data to the third database.
Optionally, the apparatus further comprises:
a data acquisition module configured to:
Receiving a data acquisition request, wherein the data acquisition request carries data to be acquired;
judging whether the data to be acquired is matched with the first data,
if so, acquiring first data matched with the data to be acquired from the first database as the data to be acquired,
if not, second data matched with the data to be acquired is acquired from the third database and is used as the data to be acquired.
Optionally, the data storage module is further configured to:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
Optionally, the data acquisition module is further configured to:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
Optionally, the third database creation module 408 is further configured to:
and under the condition that the third database reaches the preset storage condition, creating at least one second database for storing the data snapshot of the first data, and at least one third database associated with the created at least one second database, and storing the data to be stored and the data snapshot of the first data into the created at least one third database.
Optionally, the apparatus further comprises:
and a database deleting module configured to receive a deleting instruction for the third database, and delete the third database and the second database stored in the third database based on the deleting instruction.
Optionally, the apparatus further comprises:
and the database updating module is configured to receive an updating instruction aiming at the first database and update the first data stored in the first database based on updating data carried in the updating instruction.
Optionally, the apparatus further comprises:
and the database establishing module is configured to establish second data for storing the data snapshot of the updated first data based on the updated first data in the first database.
In this embodiment of the present application, the data processing device stores the same data in the obtained initial data into the common first database to form the common first data, and the data snapshot mode is used for sharing the third databases of each branch, so that the storage space can be greatly saved by adopting the data storage mode.
The above is a schematic solution of a data processing apparatus of the present embodiment. It should be noted that, the technical solution of the data processing apparatus and the technical solution of the data processing method belong to the same conception, and details of the technical solution of the data processing apparatus, which are not described in detail, can be referred to the description of the technical solution of the data processing method.
It should be noted that, the components in the apparatus claims should be understood as functional modules that are necessary to be established for implementing the steps of the program flow or the steps of the method, and the functional modules are not actually functional divisions or separate limitations. The device claims defined by such a set of functional modules should be understood as a functional module architecture for implementing the solution primarily by means of the computer program described in the specification, and not as a physical device for implementing the solution primarily by means of hardware.
An embodiment of the present application further provides a computing device including a memory, a processor, and computer instructions stored on the memory and executable on the processor, where the processor executes the instructions to implement the steps of the data processing method.
The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the data processing method.
An embodiment of the present application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of a data processing method as described above.
The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the data processing method.
The foregoing describes specific embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all necessary for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
The above-disclosed preferred embodiments of the present application are provided only as an aid to the elucidation of the present application. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of this application. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This application is to be limited only by the claims and the full scope and equivalents thereof.
Claims (12)
1. A method of data processing, comprising:
extracting the same data from the acquired initial data as first data, and taking other data except the first data as second data, wherein the first database is a public database for storing the same data acquired from branches of each scattered file;
establishing a first database based on the first data, and storing the first data to the first database;
establishing a second database storing a data snapshot of the first data based on the first data stored in the first database;
and establishing a third database associated with the second database based on the second data, and storing the second data to the third database, wherein the third database is used for storing different data acquired from branches of each scattered file, and the first database, the second database and the third database are virtual disks.
2. The data processing method according to claim 1, wherein the creating a third database associated with the second database based on the second data, and storing the second data to the third database, comprises:
At least one third database associated with the second data is established based on the second data, and a data snapshot of the second data and the first data is stored to the third database.
3. The data processing method according to claim 1, wherein after the storing of the second data in the third database, further comprising:
receiving a data storage request, wherein the data storage request carries data to be stored;
determining whether the data to be stored matches the first data,
if yes, the process is finished,
and if not, storing the data to be stored as the second data to the third database.
4. The data processing method according to claim 1, wherein after the storing of the second data in the third database, further comprising:
receiving a data acquisition request, wherein the data acquisition request carries data to be acquired;
judging whether the data to be acquired is matched with the first data,
if so, acquiring first data matched with the data to be acquired from the first database as the data to be acquired,
If not, second data matched with the data to be acquired is acquired from the third database and is used as the data to be acquired.
5. A data processing method according to claim 3, wherein said determining whether the data to be stored matches the first data comprises:
obtaining a data snapshot of the first data from the second database or the third database;
and judging whether the data to be stored is matched with the first data in the data snapshot.
6. A data processing method according to claim 3, wherein said storing said data to be stored as said second data to said third database comprises:
and under the condition that the third database reaches the preset storage condition, creating at least one second database for storing the data snapshot of the first data, and at least one third database associated with the created at least one second database, and storing the data to be stored and the data snapshot of the first data into the created at least one third database.
7. The data processing method according to claim 1 or 2, characterized in that after the storing of the second data in the third database, further comprising:
And receiving a deleting instruction aiming at the third database, and deleting the third database and the second data stored in the third database based on the deleting instruction.
8. The data processing method according to claim 1 or 2, characterized in that after the storing of the second data in the third database, further comprising:
and receiving an update instruction aiming at the first database, and updating the first data stored in the first database based on update data carried in the update instruction.
9. The method according to claim 8, wherein after updating the first data stored in the first database based on the update data carried in the update instruction, further comprising:
and establishing a second database for storing the data snapshot of the updated first data based on the updated first data in the first database.
10. A data processing apparatus, comprising:
a data extraction module configured to extract the same data from the acquired initial data as first data, and other data than the first data as second data;
A first database creation module configured to create a first database based on the first data and store the first data to the first database, wherein the first database is a common database storing the same data acquired from branches of respective loose files;
a second database creation module configured to create a second database storing a data snapshot of the first data based on the first data stored in the first database;
and a third database establishing module configured to establish a third database associated with the second database based on the second data, and store the second data to the third database, wherein the third database is used for storing different data acquired from branches of the scattered files, and the first database, the second database and the third database are virtual disks.
11. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein the processor, when executing the instructions, implements the steps of the method of any of claims 1-9.
12. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011600572.5A CN112579611B (en) | 2020-12-29 | 2020-12-29 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011600572.5A CN112579611B (en) | 2020-12-29 | 2020-12-29 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112579611A CN112579611A (en) | 2021-03-30 |
| CN112579611B true CN112579611B (en) | 2024-03-19 |
Family
ID=75144197
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011600572.5A Active CN112579611B (en) | 2020-12-29 | 2020-12-29 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579611B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1995011487A1 (en) * | 1993-10-22 | 1995-04-27 | Fdc, Inc. | Database using table rotation and bimapped queries |
| CN101477447A (en) * | 2009-01-21 | 2009-07-08 | 杭州华三通信技术有限公司 | Disk snap method and apparatus |
| CN107171825A (en) * | 2017-04-11 | 2017-09-15 | 捷开通讯(深圳)有限公司 | A kind of repetition daily record filter method of terminal |
| CN107220266A (en) * | 2016-03-22 | 2017-09-29 | 阿里巴巴集团控股有限公司 | It is a kind of to create Service Database, business datum storage and the method and device for determining business datum |
| CN109324998A (en) * | 2018-09-18 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of document handling method, apparatus and system |
| CN110292775A (en) * | 2019-07-01 | 2019-10-01 | 网易（杭州）网络有限公司 | Obtain the method and device of variance data |
| CN110618974A (en) * | 2019-09-06 | 2019-12-27 | 苏州浪潮智能科技有限公司 | Data storage method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10868674B2 (en) * | 2016-08-12 | 2020-12-15 | ALTR Solutions, Inc. | Decentralized database optimizations |
| JP6905183B2 (en) * | 2017-05-16 | 2021-07-21 | 富士通株式会社 | Information processing equipment, programs and information processing methods |
-
2020
- 2020-12-29 CN CN202011600572.5A patent/CN112579611B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1995011487A1 (en) * | 1993-10-22 | 1995-04-27 | Fdc, Inc. | Database using table rotation and bimapped queries |
| CN101477447A (en) * | 2009-01-21 | 2009-07-08 | 杭州华三通信技术有限公司 | Disk snap method and apparatus |
| CN107220266A (en) * | 2016-03-22 | 2017-09-29 | 阿里巴巴集团控股有限公司 | It is a kind of to create Service Database, business datum storage and the method and device for determining business datum |
| CN107171825A (en) * | 2017-04-11 | 2017-09-15 | 捷开通讯(深圳)有限公司 | A kind of repetition daily record filter method of terminal |
| CN109324998A (en) * | 2018-09-18 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of document handling method, apparatus and system |
| CN110292775A (en) * | 2019-07-01 | 2019-10-01 | 网易（杭州）网络有限公司 | Obtain the method and device of variance data |
| CN110618974A (en) * | 2019-09-06 | 2019-12-27 | 苏州浪潮智能科技有限公司 | Data storage method, device, equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| Dominik Leibenger等.A storage-efficient cryptography-based access control solution for subversion.SACMAT '13: Proceedings of the 18th ACM symposium on Access control models and technologies.2013,201–212. * |
| 分散式云存储加密系统设计与实现;徐霞;中国优秀硕士学位论文全文数据库 (信息科技辑);20170315;I137-249 * |
| 基于SVN的项目文档管理系统实现;丁贤林;中国优秀硕士学位论文全文数据库 (信息科技辑);20120615;I138-941 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112579611A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569311B (en) | Data synchronization method and device for database and computer storage medium | |
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| CN109359091B (en) | File management method, device, terminal and computer readable storage medium | |
| CN107748780B (en) | Recovery method and device for file of recycle bin | |
| CN109408682B (en) | Regular expression matching method, system and equipment | |
| CN108205560A (en) | A kind of method of data synchronization and device | |
| CN111949324A (en) | Distributed serial number generation method and device | |
| CN113297269A (en) | Data query method and device | |
| CN112052251B (en) | Target data updating method and related device, equipment and storage medium | |
| CN112579611B (en) | Data processing method and device | |
| CN113326810A (en) | Face recognition method, system, device, electronic equipment and storage medium | |
| CN115422270A (en) | Information processing method and device | |
| CN110737458A (en) | code updating method and related device | |
| CN111158821B (en) | Task processing method and device | |
| CN110750561A (en) | Method and device for mining associated application program | |
| CN114020962A (en) | Video system management method, electronic device, and computer-readable storage medium | |
| CN113657136A (en) | Identification method and device | |
| CN114788294A (en) | Content pushing method, device, server and storage medium | |
| CN109710254B (en) | Page view construction method and device and storage medium | |
| CN116126804A (en) | File processing method and device | |
| CN113297199B (en) | Method and device for using spatiotemporal data engine and Cassandra database system | |
| CN114527944A (en) | Resource file processing method and device | |
| CN114661774A (en) | Data processing method and device | |
| CN110147531B (en) | Method, device and storage medium for identifying similar text content | |
| CN113297319A (en) | Data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 519000 Room 102, 202, 302 and 402, No. 325, Qiandao Ring Road, Tangjiawan Town, high tech Zone, Zhuhai City, Guangdong Province, Room 102 and 202, No. 327 and Room 302, No. 329 Applicant after: Zhuhai Jinshan Digital Network Technology Co.,Ltd. Applicant after: Zhuhai Xishanju Digital Technology Co.,Ltd. Address before: 519000 Room 102, 202, 302 and 402, No. 325, Qiandao Ring Road, Tangjiawan Town, high tech Zone, Zhuhai City, Guangdong Province, Room 102 and 202, No. 327 and Room 302, No. 329 Applicant before: ZHUHAI KINGSOFT ONLINE GAME TECHNOLOGY Co.,Ltd. Applicant before: ZHUHAI SEASUN MOBILE GAME TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |