CN106990914B - Data deleting method and device - Google Patents
Data deleting method and device Download PDFInfo
- Publication number
- CN106990914B CN106990914B CN201710091431.7A CN201710091431A CN106990914B CN 106990914 B CN106990914 B CN 106990914B CN 201710091431 A CN201710091431 A CN 201710091431A CN 106990914 B CN106990914 B CN 106990914B
- Authority
- CN
- China
- Prior art keywords
- data
- objects
- mode
- processed
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
- G06F3/0641—De-duplication techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data deleting method, which comprises the following steps: acquiring data to be processed; determining objects matched with each other in data to be processed; comparing the data corresponding to the objects matched with each other, and determining repeated data among the objects matched with each other; deleting the determined duplicate data. The invention also discloses a data deleting device. The invention effectively and reasonably deletes data and improves storage space.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data deletion method and apparatus.
Background
With the continuous promotion of the information process of the social digital network, the data volume of the global IT enterprise management is increased dramatically, and the current large-scale data center has continuous requirements on the aspects of expansibility, performance, cost and the like for the management of complex data. In order to slow down the increase of the storage capacity of an enterprise, the traditional data de-duplication storage management technology and method cannot meet the service quality requirement of big data backup application, and new software and hardware technology progresses to bring opportunities for improving the big data management capacity.
Most of deduplication systems in the market at present need to rely on a metadata index algorithm based on fingerprints in order to identify redundant data blocks and locate data blocks, especially in a large data storage environment, the number of fingerprints representing data is also large, and the system needs to pay expensive cost to maintain the fingerprint data structure to ensure deduplication performance. Although the technologies can effectively reduce redundant data in the system, the main idea is to improve deduplication performance by adopting data source locality characteristics, and locality characteristics are not obvious in distributed storage and cannot effectively and reasonably delete data, so that redundant data is not deleted thoroughly, and storage space is occupied.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a data deleting method and a data deleting device, and aims to solve the problems that the local characteristics are not obvious in distributed storage, data cannot be effectively and reasonably deleted, redundant data cannot be completely deleted and storage space is occupied due to the fact that the data source local characteristics are adopted to improve the deduplication performance in the conventional main thought of deduplication.
In order to achieve the above object, the present invention provides a data deleting method, which comprises the steps of:
acquiring data to be processed;
determining objects matched with each other in data to be processed;
comparing the data corresponding to the objects matched with each other, and determining repeated data among the objects matched with each other;
deleting the determined duplicate data.
Preferably, the step of acquiring the data to be processed includes:
determining a mode of data deletion;
when the data deleting mode is a real-time deleting mode, acquiring currently stored data and historically stored data, and taking the currently stored data and the historically stored data as data to be processed;
and when the mode of data deletion is a timed deletion mode, acquiring the currently stored data, and taking the currently stored data as the data to be processed.
Preferably, before the acquiring the data to be processed, the method further includes:
receiving data to be stored, slicing the data to be stored into slice data blocks with preset sizes;
and storing each slice data block in an object mode, and storing the object and the fingerprint index of each slice data in a data structure.
Preferably, the step of determining the objects matching each other in the data to be processed includes:
determining an object contained in the data to be stored;
and adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining the objects matched with each other through a Hash algorithm.
Preferably, the step of comparing the data corresponding to the objects matched with each other and determining the repeated data between the objects matched with each other includes:
comparing the data corresponding to the matched objects, and calculating the correctness of the data through an MD5 algorithm to obtain effective data;
and determining repeated data between the matched objects from the valid data.
In addition, to achieve the above object, the present invention also provides a data deleting device, including:
the acquisition module is used for acquiring data to be processed;
the determining module is used for extracting objects which are matched with each other in the data to be processed;
the comparison module is used for comparing the data corresponding to the objects which are matched with each other and determining the repeated data among the objects which are matched with each other;
a deletion module to delete the determined duplicate data.
Preferably, the acquisition module comprises
A determination unit configured to determine a mode of data deletion;
the acquisition unit is used for acquiring currently stored data and historically stored data when the data deletion mode is a real-time deletion mode, and taking the currently stored data and the historically stored data as data to be processed; the acquisition unit is also used for
And when the mode of data deletion is a timed deletion mode, acquiring the currently stored data, and taking the currently stored data as the data to be processed.
Preferably, the apparatus further comprises:
the processing module is used for receiving data to be stored, slicing the data to be stored and cutting the data into slice data blocks with preset sizes;
and the storage module is used for storing each slice data block in an object mode and storing the object and the fingerprint index of each slice data in a data structure.
Preferably, the determining module is further configured to determine an object included in the data to be stored; the determination module is also used for
And adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining the objects matched with each other through a Hash algorithm.
Preferably, the comparison module is further configured to compare data corresponding to the objects that are matched with each other, and calculate correctness of the data through an MD5 algorithm to obtain valid data; the comparison module is also used for
And determining repeated data between the matched objects from the valid data.
The invention provides an object-based data deduplication mode, which effectively overcomes the defect of insufficient local characteristics, finds out repetitive data among objects matched with each other by using the objects, deletes the repetitive data, realizes high-performance parallel data deduplication processing in a single node, and expands the throughput of data deduplication. The data is effectively and reasonably deleted, and the storage space is improved.
Drawings
FIG. 1 is a flow chart illustrating an embodiment of a data deleting method according to the present invention;
FIG. 2 is a schematic flow chart illustrating the process of acquiring data to be processed according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating data storage according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart illustrating the determination of duplicate data according to an embodiment of the present invention;
FIG. 5 is an architecture diagram of deduplication in an embodiment of the present invention;
FIG. 6 is a schematic diagram of data deletion in an embodiment of the present invention;
FIG. 7 is a functional block diagram of a data deleting device according to an embodiment of the present invention;
FIG. 8 is a block diagram illustrating a detailed function of the obtaining module shown in FIG. 7 according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a data deleting method.
Referring to fig. 1, fig. 1 is a flowchart illustrating a data deleting method according to an embodiment of the present invention.
In one embodiment, the data deleting method includes:
step S10, acquiring data to be processed;
in this embodiment, when data needs to be processed, the processed data is to-be-processed data, for example, when data needs to be deleted, the to-be-deleted data is to-be-processed data; or, for another example, when data needs to be deleted, the historical data and the data to be currently deleted are to-be-processed data. Referring to fig. 2, the manner of acquiring the data to be processed includes:
step S11, determining a data deletion mode;
step S12, when the data deleting mode is a real-time deleting mode, acquiring the currently stored data and the historically stored data, and taking the currently stored data and the historically stored data as the data to be processed;
step S13, when the mode of data deletion is the timed deletion mode, acquiring the currently stored data, and using the currently stored data as the data to be processed.
The data deleting mode includes a timing deleting mode and a real-time deleting mode. The setting of the data deletion mode is set by a user after the system is started, or the data deletion mode is operated in a default deletion mode. When data needs to be deleted, a mode of data deletion is determined, for example, a timed deletion mode or a real-time deletion mode. When the real-time deletion mode is determined, taking the historical data and the current data as the data to be processed, or directly taking the current data as the data to be processed; and when the mode is determined to be the timed deletion mode, taking all the currently stored data as the data to be processed, or taking the data stored between the previous time node and the current time node as the data to be processed. The definition of the data to be processed in the real-time and timed deletion modes is set by a user or default of a system and is a corresponding definition mode. And when the data needs to be deleted, acquiring the data to be processed according to a set mode.
Step S20, determining objects matched with each other in the data to be processed;
during data storage, the data storage mode is storage according to an object mode, namely, data are classified according to objects, then data corresponding to the objects, namely, data of a receiving client side are respectively stored, and the data are respectively stored in the distributed storage system according to the object classification of the received data. Referring to fig. 3, the process of data storage includes:
step S21, receiving data to be stored, slicing the data to be stored into slices of data blocks with preset sizes;
in step S22, each slice data block is stored in an object manner, and the object and the fingerprint index of each slice data are stored in a data structure.
The data sent by the client is received, the data of the client is sliced, that is, the data transmitted by the client is divided into a plurality of small data blocks, each data block is divided into 128M (according to different requirements or system performance settings or according to the total size of the transmitted data, for example, 64M or 256M and the like) in our distributed storage system, and the data is sent to the data storage system. After slicing the data, stability and reliability of data transmission are improved, and if the data are not segmented into a plurality of data blocks under the condition of poor network conditions, a large amount of data are transmitted at one time, data can be subjected to data verification for many times in the transmission process, transmission time is prolonged, data loss or data north blocking can be caused, and stability and reliability of the data are influenced.
The method is characterized in that effective data and fingerprint indexes are stored in a cache in a data structure form, and then the whole object is transmitted to a storage system through a network, so that the mutual relation between the data and the fingerprint indexes is established, and the subsequent operations of searching and acquiring the data and the like are facilitated.
Step S30, comparing the data corresponding to the matched objects and determining the repeated data between the matched objects;
in the real-time deletion mode, when data are received, the currently received data are checked, historical data are compared with the currently received data, and the similarity between the current data and the historical data is found, namely the similar data are found; in the case of the timed deletion mode, when the timed time is reached, the manner of determining similar data is the same as that of deleting data in real time, but the compared data is data to be processed, for example, the data is stored in whole, or the data from the last time node to the current stored data is compared with the data stored before the last time node, so as to find similar data.
Specifically, the process of determining duplicate data includes:
step S31, determining an object included in the data to be stored;
step S32, adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining the objects matched with each other through a Hash algorithm;
s33, comparing the data corresponding to the matched objects, and calculating the correctness of the data through an MD5 algorithm to obtain effective data;
in step S34, duplicate data between objects that match each other is determined from the valid data.
In the process of deleting real-time or timing data, different objects are added into an index fingerprint queue, object fingerprint comparison is firstly carried out, objects with higher similarity (the similarity is greater than a preset threshold, for example, the preset threshold is 80% similar or 70% similar and the like) are extracted through a Hash algorithm, then the correctness of the data is calculated according to an MD5 algorithm, if the data is found to be unmodified and the same copy exists in the data, the repeated data is found, and the found repeated data is used as the repeated data among the objects which are matched with each other.
In step S40, the determined duplicate data is deleted.
The repeated data is deleted, namely, the redundant data is deleted, so that a large amount of storage space is saved, the storage capacity of the disk is improved, and the storage efficiency is improved due to the improvement of the storage capacity.
For better describing the embodiment of the present invention, refer to fig. 5, which is an architecture diagram of data de-duplication according to the present invention, and refer to fig. 6, which is a schematic diagram of data de-duplication.
The embodiment provides an object-based data deduplication mode, which effectively overcomes the defect of insufficient local features, finds out repetitive data among objects matched with each other by using the objects, deletes the repetitive data, realizes high-performance parallel data deduplication processing in a single node, and expands throughput of data deduplication. The data is effectively and reasonably deleted, and the storage space is improved.
The invention further provides a data deleting device.
Referring to fig. 7, fig. 7 is a functional module diagram of a data deleting device according to an embodiment of the present invention.
In one embodiment, the data deleting device includes: the device comprises an acquisition module 10, a determination module 20, a processing module 30, a storage module 40, a comparison module 50 and a deletion module 60.
The acquiring module 10 is configured to acquire data to be processed;
in this embodiment, when data needs to be processed, the processed data is to-be-processed data, for example, when data needs to be deleted, the to-be-deleted data is to-be-processed data; or, for another example, when data needs to be deleted, the historical data and the data to be currently deleted are to-be-processed data. Referring to fig. 8, the acquisition module 10 includes: a determination unit 11 and an acquisition unit 12,
the determining unit 11 is configured to determine a mode of data deletion;
the acquiring unit 12 is configured to acquire currently stored data and historically stored data when the data deletion mode is the real-time deletion mode, and use the currently stored data and the historically stored data as data to be processed; the acquisition unit 12 is also used for
And when the mode of data deletion is a timed deletion mode, acquiring the currently stored data, and taking the currently stored data as the data to be processed.
The data deleting mode includes a timing deleting mode and a real-time deleting mode. The setting of the data deletion mode is set by a user after the system is started, or the data deletion mode is operated in a default deletion mode. When data needs to be deleted, a mode of data deletion is determined, for example, a timed deletion mode or a real-time deletion mode. When the real-time deletion mode is determined, taking the historical data and the current data as the data to be processed, or directly taking the current data as the data to be processed; and when the mode is determined to be the timed deletion mode, taking all the currently stored data as the data to be processed, or taking the data stored between the previous time node and the current time node as the data to be processed. The definition of the data to be processed in the real-time and timed deletion modes is set by a user or default of a system and is a corresponding definition mode. And when the data needs to be deleted, acquiring the data to be processed according to a set mode.
The determining module 20 is configured to determine objects matching with each other in the data to be processed;
during data storage, the data storage mode is storage according to an object mode, namely, data are classified according to objects, then data corresponding to the objects, namely, data of a receiving client side are respectively stored, and the data are respectively stored in the distributed storage system according to the object classification of the received data.
The processing module 30 is configured to receive data to be stored, slice the data to be stored, and cut the data into slice data blocks of a preset size;
the storage module 40 is configured to store each slice data block in an object manner, and store a data structure formed by the object and the fingerprint index of each slice data.
The data sent by the client is received, the data of the client is sliced, that is, the data transmitted by the client is divided into a plurality of small data blocks, each data block is divided into 128M (according to different requirements or system performance settings or according to the total size of the transmitted data, for example, 64M or 256M and the like) in our distributed storage system, and the data is sent to the data storage system. After slicing the data, stability and reliability of data transmission are improved, and if the data are not segmented into a plurality of data blocks under the condition of poor network conditions, a large amount of data are transmitted at one time, data can be subjected to data verification for many times in the transmission process, transmission time is prolonged, data loss or data north blocking can be caused, and stability and reliability of the data are influenced.
The method is characterized in that effective data and fingerprint indexes are stored in a cache in a data structure form, and then the whole object is transmitted to a storage system through a network, so that the mutual relation between the data and the fingerprint indexes is established, and the subsequent operations of searching and acquiring the data and the like are facilitated.
The comparing module 50 is configured to compare data corresponding to the objects that are matched with each other, and determine repeated data between the objects that are matched with each other;
in the real-time deletion mode, when data are received, the currently received data are checked, historical data are compared with the currently received data, and the similarity between the current data and the historical data is found, namely the similar data are found; in the case of the timed deletion mode, when the timed time is reached, the manner of determining similar data is the same as that of deleting data in real time, but the compared data is data to be processed, for example, the data is stored in whole, or the data from the last time node to the current stored data is compared with the data stored before the last time node, so as to find similar data.
The determining module 20 is further configured to determine an object included in the data to be stored; the determination module 20 is also used for
Adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining objects matched with each other through a Hash algorithm;
the comparison module 50 is further configured to compare data corresponding to the objects that are matched with each other, and calculate correctness of the data through an MD5 algorithm to obtain valid data; the alignment module 50 is also used for
And determining repeated data between the matched objects from the valid data.
In the process of deleting real-time or timing data, different objects are added into an index fingerprint queue, object fingerprint comparison is firstly carried out, objects with higher similarity (the similarity is greater than a preset threshold, for example, the preset threshold is 80% similar or 70% similar and the like) are extracted through a Hash algorithm, then the correctness of the data is calculated according to an MD5 algorithm, if the data is found to be unmodified and the same copy exists in the data, the repeated data is found, and the found repeated data is used as the repeated data among the objects which are matched with each other.
The deleting module 60 is configured to delete the determined duplicate data.
The repeated data is deleted, namely, the redundant data is deleted, so that a large amount of storage space is saved, the storage capacity of the disk is improved, and the storage efficiency is improved due to the improvement of the storage capacity.
For better describing the embodiment of the present invention, refer to fig. 5, which is an architecture diagram of data de-duplication according to the present invention, and refer to fig. 6, which is a schematic diagram of data de-duplication.
The embodiment provides an object-based data deduplication mode, which effectively overcomes the defect of insufficient local features, finds out repetitive data among objects matched with each other by using the objects, deletes the repetitive data, realizes high-performance parallel data deduplication processing in a single node, and expands throughput of data deduplication. The data is effectively and reasonably deleted, and the storage space is improved.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (6)
1. A method for deleting data, comprising the steps of:
receiving data to be stored, and cutting the data to be stored into slice data blocks with preset sizes;
storing each slice data block in an object mode, and combining the object of each slice data and the fingerprint index into a data structure for storage;
acquiring data to be processed;
determining objects matched with each other in data to be processed;
comparing the data corresponding to the objects matched with each other, and determining repeated data among the objects matched with each other;
deleting the determined duplicate data;
the method comprises the following steps of determining objects matched with each other in data to be processed, wherein the steps comprise:
determining an object corresponding to each slice data block in the data to be stored;
and adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining the objects matched with each other through a Hash algorithm.
2. The data deletion method of claim 1, wherein the step of obtaining the data to be processed comprises:
determining a mode of data deletion;
when the data deleting mode is a real-time deleting mode, acquiring currently stored data and historically stored data, and taking the currently stored data and the historically stored data as data to be processed;
and when the mode of data deletion is a timed deletion mode, acquiring the currently stored data, and taking the currently stored data as the data to be processed.
3. The data deletion method of claim 1, wherein the step of comparing data corresponding to objects that match each other and determining data that is repeated between the objects that match each other comprises:
comparing the data corresponding to the matched objects, and calculating the correctness of the data through an MD5 algorithm to obtain effective data;
and determining repeated data between the matched objects from the valid data.
4. A data deletion apparatus, comprising:
the processing module is used for receiving data to be stored, slicing the data to be stored and cutting the data into slice data blocks with preset sizes;
the storage module is used for storing each slice data block in an object mode, and forming the object and the fingerprint index of each slice data into a data structure for storage;
the acquisition module is used for acquiring data to be processed;
the determining module is used for extracting objects which are matched with each other in the data to be processed;
the comparison module is used for comparing the data corresponding to the objects which are matched with each other and determining the repeated data among the objects which are matched with each other;
a deletion module for deleting the determined duplicate data;
the determining module is further configured to determine an object included in the data to be stored; the determination module is also used for
And adding the determined objects into a fingerprint index queue, performing object fingerprint comparison, and determining the objects matched with each other through a Hash algorithm.
5. The data deletion apparatus according to claim 4, wherein the acquisition module includes a determination unit configured to determine a mode of data deletion;
the acquisition unit is used for acquiring currently stored data and historically stored data when the data deletion mode is a real-time deletion mode, and taking the currently stored data and the historically stored data as data to be processed; the acquisition unit is also used for
And when the mode of data deletion is a timed deletion mode, acquiring the currently stored data, and taking the currently stored data as the data to be processed.
6. The data deleting device according to claim 5, wherein the comparing module is further configured to compare data corresponding to objects that match each other, and calculate correctness of the data through an MD5 algorithm to obtain valid data; the comparison module is also used for
And determining repeated data between the matched objects from the valid data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710091431.7A CN106990914B (en) | 2017-02-17 | 2017-02-17 | Data deleting method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710091431.7A CN106990914B (en) | 2017-02-17 | 2017-02-17 | Data deleting method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106990914A CN106990914A (en) | 2017-07-28 |
| CN106990914B true CN106990914B (en) | 2020-06-12 |
Family
ID=59414503
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710091431.7A Active CN106990914B (en) | 2017-02-17 | 2017-02-17 | Data deleting method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106990914B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108037895B (en) * | 2017-12-06 | 2021-06-22 | Tcl移动通信科技（宁波）有限公司 | Mobile terminal, data information storage control method and storage medium |
| CN109544931A (en) * | 2018-12-18 | 2019-03-29 | 广东赛诺科技股份有限公司 | One kind is based on effective judgment method in traffic overrun and overload data 24 hours |
| CN110442404B (en) * | 2019-08-13 | 2022-07-05 | 武汉斗鱼鱼乐网络科技有限公司 | Object release method, device, equipment and storage medium |
| CN110618789B (en) * | 2019-08-14 | 2021-08-20 | 华为技术有限公司 | Method and device for deleting repeated data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102202098A (en) * | 2011-05-25 | 2011-09-28 | 成都市华为赛门铁克科技有限公司 | Data processing method and device |
| US8041641B1 (en) * | 2006-12-19 | 2011-10-18 | Symantec Operating Corporation | Backup service and appliance with single-instance storage of encrypted data |
| CN102629258A (en) * | 2012-02-29 | 2012-08-08 | 浪潮(北京)电子信息产业有限公司 | Repeating data deleting method and device |
| CN102831222A (en) * | 2012-08-24 | 2012-12-19 | 华中科技大学 | Differential compression method based on data de-duplication |
| CN102915278A (en) * | 2012-09-19 | 2013-02-06 | 浪潮(北京)电子信息产业有限公司 | Data deduplication method |
| CN103870514A (en) * | 2012-12-18 | 2014-06-18 | 华为技术有限公司 | Repeating data deleting method and device |
-
2017
- 2017-02-17 CN CN201710091431.7A patent/CN106990914B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8041641B1 (en) * | 2006-12-19 | 2011-10-18 | Symantec Operating Corporation | Backup service and appliance with single-instance storage of encrypted data |
| CN102202098A (en) * | 2011-05-25 | 2011-09-28 | 成都市华为赛门铁克科技有限公司 | Data processing method and device |
| CN102629258A (en) * | 2012-02-29 | 2012-08-08 | 浪潮(北京)电子信息产业有限公司 | Repeating data deleting method and device |
| CN102831222A (en) * | 2012-08-24 | 2012-12-19 | 华中科技大学 | Differential compression method based on data de-duplication |
| CN102915278A (en) * | 2012-09-19 | 2013-02-06 | 浪潮(北京)电子信息产业有限公司 | Data deduplication method |
| CN103870514A (en) * | 2012-12-18 | 2014-06-18 | 华为技术有限公司 | Repeating data deleting method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106990914A (en) | 2017-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106990914B (en) | Data deleting method and device | |
| US10938961B1 (en) | Systems and methods for data deduplication by generating similarity metrics using sketch computation | |
| US8396873B2 (en) | Index searching using a bloom filter | |
| US8224875B1 (en) | Systems and methods for removing unreferenced data segments from deduplicated data systems | |
| US20120303595A1 (en) | Data restoration method for data de-duplication | |
| CN106611035A (en) | Retrieval algorithm for deleting repetitive data in cloud storage | |
| CN109213738B (en) | Cloud storage file-level repeated data deletion retrieval system and method | |
| CN103020255B (en) | Classification storage means and device | |
| CN111033459B (en) | Microservice-based deduplication optimized for wide area networks | |
| CN105656981A (en) | Data transmission method and system | |
| CN106980680B (en) | Data storage method and storage device | |
| CN111033487A (en) | Microservice-based deduplication | |
| CN103823807A (en) | Data de-duplication method, device and system | |
| US11995050B2 (en) | Systems and methods for sketch computation | |
| CN112632008B (en) | Data slicing transmission method and device and computer equipment | |
| CN109033295B (en) | Method and device for merging super-large data sets | |
| EP4078340A1 (en) | Systems and methods for sketch computation | |
| US20210191640A1 (en) | Systems and methods for data segment processing | |
| Du et al. | Deduplicated disk image evidence acquisition and forensically-sound reconstruction | |
| CN106909623B (en) | A kind of data set and date storage method for supporting efficient mass data to analyze and retrieve | |
| CN111061428B (en) | Data compression method and device | |
| WO2024021491A1 (en) | Data slicing method, apparatus and system | |
| CN110633257A (en) | Real-time synchronization method and system for bank parameter files in private cloud environment | |
| CN105302669A (en) | Method and system for data deduplication in cloud backup process | |
| US10877945B1 (en) | Optimized block storage for change block tracking systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20190823 Address after: 100089 Floor 1-4, No. 2 Building, No. 9 Courtyard, Dijin Road, Haidian District, Beijing Applicant after: Beijing Toyou Feiji Electronics Co., Ltd. Address before: 518000 Feiyada Science and Technology Building, Yuehai Street, Nanshan District, Shenzhen City, Guangdong Province, 1402E Applicant before: Shenzhen City Rui Bo deposit information technology Co. Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210721 Address after: Room g0552, headquarters building, Changsha Zhongdian Software Park Co., Ltd., 39 Jianshan Road, Changsha hi tech Development Zone, Changsha, Hunan 410000 Patentee after: Hunan Tongyou feiji Technology Co.,Ltd. Address before: 100089 floor 1-4, building 2, yard 9, Dijin Road, Haidian District, Beijing Patentee before: BEIJING TOYOU FEIJI ELECTRONICS Co.,Ltd. |