CN112668033B - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN112668033B CN112668033B CN201910980236.9A CN201910980236A CN112668033B CN 112668033 B CN112668033 B CN 112668033B CN 201910980236 A CN201910980236 A CN 201910980236A CN 112668033 B CN112668033 B CN 112668033B
- Authority
- CN
- China
- Prior art keywords
- data
- file
- compressed
- target database
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000007906 compression Methods 0.000 claims abstract description 47
- 230000006835 compression Effects 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000012545 processing Methods 0.000 claims abstract description 19
- 238000013507 mapping Methods 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000006837 decompression Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 238000011156 evaluation Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 238000013144 data compression Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000000547 structure data Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure provides a data processing method, a device and electronic equipment, belonging to the technical field of data processing, wherein the method comprises the following steps: acquiring write-in data to be stored in a target database; storing the writing data into a data buffer layer of a memory type to obtain buffer data containing the writing data; after the buffer data in the data buffer layer exceeds a preset threshold value, performing compression encryption operation on the buffer data in the buffer layer to obtain compression encryption data; and storing the compressed and encrypted data into a storage file of the target database. The scheme disclosed by the invention can improve the efficiency of data processing.
Description
Technical Field
The disclosure relates to the technical field of data processing, and in particular relates to a data processing method, a data processing device and electronic equipment.
Background
There are two main means of encryption of databases: (1) With a more complex database password verification system, if the database file is stolen, the database file can be directly analyzed to acquire the content. (2) The data is encrypted in advance and then stored in the database, and in this way, the data loses the searching capability, and when the database needs to be searched, the data needs to be completely extracted from the database, decompressed and searched again, so that the efficiency is poor.
In addition, the file system can be directly encrypted, the encryption method (such as AES) has relatively poor performance, and can be used as offline encryption, but if the encryption method is used as online service, the encryption method is unacceptable, the encryption method can only be used as data backup, and a large amount of computing resources are consumed.
Disclosure of Invention
In view of the above, embodiments of the present disclosure provide a data processing method, apparatus and electronic device, which at least partially solve the problems in the prior art.
In a first aspect, an embodiment of the present disclosure provides a data processing method, including:
acquiring write-in data to be stored in a target database;
storing the writing data into a data buffer layer of a memory type to obtain buffer data containing the writing data;
after the buffer data in the data buffer layer exceeds a preset threshold value, performing compression encryption operation on the buffer data in the buffer layer to obtain compression encryption data; the compressed encrypted data is encrypted data obtained by executing encryption operation on the second compressed file; the second compressed file is obtained by embedding a data dictionary related to a compression algorithm into the first compressed file; the first compressed file is a compressed file obtained by compressing all buffer data in a buffer layer by adopting a dictionary-based compression algorithm;
and storing the compressed and encrypted data into a storage file of the target database.
According to a specific implementation manner of the embodiment of the present disclosure, the performing a compressed encryption operation on the buffered data in the buffer layer includes:
compressing all buffer data in the buffer layer by adopting a dictionary-based compression algorithm to obtain a first compressed file;
embedding a data dictionary related to the compression algorithm in the first compressed file, so that the compressed file comprises the data dictionary and a second compressed file for all buffered data;
and performing encryption operation on the second compressed file to obtain the compressed and encrypted data.
According to a specific implementation manner of the embodiment of the present disclosure, the performing an encryption operation on the second compressed file to obtain the compressed encrypted data includes:
obtaining a key associated with the encryption from a key generator using a tlns encoder;
performing tANS coding by using the key to obtain a coded file after coding compression, wherein the coded file comprises a coded file generated for the second compressed file and a coded table after encryption by using the key;
and storing the secret key into a KMS, and recording the mapping relation between the secret key and the coded file.
According to a specific implementation manner of the embodiment of the present disclosure, after the storing the compressed encrypted data in the storage file of the target database, the method further includes:
and executing merging operation on the storage files in the target database based on the acquired merging request for the storage files.
According to a specific implementation manner of the embodiment of the present disclosure, the performing a merging operation on the storage file in the target database includes:
obtaining merging keys of all storage files associated with the merging operation through the KMS;
after obtaining source data of a storage file for executing a merging operation, executing the merging operation on the source data to obtain a merging file;
and executing compression encryption operation on the combined file by using the combined key.
According to a specific implementation manner of the embodiment of the present disclosure, after the obtaining, by the KMS, the merging keys of all the storage files associated with the merging operation, the method further includes:
decrypting the coding table in the storage file by using the merging key to obtain a coding table analysis data file;
analyzing the data file by using the coding table to obtain dictionary compressed data segments;
and decompressing the dictionary compressed data segment to obtain the source data of the storage file.
According to a specific implementation manner of the embodiment of the present disclosure, after the storing the compressed encrypted data in the storage file of the target database, the method further includes:
acquiring a retrieval request for a storage file;
performing segmented decompression on the storage file stored in the target database to obtain a segmented decompressed file;
and executing the retrieval operation related to the retrieval request on the segmented decompressed file.
According to a specific implementation manner of an embodiment of the present disclosure, the obtaining write data to be stored in a target database includes:
acquiring a data writing request of an application program aiming at the target database;
and determining the writing data to be stored to the target database based on the data writing request.
In a second aspect, embodiments of the present disclosure provide a data processing apparatus, including:
the acquisition module is used for acquiring the writing data to be stored in the target database;
the first storage module is used for storing the writing data into a data buffer layer of a memory type so as to obtain buffer data containing the writing data;
the execution module is used for executing compression encryption operation on the buffer data in the buffer layer after the buffer data in the data buffer layer exceeds a preset threshold value so as to obtain compression encryption data; the compressed encrypted data is encrypted data obtained by executing encryption operation on the second compressed file; the second compressed file is obtained by embedding a data dictionary related to a compression algorithm into the first compressed file; the first compressed file is a compressed file obtained by compressing all buffer data in a buffer layer by adopting a dictionary-based compression algorithm;
and the second storage module is used for storing the compressed and encrypted data into a storage file of the target database.
In a third aspect, embodiments of the present disclosure further provide an electronic device, including:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of the implementations of the Ren Di or first aspect described above.
In a fourth aspect, embodiments of the present disclosure also provide a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the data processing method of the first aspect or any implementation manner of the first aspect.
In a fifth aspect, embodiments of the present disclosure also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the data processing method of the first aspect or any implementation of the first aspect.
The data processing scheme in the embodiment of the disclosure comprises the steps of obtaining write-in data to be stored in a target database; storing the writing data into a data buffer layer of a memory type to obtain buffer data containing the writing data; after the buffer data in the data buffer layer exceeds a preset threshold value, performing compression encryption operation on the buffer data in the buffer layer to obtain compression encryption data; and storing the compressed and encrypted data into a storage file of the target database. By the scheme, the efficiency of data processing in the database can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.
FIG. 1 is a schematic diagram of a data processing flow provided in an embodiment of the disclosure;
FIG. 2 is a schematic diagram of a data processing process according to an embodiment of the disclosure;
FIG. 3 is a schematic diagram of another data processing flow provided in an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of another data processing flow provided in an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 6 is a schematic diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present disclosure will become readily apparent to those skilled in the art from the following disclosure, which describes embodiments of the present disclosure by way of specific examples. It will be apparent that the described embodiments are merely some, but not all embodiments of the present disclosure. The disclosure may be embodied or practiced in other different specific embodiments, and details within the subject specification may be modified or changed from various points of view and applications without departing from the spirit of the disclosure. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure are intended to be within the scope of this disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
It should also be noted that the illustrations provided in the following embodiments merely illustrate the basic concepts of the disclosure by way of illustration, and only the components related to the disclosure are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided in order to provide a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
The embodiment of the disclosure provides a data processing method. The data processing method provided in this embodiment may be performed by a computing device, which may be implemented as software, or as a combination of software and hardware, and the computing device may be integrally provided in a server, a terminal device, or the like.
Referring to fig. 1 and 2, a data processing method provided in an embodiment of the disclosure includes the following steps:
s101, obtaining the writing data to be stored in the target database.
The target database is a database capable of performing storage operation, and as an application scene, the target database can be a MyRocks database, and other business providing services of MyRocks. MyRocks is a novel database of Facebook developed by using a MySQL database and a RocksDB storage engine, wherein the upper layer of the database uses a MySQL protocol to provide services for users, and the lower layer uses the RocksDB for data storage and reading.
The target database is connected with an external application program through a communication interface, and the external application program can generate writing data to be stored, so that the writing data to be stored in the target database can be acquired based on the communication interface of the target database.
S102, storing the writing data into a data buffer layer of a memory type to obtain buffer data containing the writing data.
The data in the target database is generally stored in a nonvolatile memory (e.g., a hard disk), and in order to improve the storage efficiency of the target memory, a memory type data buffer layer is provided, and the data buffer layer is provided in the nonvolatile memory (e.g., a DRAM), so that the volatile memory can have a faster read/write speed, and the read/write speed of the write data in the target database can be improved.
As an application scenario, a storage table structure (MemTable) may be set before the target database, and data of the structure type is stored as write data to a buffer layer before the target database, where the structure data belongs to a pure memory structure, so as to improve the read-write speed of the data.
S103, after the buffer data in the data buffer layer exceeds a preset threshold, performing compression encryption operation on the buffer data in the buffer layer to obtain compression encryption data.
The data buffer layer has a certain storage capacity, and after the data in the data buffer layer reaches a threshold value, the data can be compressed and encrypted by the writing data stored in the data buffer layer.
Firstly, compressing all data in a data buffer layer for the first time by adopting a dictionary-based compression algorithm, and then embedding a data dictionary into a file, namely, the compressed data comprises compressed data and the dictionary itself.
The compressed data may then be encoded based on the tANS encoding. Specifically, the tANS encoder first obtains a key from the key generator, where the key corresponds to a password, typically a random string of 256 lengths, or may be generated by a KMS (key management system) using other methods.
After the key is taken, tANS coding is carried out by using the key, data encryption is carried out by using the key in the coding process, two sections of data are obtained, one section is the data after coding compression, and the other section is a coding table (coding table) after the key encryption is used
And finally, storing the secret key into a KMS system, and recording the mapping relation between the secret key and the file.
S104, storing the compressed and encrypted data into a storage file of the target database.
The storage files stored in the target database are stored in the format of SST files, which are data structures used by the target database RocksDB. The final compressed and encrypted data file itself becomes part of the SST.
The scheme in the embodiment combines an online database (such as MyRocks) with a compression encryption algorithm, so that the efficiency of data encryption in a target database is improved. Meanwhile, dictionary compression is firstly carried out on the data in the storage target database, and then tANS encryption compression is carried out, so that the data compression efficiency is improved.
The disclosure may be used in a RocksDB, where data is encrypted before being written to a file system, extracted from the file system, and decrypted.
Referring to fig. 3, according to a specific implementation of an embodiment of the disclosure, the performing a compressed encryption operation on the buffered data in the buffer layer includes:
s301, compressing all buffer data in the buffer layer by adopting a dictionary-based compression algorithm to obtain a first compressed file.
The data buffer layer has a certain storage capacity, and after the data in the data buffer layer reaches a threshold value, the data can be compressed and encrypted by the writing data stored in the data buffer layer.
All data in the data buffer layer is first compressed by adopting a dictionary-based compression algorithm, which is a common algorithm in the field, and the dictionary-based compression algorithm is not limited by the scheme of the present disclosure.
S302, embedding a data dictionary related to the compression algorithm in the first compressed file, so that the compressed file comprises the data dictionary and a second compressed file of all buffered data.
S302, performing encryption operation on the second compressed file to obtain the compressed and encrypted data.
Specifically, a tls encoder may be used to obtain a key associated with encryption from a key generator, to perform tls encoding using the key to obtain an encoded compressed encoded file, where the encoded file includes an encoded file generated for the second compressed file, and an encoded table encrypted using the key. And finally, storing the secret key into a KMS, and recording the mapping relation between the secret key and the coded file. By the scheme of the embodiment, the encryption operation can be performed while the file is compressed.
According to a specific implementation manner of the embodiment of the present disclosure, after the storing the compressed encrypted data in the storage file of the target database, the method further includes: and executing merging operation on the storage files in the target database based on the acquired merging request for the storage files.
Referring to fig. 4, according to a specific implementation manner of the embodiment of the present disclosure, the performing a merging operation on a storage file in the target database includes:
s401, obtaining the merging keys of all the storage files associated with the merging operation through the KMS.
And finding out the merging keys of all the storage files associated with the merging operation according to the one-to-one correspondence between the storage files and the keys through the KMS.
S402, after obtaining source data of a storage file for executing the merging operation, executing the merging operation on the source data to obtain a merged file.
The merging of the files can be performed in a conventional file merging manner, and after merging, a merged file is formed.
S403, performing compression encryption operation on the combined file by using the combined key.
The dictionary-based compression may be performed on the combined file in a similar manner to step S103, and then the encryption operation may be performed by the combined key, thereby completing the compression encryption operation of the combined file.
By the way in the above embodiment, the merging operation can be performed on the data in the storage file.
According to a specific implementation manner of the embodiment of the present disclosure, after the obtaining, by the KMS, the merging keys of all the storage files associated with the merging operation, the method further includes: decrypting the coding table in the storage file by using the merging key to obtain a coding table analysis data file; analyzing the data file by using the coding table to obtain dictionary compressed data segments; and decompressing the dictionary compressed data segment to obtain the source data of the storage file.
According to a specific implementation manner of the embodiment of the present disclosure, after the storing the compressed encrypted data in the storage file of the target database, the method further includes: acquiring a retrieval request for a storage file; performing segmented decompression on the storage file stored in the target database to obtain a segmented decompressed file; and executing the retrieval operation related to the retrieval request on the segmented decompressed file. By means of the method in the embodiment, the stored content in the database can be quickly searched.
According to a specific implementation manner of an embodiment of the present disclosure, the obtaining write data to be stored in a target database includes: acquiring a data writing request of an application program aiming at the target database; and determining the writing data to be stored to the target database based on the data writing request. By the way in this embodiment, an operation of writing data can be performed based on a write request.
Corresponding to the above method embodiment, referring to fig. 5, the disclosed embodiment further provides a data processing apparatus 50, including:
the obtaining module 501 is configured to obtain write data to be stored in the target database.
The target database is a database capable of performing storage operation, and as an application scene, the target database can be a MyRocks database, and other business providing services of MyRocks. MyRocks is a novel database of Facebook developed by using a MySQL database and a RocksDB storage engine, wherein the upper layer of the database uses a MySQL protocol to provide services for users, and the lower layer uses the RocksDB for data storage and reading.
The target database is connected with an external application program through a communication interface, and the external application program can generate writing data to be stored, so that the writing data to be stored in the target database can be acquired based on the communication interface of the target database.
The first storage module 502 is configured to store the write data into a data buffer layer of a memory type, so as to obtain buffered data including the write data.
The data in the target database is generally stored in a nonvolatile memory (e.g., a hard disk), and in order to improve the storage efficiency of the target memory, a memory type data buffer layer is provided, and the data buffer layer is provided in the nonvolatile memory (e.g., a DRAM), so that the volatile memory can have a faster read/write speed, and the read/write speed of the write data in the target database can be improved.
As an application scenario, a storage table structure (MemTable) may be set before the target database, and data of the structure type is stored as write data to a buffer layer before the target database, where the structure data belongs to a pure memory structure, so as to improve the read-write speed of the data.
And the execution module 503 is configured to execute a compressed encryption operation on the buffered data in the data buffer layer after the buffered data in the data buffer layer exceeds a preset threshold value, so as to obtain compressed encrypted data. The compressed encrypted data is encrypted data obtained by executing encryption operation on the second compressed file; the second compressed file is obtained by embedding a data dictionary related to a compression algorithm into the first compressed file; the first compressed file is a compressed file obtained by compressing all buffer data in the buffer layer by adopting a dictionary-based compression algorithm.
The data buffer layer has a certain storage capacity, and after the data in the data buffer layer reaches a threshold value, the data can be compressed and encrypted by the writing data stored in the data buffer layer.
Firstly, compressing all data in a data buffer layer for the first time by adopting a dictionary-based compression algorithm, and then embedding a data dictionary into a file, namely, the compressed data comprises compressed data and the dictionary itself.
The compressed data may then be encoded based on the tANS encoding. Specifically, the tANS encoder first obtains a key from the key generator, where the key corresponds to a password, typically a random string of 256 lengths, or may be generated by a KMS (key management system) using other methods.
After the key is taken, tANS coding is carried out by using the key, data encryption is carried out by using the key in the coding process, two sections of data are obtained, one section is the data after coding compression, and the other section is a coding table (coding table) after the key encryption is used
And finally, storing the secret key into a KMS system, and recording the mapping relation between the secret key and the file.
A second storage module 504, configured to store the compressed encrypted data into a storage file of the target database.
The files stored in the target database are stored in the format of SST files, which are data structures used by the target database RocksDB. The final compressed and encrypted data file itself becomes part of the SST.
According to the scheme in the embodiment, an encryption database system is built based on the MyRocks and tANS algorithms of the target database, and the online database (namely MyRocks) and the compression encryption algorithm are combined, so that the efficiency of data encryption in the target database is improved. Meanwhile, dictionary compression is firstly carried out on the data in the storage target database, and then tANS encryption compression is carried out, so that the data compression efficiency is improved.
The disclosure may be used in a RocksDB, where data is encrypted before being written to a file system, extracted from the file system, and decrypted.
The apparatus shown in fig. 5 may correspondingly execute the content in the foregoing method embodiment, and the portions not described in detail in this embodiment refer to the content described in the foregoing method embodiment, which are not described herein again.
Referring to fig. 6, an embodiment of the present disclosure also provides an electronic device 60, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method of the foregoing method embodiments.
The disclosed embodiments also provide a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the foregoing method embodiments.
The disclosed embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the data processing method of the foregoing method embodiments.
Referring now to fig. 6, a schematic diagram of an electronic device 60 suitable for use in implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 6 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 6, the electronic device 60 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
In general, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; a communication device 610. The communication means 610 may allow the electronic device 60 to communicate wirelessly or by wire with other devices to exchange data. While an electronic device 60 having various means is shown, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communications device 610, or from storage 608, or from ROM 602. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 601.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects an internet protocol address from the at least two internet protocol addresses and returns the internet protocol address; receiving an Internet protocol address returned by the node evaluation equipment; wherein the acquired internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
The foregoing is merely specific embodiments of the disclosure, but the protection scope of the disclosure is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the disclosure are intended to be covered by the protection scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A method of data processing, comprising:
acquiring write-in data to be stored in a target database;
storing the writing data into a data buffer layer of a memory type to obtain buffer data containing the writing data;
after the buffer data in the data buffer layer exceeds a preset threshold value, performing compression encryption operation on the buffer data in the buffer layer to obtain compression encryption data; the compressed encrypted data is encrypted data obtained by executing encryption operation on the second compressed file; the second compressed file is obtained by embedding a data dictionary related to a compression algorithm into the first compressed file; the first compressed file is a compressed file obtained by compressing all buffer data in a buffer layer by adopting a dictionary-based compression algorithm;
and storing the compressed and encrypted data into a storage file of the target database.
2. The method of claim 1, wherein the compressed encrypted data is obtained by:
obtaining a key associated with the encryption from a key generator using a tlns encoder;
performing tANS coding by using the key to obtain a coded file after coding compression, wherein the coded file comprises a coded file generated for the second compressed file and a coded table after encryption by using the key;
and storing the secret key into a KMS, and recording the mapping relation between the secret key and the coded file.
3. The method of claim 2, wherein after storing the compressed encrypted data in the storage file of the target database, the method further comprises:
and executing merging operation on the storage files in the target database based on the acquired merging request for the storage files.
4. The method of claim 3, wherein the performing a merge operation on the stored files in the target database comprises:
obtaining merging keys of all storage files associated with the merging operation through the KMS;
after obtaining source data of a storage file for executing a merging operation, executing the merging operation on the source data to obtain a merging file;
and executing compression encryption operation on the combined file by using the combined key.
5. The method of claim 4, wherein after the obtaining, by the KMS, the merge keys for all stored files associated with the merge operation, the method further comprises:
decrypting the coding table in the storage file by using the merging key to obtain a coding table analysis data file;
analyzing the data file by using the coding table to obtain dictionary compressed data segments;
and decompressing the dictionary compressed data segment to obtain the source data of the storage file.
6. The method of claim 1, wherein after storing the compressed encrypted data in the storage file of the target database, the method further comprises:
acquiring a retrieval request for a storage file;
performing segmented decompression on the storage file stored in the target database to obtain a segmented decompressed file;
and executing the retrieval operation related to the retrieval request on the segmented decompressed file.
7. The method of claim 1, wherein the obtaining write data to be stored to the target database comprises:
acquiring a data writing request of an application program aiming at the target database;
and determining the writing data to be stored to the target database based on the data writing request.
8. A data processing apparatus, comprising:
the acquisition module is used for acquiring the writing data to be stored in the target database;
the first storage module is used for storing the writing data into a data buffer layer of a memory type so as to obtain buffer data containing the writing data;
the execution module is used for executing compression encryption operation on the buffer data in the buffer layer after the buffer data in the data buffer layer exceeds a preset threshold value so as to obtain compression encryption data; the compressed encrypted data is encrypted data obtained by executing encryption operation on the second compressed file; the second compressed file is obtained by embedding a data dictionary related to a compression algorithm into the first compressed file; the first compressed file is a compressed file obtained by compressing all buffer data in a buffer layer by adopting a dictionary-based compression algorithm;
and the second storage module is used for storing the compressed and encrypted data into a storage file of the target database.
9. An electronic device, the electronic device comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of the preceding claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the data processing method of any preceding claim 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980236.9A CN112668033B (en) | 2019-10-15 | 2019-10-15 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910980236.9A CN112668033B (en) | 2019-10-15 | 2019-10-15 | Data processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112668033A CN112668033A (en) | 2021-04-16 |
| CN112668033B true CN112668033B (en) | 2023-04-25 |
Family
ID=75400034
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910980236.9A Active CN112668033B (en) | 2019-10-15 | 2019-10-15 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112668033B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817181A (en) * | 2022-04-18 | 2022-07-29 | 深圳市汉云科技有限公司 | Storage method and decompression method of structured database and electronic device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891999A (en) * | 2012-09-26 | 2013-01-23 | 南昌大学 | Combined image compression/encryption method based on compressed sensing |
| CN109450450A (en) * | 2018-10-17 | 2019-03-08 | 杭州费尔斯通科技有限公司 | A kind of compression of JSON data real non-destructive and decompressing method |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101730297A (en) * | 2008-10-13 | 2010-06-09 | 北京摩软科技有限公司 | Mobile terminal, method and server for realizing data management |
| US8645713B2 (en) * | 2011-03-22 | 2014-02-04 | Fujitsu Limited | Encrypting method, recording medium of encrypting program, decrypting method, and recording medium of decrypting program |
| CN103457609B (en) * | 2013-08-20 | 2016-04-13 | 许继集团有限公司 | Fault recorder data Lossless Compression, decompression method |
| KR102114388B1 (en) * | 2013-10-18 | 2020-06-05 | 삼성전자주식회사 | Method and apparatus for compressing memory of electronic device |
| US9473418B2 (en) * | 2013-12-12 | 2016-10-18 | International Business Machines Corporation | Resource over-subscription |
| US10169621B2 (en) * | 2014-03-26 | 2019-01-01 | Tivo Solutions Inc. | Multimedia pipeline architecture |
| JP6613568B2 (en) * | 2015-01-19 | 2019-12-04 | 富士通株式会社 | Processing program, processing apparatus, and processing method |
| CN106202172B (en) * | 2016-06-24 | 2019-07-30 | 中国农业银行股份有限公司 | Text compression methods and device |
-
2019
- 2019-10-15 CN CN201910980236.9A patent/CN112668033B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891999A (en) * | 2012-09-26 | 2013-01-23 | 南昌大学 | Combined image compression/encryption method based on compressed sensing |
| CN109450450A (en) * | 2018-10-17 | 2019-03-08 | 杭州费尔斯通科技有限公司 | A kind of compression of JSON data real non-destructive and decompressing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112668033A (en) | 2021-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10958416B2 (en) | Encrypted and compressed data transmission with padding | |
| CN110222775B (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN112149168B (en) | File data encryption method and device and electronic equipment | |
| CN112256275B (en) | Code confusion method, device, electronic equipment and medium | |
| CN111245811A (en) | Information encryption method and device and electronic equipment | |
| CN110545542B (en) | Main control key downloading method and device based on asymmetric encryption algorithm and computer equipment | |
| CN112668033B (en) | Data processing method and device and electronic equipment | |
| CN110908752A (en) | Control setting method and device, electronic equipment and interaction system | |
| CN112905575A (en) | Data acquisition method, system, storage medium and electronic equipment | |
| CN110765238A (en) | Data encryption query method and device | |
| CN111143355B (en) | Data processing method and device | |
| US9654140B1 (en) | Multi-dimensional run-length encoding | |
| CN113742663B (en) | Watermark file acquisition method and device and electronic equipment | |
| CN111240657A (en) | Method and device for generating SDK and electronic equipment | |
| CN110568994B (en) | Logic block device access method and device and electronic device | |
| CN113778387B (en) | Method and device for generating code | |
| CN110968265A (en) | Fragmentation expansion method and device and electronic equipment | |
| CN112214549B (en) | File feature code generation method and device and electronic equipment | |
| CN111639055B (en) | Differential packet calculation method, differential packet calculation device, differential packet calculation equipment and storage medium | |
| CN112732269B (en) | Access method and device applied to physical address of AVS2 hardware encoder | |
| CN113032808B (en) | Data processing method and device, readable medium and electronic equipment | |
| CN113742774B (en) | Data processing method and device, readable medium and electronic equipment | |
| CN115221492B (en) | Authentication method and device based on hardware key, electronic equipment and storage medium | |
| CN113347278B (en) | Data processing method and device and electronic equipment | |
| CN110263797B (en) | Method, device and equipment for estimating key points of skeleton and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |