CN110147201A - Line compression method, apparatus, computer equipment and storage medium - Google Patents
Line compression method, apparatus, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110147201A CN110147201A CN201910337896.5A CN201910337896A CN110147201A CN 110147201 A CN110147201 A CN 110147201A CN 201910337896 A CN201910337896 A CN 201910337896A CN 110147201 A CN110147201 A CN 110147201A
- Authority
- CN
- China
- Prior art keywords
- data
- compressed
- downloading
- compression
- compressed data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of line compression method, apparatus, computer equipment and storage mediums, wherein, the line compression method includes the following steps that major-minor book server executes: obtaining the data write request that client is sent, data write request includes initial data；Initial data is compressed, obtains compressed data, and compressed data is written in object storage device, compressed data carries compression tag and reduction length；Corresponding initial data attribute is obtained based on initial data；Compressed data, compression tag, reduction length and initial data attribute transfer are given from replica server using individual thread, cost squeeze is saved, improves compression efficiency.
Description
Technical field
The present invention relates to computer development field more particularly to a kind of line compression method, apparatus, computer equipment and deposit
Storage media.
Background technique
In distributed storage, in order to improve the reliability of data, often using more copies (including primary copy and more
It is a from copy) method, an object/file is stored in multiple places in the corresponding cluster of more copies, such as: ceph (point
Cloth storage system) in default using three copies (primary copy and two from copy) scheme.But more copy storages increase
The occupancy of memory space also increases business data carrying cost.In order to reduce data carrying cost, accounting for for memory space is reduced
With many distributed storages provide compression function, as ceph introduces bluestore in J version and provides line compression function
Can, but each copy data being written will compress once in the memory node where respective copy, increase cost squeeze, prolong
Compression time has been grown, compression efficiency is reduced.
Summary of the invention
The embodiment of the present invention provides a kind of line compression method, apparatus, computer equipment and storage medium, to solve copy
Data, which will carry out compression in the memory node where respective copy, leads to the problem that cost squeeze is big and compression efficiency is lower.
A kind of line compression method is applied to major-minor book server, comprising:
The data write request that client is sent is obtained, the data write request includes initial data；
The initial data is compressed, obtains compressed data, and object storage device is written into the compressed data
In, the compressed data carries compression tag and reduction length；
Corresponding initial data attribute is obtained based on the initial data；
Using individual thread by the compressed data, the compression tag, the reduction length and the initial data
Attribute transfer is given from replica server.
A kind of line compression device, including major-minor book server, the major-minor book server include:
Write request obtains module, for obtaining the data write request of client transmission, the data write request packet
Include initial data；
Data compressing module obtains compressed data for compressing to the initial data, and by the compressed data
It is written in object storage device, the compressed data carries compression tag and reduction length；
Data attribute obtains module, for obtaining corresponding initial data attribute based on the initial data；
Data transmission module, for being grown the compressed data, the compression tag, the compression using individual thread
Degree and the initial data attribute transfer are given from replica server.
A kind of computer equipment, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, the processor realize above-mentioned line compression method when executing the computer program.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter
Calculation machine program realizes above-mentioned line compression method when being executed by processor.
A kind of line compression method is applied to from replica server, comprising:
Obtain compressed data, the compression tag, reduction length and original that major-minor book server is sent by individual thread
Data attribute；
If the compression tag be it is true, the physical length of the compressed data is obtained, by the physical length and described
Reduction length is compared；
If the physical length and reduction length are equal, then it represents that the compressed data receives successfully；
The compressed data is written in corresponding physical disks, and according to the initial data attribute more new metadata, it will
Object logics address and physical address in the metadata are mapped.
A kind of line compression device, including from replica server, it is described to include: from replica server
Data acquisition module, for obtaining compressed data, the compression mark that major-minor book server is sent by individual thread
Label, reduction length and initial data attribute；
Data processing module, if being very, the physical length of the compressed data to be obtained, by institute for the compression tag
It states physical length and the reduction length is compared；
Data judgment module, if equal for the physical length and reduction length, then it represents that the compressed data receives
Success；
Data memory module, for the compressed data to be written in corresponding physical disks, and according to the initial data
Attribute more new metadata, by the metadata object logics address and physical address be mapped.
A kind of computer equipment, including memory, processor and storage are in memory and can be on the processor
The computer program of operation, the processor realize above-mentioned line compression method when executing the computer program.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter
Calculation machine program realizes above-mentioned line compression method when being executed by processor.
Above-mentioned line compression method, apparatus, computer equipment and storage medium, major-minor book server are asked by data write-in
Acquisition initial data is sought, then initial data is compressed, between acquisition compressed data.After obtaining compressed data, primary copy
Compressed data, compression tag, reduction length and initial data attribute are transferred to by individual thread from copy and are taken by server
Be engaged in device, from replica server by judge whether compression tag is true, it is determined whether needs to the compressed data of acquisition again into
Row compression, if compression tag is very, not need to compress compressed data again from replica server.Due to network transmission
Uncertainty, determine compression tag be very after, it is also necessary to reduction length and physical length are compared, if physical length
It is equal with reduction length, then it represents that compressed data receives successfully, compressed data is written in corresponding physical disks, and according to original
Data attribute more new metadata, so that compressed data is still having enough physical spaces to carry out after decompressing in replica server
Cost squeeze is saved in storage, improves compression efficiency.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without any creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is the application environment schematic diagram of line compression method in one embodiment of the invention；
Fig. 2 is the flow chart of line compression method in one embodiment of the invention；
Fig. 3 is a specific flow chart of step S16 in Fig. 2；
Fig. 4 is another specific flow chart of step S16 in Fig. 2；
Fig. 5 is the schematic diagram of line compression device in one embodiment of the invention；
Fig. 6 is the schematic diagram of computer equipment in one embodiment of the invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
Line compression method provided by the present application, can be applicable in the application environment such as Fig. 1, and client passes through network and master
It replica server and is communicated from replica server.Wherein, client includes but is not limited to various personal computers, notebook
Computer, smart phone, tablet computer and portable wearable device.Major-minor book server refers in distributed storage, for depositing
Store up the server of the primary copy of data.Refer in distributed storage from replica server, the clothes of slave copy for storing data
Business device.In distributed storage, default primary copy be one, from copy be it is multiple, therefore, the major-minor book server of the application can
To be realized with independent server, can be realized from replica server with the server cluster that multiple servers form.
A kind of line compression method is applied to major-minor book server, comprising:
The data write request that client is sent is obtained, data write request includes initial data；
Initial data is compressed, compressed data is obtained, and compressed data is written in object storage device, compresses number
According to carrying compression tag and reduction length；
Corresponding initial data attribute is obtained based on initial data；
Compressed data, compression tag, reduction length and initial data attribute transfer are given from copy using individual thread
Server.
A kind of line compression method is applied to from replica server, comprising:
Obtain compressed data, the compression tag, reduction length and original that major-minor book server is sent by individual thread
Data attribute；
If compression tag is very, to obtain the physical length of compressed data, physical length and reduction length are compared；
If physical length and reduction length are equal, then it represents that compressed data receives successfully；
Compressed data is written in corresponding physical disks, and according to initial data attribute more new metadata, it will be in metadata
Object logics address and physical address be mapped.
In one embodiment, as shown in Fig. 2, providing a kind of line compression method, the major-minor in Fig. 1 is applied in this way
It book server and is illustrated, includes the following steps: for replica server
S11: major-minor book server obtains the data write request that client is sent, and data write request includes original number
According to.
Wherein, data write request refers to the request being written to initial data in the hard disk of major-minor book server.Original number
According to the data for referring to that client issues, which is the data handle that user passes on the client.
Specifically, user inputs initial data in client, is then sent to initial data by data write request
Major-minor book server, so that major-minor book server executes the subsequent processing steps to initial data.
S12: major-minor book server compresses initial data, obtains compressed data, and object is written in compressed data
It stores in equipment, compressed data carries compression tag and reduction length.
Object storage device used in the present embodiment is OSD (Object Storage Device), and OSD is a kind of base
In the equipment of object storage technology.Object storage (Object-based Storage) is a kind of new Network Storage Structure,
Component part includes the client of object, object storage device, meta data server and object storage system.
Specifically, major-minor book server can carry out initial data based on data write request after obtaining initial data
Compression obtains compressed data.Wherein, compressed data refers to the data obtained after initial data is compressed.Obtaining initial data
After corresponding compressed data, which carries compression tag and reduction length.Wherein, reduction length refers to initial data quilt
Compressed data length.Compression tag refers to the compression tag for indicating the compressed label of initial data, in the present embodiment
Including true and false, when compression tag is true, compressing original data success is indicated；When compression tag is fictitious time, expression original number
Fail according to compression, identifies whether initial data compresses success by compression tag fast and easy.
S13: major-minor book server is based on initial data and obtains corresponding initial data attribute.
Wherein, initial data attribute refers to the corresponding attribute of initial data, offset (offset) including initial data and
Length (length).Specifically, initial data carries its corresponding initial data attribute,
Major-minor book server can obtain initial data attribute based on initial data, be subsequent step after obtaining initial data
It is rapid that data source is provided.
S14: major-minor book server uses individual thread by compressed data, compression tag, reduction length and initial data
Attribute transfer is given from replica server.
Individual thread in the present embodiment refers to major-minor book server to the thread from replica server transmission information.It needs
Illustrate, the slave replica server in the present embodiment includes individual threads concrete applications that are multiple, creating for the ease of differentiation
In which from replica server, each there is Thread Id from the corresponding individual thread of replica server.
Specifically, major-minor book server can obtain corresponding individual thread according to Thread Id after obtaining compressed data,
It is corresponding from replica server that compressed data, compression tag, reduction length and initial data attribute are sent to Thread Id, opened up
Individual thread transmits data to from replica server for major-minor book server, effectively improves efficiency of transmission.
In step S11- step S14, major-minor book server obtains initial data by data write request, then to original
Data are compressed, and compressed data are obtained, to save data space.After obtaining compressed data, major-minor book server will
Compressed data, compression tag, reduction length and initial data attribute are transferred to by individual thread from replica server, to mention
High-transmission efficiency.
S21: compressed data, the compression mark that major-minor book server is sent by individual thread are obtained from replica server
Label, reduction length and initial data attribute.
S22: if compression tag be it is true, the physical length of compressed data is obtained from replica server, by physical length and
Reduction length is compared.
Specifically, compressed data, the pressure that major-minor book server is sent by individual thread are got from replica server
After contracting label and initial data attribute, judge compressed data whether by compressed data according to compression tag.Work as compression
When label is true, then it represents that compressed data is the data of compression of having succeeded, and does not need second compression again, then obtains compressed data
Physical length and reduction length are compared by physical length.Wherein, physical length, which refers to from replica server, is an actually-received pressure
The length of contracting data.
Further, when compression tag is false, then it represents that compressed data is the data of not successful compression, is pressed
Contracting.
S23: if physical length and reduction length are equal, then it represents that received successfully from the compressed data of replica server.
Specifically, due to the uncertainty of network transmission, compressed data is passing through data flow from primary copy server transport
To during the replica server it is possible that the case where packet loss.When there is the case where packet loss, obtained from replica server
The reduction length for the compressed data that the physical length of the compressed data taken will be sent with major-minor book server is unequal.Therefore,
After the compression tag for judging that compressed data carries from replica server is very, it is also necessary to the compression sent to major-minor book server
The physical length of length and the compressed data received from replica server is compared, if the equal length of the two, then it represents that
Compressed data receives successfully.Wherein, data flow (data stream) refers to a kind of digital coding letter formed in data transmission procedure
Number sequence.
S24: being written in corresponding physical disks from replica server by compressed data, and according to initial data attribute more Singapore dollar
Data, by metadata object logics address and physical address be mapped.
Wherein, metadata is a kind of data (data about data) for describing initial data, is equivalent to a kind of electronic type
Catalogue, is mainly used for the information of record description initial data attribute (property), including but not limited to the title of initial data,
Status informations and object logics address and the physics such as date created, modification date, content-length, version number and storage classification
The mapping relations of address.
Specifically, after corresponding physical disks are written in the compressed data that will acquire from replica server, according to initial data
The initial data such as offset and length attribute more new metadata, object logics address and physical address are mapped, so that
Compressed data still has enough physical spaces to be stored after decompression.Wherein, physical address refers to that being loaded into memory address posts
Address in storage, the real address of internal storage location.Object logics address refers to the address that compressed data generates in CPU.
In step S21- step S24, from replica server by judging whether compression tag is true, it is determined whether needs pair
The compressed data of acquisition is compressed again, if compression tag is very, not needed from replica server to compressed data again
It is compressed, saves cost squeeze, improve compression efficiency.Due to the uncertainty of network transmission, determining that compression tag is true
Afterwards, it is also necessary to reduction length and physical length are compared, if physical length and reduction length are equal, then it represents that compressed data
It receives successfully, compressed data is written in corresponding physical disks, and according to initial data attribute more new metadata, so that compression number
According to from replica server decompress after still thering are enough physical spaces to be stored.
The embodiment that step S11- step S24 is provided, major-minor book server obtain initial data by data write request,
Then initial data is compressed, between acquisition compressed data.After obtaining compressed data, major-minor book server will compress number
According to, compression tag, reduction length and initial data attribute be transferred to from replica server by individual thread, from Copy Service
Device is by judging whether compression tag is true, it is determined whether needs compress the compressed data of acquisition again, if compression mark
Label are very, then not need to compress compressed data again from replica server, save cost squeeze.Due to network transmission
Uncertainty, determine compression tag be very after, it is also necessary to reduction length and physical length are compared, if physical length and
Reduction length is equal, then it represents that compressed data receives successfully, compressed data is written in corresponding physical disks, and according to original number
According to attribute more new metadata, so that compressed data is still having enough physical spaces to be deposited after decompressing in replica server
Storage, by the true and false of compression tag, judges whether compressed from replica server, improves compression efficiency.
In one embodiment, as shown in Fig. 2, in step S14, after object storage device is written in compressed data, online
Compression method further includes following steps:
S15: major-minor book server obtains the data download request that client is sent, and data download request includes that downloading is crucial
Word.
Wherein, data download request refers to the request for the downloading data that user is sent by client.
Specifically, due to the data download request that user sends in client, it can be both sent to major-minor book server,
It can be sent to from replica server, therefore, user needs to select primary copy service when client sends data download request
Device or from replica server, is major-minor book server with determination or makes sound to the data download request from replica server
It answers.What is selected in the present embodiment is major-minor book server.Wherein, which includes downloading keyword, the present embodiment
In downloading keyword include but is not limited to data downloading title and data version number.Data downloading title refers to that user needs to download
Data title.Versions of data number refers to the version number for the data that user needs to download.
S16: major-minor book server is based on downloading keyword query object storage device, will be corresponding with downloading keyword
Compressed data is determined as target data, and target data is sent to client by data download request.
Wherein, target data refers to compressed data corresponding with downloading keyword.Specifically, major-minor book server is being got
, can be based on downloading keyword query object storage device after downloading keyword, it will compressed data corresponding with downloading keyword
It is determined as target data, and target data is sent to client by data download request.
In step S15- step S16, major-minor book server passes through the downloading keyword query object in data download request
Equipment is stored, is passed through target data after obtaining target data based on downloading keyword quick obtaining target data with realizing
Data download request is sent to client, with the response of complete paired data downloading request.
It in one embodiment,, will be with based on downloading keyword query object storage device as shown in figure 3, in step S16
The corresponding compressed data of downloading keyword is determined as target data, specifically comprises the following steps:
S161: it if downloading keyword only includes data downloading title, is set based on data downloading name query object storage
It is standby, compressed data corresponding with data downloading title is determined as target data.
Specifically, if only including that data download title in the data key words that major-minor book server obtains, data are based on
Name query object storage device is downloaded, all compressed datas corresponding with data downloading title are determined as target data.
Wherein, downloading title refers to the title for the data that user downloads in the needs that client is filled in.In order to meet the different demands of user,
Downloading title in the present embodiment is either one or more.What is inputted in client as user is the downloading of data
Title, major-minor book server can download name query object storage device based on a data, will download title phase with the data
Corresponding compressed data is determined as target data；When user client input be to a data download title, primary copy clothes
Being engaged in device can be based on multiple data downloading name query object storage device, will compression corresponding with each data downloading title
Data are respectively as target data.
S162: if downloading keyword includes data downloading title and data version number, title sum number is downloaded based on data
Equipment is stored according to version number's query object, compressed data corresponding with data downloading title and data version number is determined as mesh
Mark data.
Specifically, if in the data key words that major-minor book server obtains including data downloading title and data version number,
Equipment is then stored based on data downloading title and data version number query object, title and data version number phase will be downloaded with data
Corresponding compressed data is determined as target data.It should be noted that the downloading keyword in the present embodiment may include one
Data download title, also may include multiple data downloading titles, may include a data version number, also may include multiple
Versions of data number.
In step S161- step S162, user only needs that downloading keyword is arranged in client, and major-minor book server will
Target data is obtained from object storage device by downloading keyword, can preferably meet user demand.
In one embodiment, as shown in figure 4, in step S16, i.e., target data is sent to visitor by data download request
Family end, specifically comprises the following steps:
S163: the quantity of target data is counted.
Specifically, if downloading keyword only includes data downloading title, major-minor book server just downloads title according to data
Will all compressed datas corresponding with the downloading title as target data.It is to be appreciated that data downloading title can be with
A corresponding compressed data, can also correspond to multiple compressed datas.If downloading keyword includes data downloading title and data version
This number, then equipment is stored based on data downloading title and data version number query object, title and data version will be downloaded with data
This number corresponding compressed data is determined as target data.
The present embodiment is not identical to the processing step of a target data and multiple target datas, therefore, is obtaining target
After data, major-minor book server needs to count the quantity of target data, to determine the subsequent execution step of target data.
S164: if the quantity of target data is one, target data is directly sent to visitor by data download request
Family end.
Specifically, if the quantity of target data is one, major-minor book server is directly downloaded target data by data
Request is sent to client.
S165: if the quantity of target data is two or more, carrying out automatic compression processing to target data, and
Compressed target data is sent to client by data download request.
Specifically, if the quantity of target data is two or more, using automatic compression sentence to target data
Automatically it is compressed, forms targeted compression packet, and the targeted compression packet is sent to corresponding client by data download request
The function that batch downloads target data may be implemented in end, download time is saved, furthermore it is possible to which it is independent to avoid the occurrence of target data
Occur partial target data when downloading to download successfully, partially downloads unsuccessful situation.Wherein, targeted compression packet refers to target data
The compressed package formed after compression.
In step S163- step S165, if the quantity of target data is one, major-minor book server leads to target data
It crosses data download request and is transmitted directly to client；If the quantity of target data is two or more, primary copy service
Device needs to carry out automatic compression processing to target data, forms targeted compression packet, is then sent to visitor by data download request
The function that batch downloads target data may be implemented in family end, saves download time.
In one embodiment, as shown in Fig. 2, in step S24, compressed data is written in corresponding physical disks, and according to
Initial data attribute more new metadata, it is online to press by the object logics address in metadata and after physical address is mapped
Contracting method further includes following steps:
S25: the data download request that client is sent is obtained from replica server, data download request includes that downloading is crucial
Word.
It specifically, will from replica server when the data download request of client is sent to from replica server
The data download request is responded, the downloading keyword in the data download request is obtained.
S26: being based on downloading keyword query physical disks from replica server, will compression number corresponding with downloading keyword
According to being determined as target data, and target data is sent to client by data download request.
Specifically, it from replica server after obtaining downloading keyword, will be obtained according to downloading keyword query physical disks
Take compressed data corresponding with downloading keyword as target data.
If downloading keyword only includes data downloading title, name query physical disks are downloaded based on data, it will be with data
The corresponding compressed data of downloading title is determined as target data；If downloading keyword includes data downloading title and versions of data
Number, then physical disks are inquired based on data downloading title and data version number, title will be downloaded with data and data version number is opposite
The compressed data answered is determined as target data.
After obtaining target data, the quantity of target data can be counted from replica server.If the quantity of target data is
One, then target data is directly sent to client by data download request；If the quantity of target data is two or two
More than a, then automatic compression processing is carried out to target data, and compressed target data is sent by data download request
To client.
In step S25- step S26, pass through the downloading keyword query physics in data download request from replica server
Disk counts the quantity of target data after obtaining target data based on downloading keyword quick obtaining target data to realize.
If the quantity of target data is one, target data is transmitted directly to client by data download request；If number of targets
According to quantity be two or more, then need to carry out automatic compression processing to target data, form targeted compression packet, then
Being sent to client by data download request may be implemented the function that batch downloads target data, save download time.
Line compression method provided by the invention, major-minor book server obtain initial data by data write request, so
Initial data is compressed afterwards, compressed data is obtained, to save data space.After obtaining compressed data, primary copy
Compressed data, compression tag, reduction length and initial data attribute are transferred to by individual thread from copy and are taken by server
Business device, to improve efficiency of transmission.By the downloading keyword query object storage device in data download request, it is based on realizing
Download keyword quick obtaining target data.After obtaining target data, if the quantity of target data is one, by number of targets
Client is transmitted directly to according to by data download request；If the quantity of target data is two or more, need pair
Target data carries out automatic compression processing, forms targeted compression packet, and being then sent to client by data download request can be with
It realizes that batch downloads the function of target data, saves download time.
From replica server by judge whether compression tag is that very, determination need to not needed to the compressed data of acquisition again
It is compressed, if compression tag is very, not need to compress compressed data again from replica server, saving is compressed into
This.Due to the uncertainty of network transmission, determine compression tag be very after, it is also necessary to reduction length and physical length are carried out
Compare, if physical length and reduction length are equal, then it represents that compressed data receives successfully, and corresponding physics is written in compressed data
In disk, and according to initial data attribute more new metadata, so that compressed data is still having foot after decompressing in replica server
Enough physical spaces are stored.By the downloading keyword query physical disks in data download request, to realize based on downloading
Keyword quick obtaining target data counts the quantity of target data after obtaining target data, if the quantity of target data is
One, then target data is transmitted directly to client by data download request；If the quantity of target data is two or two
More than a, then need to carry out automatic compression processing to target data, form targeted compression packet, then sent out by data download request
Giving client may be implemented the function that batch downloads target data, save download time.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
In one embodiment, a kind of line compression device is provided, which presses online with above-described embodiment
Contracting method corresponds.As shown in figure 5, the line compression device includes major-minor book server 10, the major-minor book server 10 packet
It includes write request and obtains module 11, data compressing module 12, data attribute acquisition module 13 and data transmission module 14.Each function
Detailed description are as follows for module:
Write request obtains module 11, and for obtaining the data write request of client transmission, data write request includes
Initial data.
Data compressing module 12 obtains compressed data for compressing to initial data, and by compressed data write-in pair
As in storage equipment, compressed data carries compression tag and reduction length.
Data attribute obtains module 13, for obtaining corresponding initial data attribute based on initial data.
Data transmission module 14, for using individual thread by compressed data, compression tag, reduction length and original number
It gives according to attribute transfer from replica server.
Further, which further includes at the first downloading request module 15 and first object data
Manage module 16.
First downloading request module 15, for obtaining the data download request of client transmission, data download request
Including downloading keyword.
First object data processing module 16, for that will be closed with downloading based on downloading keyword query object storage device
The corresponding compressed data of key word is determined as target data, and target data is sent to client by data download request.
Further, target data processing module includes that first object data capture unit and the second target data obtain list
Member.
First object data capture unit, if only including data downloading title for downloading keyword, based under data
Name query object storage device is carried, compressed data corresponding with data downloading title is determined as target data.
Second target data acquiring unit, if including that data download title and data version number for downloading keyword,
Title is downloaded based on data and data version number query object stores equipment, will download title with data and data version number is opposite
The compressed data answered is determined as target data.
Further, target data processing module further includes target data quantity statistics unit, the transmission of first object data
Unit and the second target data transmission unit.
Target data quantity statistics unit, for counting the quantity of target data.
First object data transmission unit directly passes through target data if the quantity for target data is one
Data download request is sent to client.
Second target data transmission unit, if the quantity for target data is two or more, to number of targets
According to the automatic compression processing of progress, and compressed target data is sent to client by data download request.
In one embodiment, a kind of line compression device is provided, which presses online with above-described embodiment
Contracting method corresponds.The line compression device includes from replica server 20, should include data acquisition from replica server 20
Module 21, data processing module 22, data judgment module 23 and data memory module 24.Detailed description are as follows for each functional module:
Data acquisition module 21, for obtaining compressed data, the compression that major-minor book server is sent by individual thread
Label, reduction length and initial data attribute.
Data processing module 22, if being very, the physical length of compressed data to be obtained, by physical length for compression tag
It is compared with reduction length.
Data judgment module 23, if equal for physical length and reduction length, then it represents that compressed data receives successfully.
Data memory module 24, for compressed data to be written in corresponding physical disks, and more according to initial data attribute
New metadata, by metadata object logics address and physical address be mapped.
Further, from should further including the second downloading request module 25 and the second target data from replica server 20
Manage module 26.
Second downloading request module 25, for obtaining the data download request of client transmission, data download request
Including downloading keyword.
Second target data processing module 26 will be with downloading keyword phase for being based on downloading keyword query physical disks
Corresponding compressed data is determined as target data, and target data is sent to client by data download request.
Specific about line compression device limits the restriction that may refer to above for line compression method, herein not
It repeats again.Modules in above-mentioned line compression device can be realized fully or partially through software, hardware and combinations thereof.On
Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form
In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be server, internal junction
Composition can be as shown in Figure 6.The computer equipment include by system bus connect processor, memory, network interface and
Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment
Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data
Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating
The database of machine equipment is for storing the data being related in line compression method.The network interface of the computer equipment be used for
External terminal passes through network connection communication.To realize a kind of line compression method when the computer program is executed by processor.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory
And the computer program that can be run on a processor, processor realize the online pressure in above-described embodiment when executing computer program
The step of contracting method, such as step S11 shown in Fig. 2 to step S26.Alternatively, step as shown in Figure 3 to Figure 4.Alternatively, place
Reason device realizes the function of each module/unit of line compression device in above-described embodiment, such as Fig. 5 institute when executing computer program
Show the function of module 11 to module 26.To avoid repeating, details are not described herein again.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program realizes the step of line compression method in above-described embodiment, such as step S11 shown in Fig. 2 when being executed by processor
To step S26.Alternatively, step as shown in Figure 3 to Figure 4.Alternatively, realizing above-mentioned implementation when computer program is executed by processor
The function of each module/unit of line compression device in example, such as module 11 shown in Fig. 5 is to the function of module 26.To avoid weight
Multiple, details are not described herein again.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of line compression method is applied to major-minor book server characterized by comprising
The data write request that client is sent is obtained, the data write request includes initial data；
The initial data is compressed, obtains compressed data, and the compressed data is written in object storage device, institute
It states compressed data and carries compression tag and reduction length；
Corresponding initial data attribute is obtained based on the initial data；
Using individual thread by the compressed data, the compression tag, the reduction length and the initial data attribute
It is transferred to from replica server.
2. line compression method as described in claim 1, which is characterized in that deposit compressed data write-in object described
After storing up in equipment, the line compression method further include:
The data download request that client is sent is obtained, the data download request includes downloading keyword；
It, will compressed data corresponding with the downloading keyword based on object storage device described in the downloading keyword query
It is determined as target data, and the target data is sent to the client by the data download request.
3. line compression method as claimed in claim 2, which is characterized in that described based on described in the downloading keyword query
Compressed data corresponding with the downloading keyword is determined as target data by object storage device, comprising:
If the downloading keyword only includes data downloading title, the storage of object described in name query is downloaded based on the data
Compressed data corresponding with data downloading title is determined as target data by equipment；
If the downloading keyword includes data downloading title and data version number, title and described is downloaded based on the data
Versions of data number inquires the object storage device, will pressure corresponding with data downloading title and the versions of data number
Contracting data are determined as target data.
4. line compression method as claimed in claim 2, which is characterized in that described that the target data is passed through the data
Downloading request is sent to the client, comprising:
Count the quantity of target data；
If the quantity of the target data is one, directly the target data is sent to by the data download request
The client；
If the quantity of the target data is two or more, automatic compression processing is carried out to the target data, and
Compressed target data is sent to the client by the data download request.
5. a kind of line compression method is applied to from replica server characterized by comprising
Obtain compressed data, compression tag, reduction length and initial data that major-minor book server is sent by individual thread
Attribute；
If the compression tag is very, the physical length of the compressed data to be obtained, by the physical length and the compression
Length is compared；
If the physical length and reduction length are equal, then it represents that the compressed data receives successfully；
The compressed data is written in corresponding physical disks, and according to the initial data attribute more new metadata, it will be described
Object logics address and physical address in metadata are mapped.
6. line compression method as claimed in claim 5, which is characterized in that described that compressed data write-in is corresponding
In physical disks, and according to the initial data attribute more new metadata, by the metadata object logics address and physics
After address is mapped, the line compression method further include:
The data download request that client is sent is obtained, the data download request includes downloading keyword；
Based on physical disks described in the downloading keyword query, compressed data corresponding with the downloading keyword is determined as
Target data, and the target data is sent to the client by the data download request.
7. a kind of line compression device, which is characterized in that including major-minor book server, the major-minor book server includes:
Write request obtains module, and for obtaining the data write request of client transmission, the data write request includes original
Beginning data；
Data compressing module obtains compressed data, and the compressed data is written for compressing to the initial data
In object storage device, the compressed data carries compression tag and reduction length；
Data attribute obtains module, for obtaining corresponding initial data attribute based on the initial data；
Data transmission module, for using individual thread by the compressed data, the compression tag, the reduction length and
The initial data attribute transfer is given from replica server.
8. a kind of line compression device, which is characterized in that described to include: from replica server including from replica server
Data acquisition module, compressed data, compression tag, the pressure sent for obtaining major-minor book server by individual thread
Contracting length and initial data attribute；
Data processing module, if being very, the physical length of the compressed data to be obtained, by the reality for the compression tag
Border length and the reduction length are compared；
Data judgment module, if equal for the physical length and reduction length, then it represents that the compressed data receives successfully；
Data memory module, for the compressed data to be written in corresponding physical disks, and according to the initial data attribute
More new metadata, by the metadata object logics address and physical address be mapped.
9. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor
The computer program of upper operation, which is characterized in that the processor realized when executing the computer program as claim 1 to
Any one of 4 line compression methods, alternatively, the processor realizes such as claim 5 to 6 when executing the computer program
Any one line compression method.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In, the line compression method as described in any one of Claims 1-4 is realized when the computer program is executed by processor, alternatively,
The processor realizes the line compression method as described in any one of claim 5 to 6 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910337896.5A CN110147201A (en) | 2019-04-25 | 2019-04-25 | Line compression method, apparatus, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910337896.5A CN110147201A (en) | 2019-04-25 | 2019-04-25 | Line compression method, apparatus, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110147201A true CN110147201A (en) | 2019-08-20 |
Family
ID=67594410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910337896.5A Pending CN110147201A (en) | 2019-04-25 | 2019-04-25 | Line compression method, apparatus, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110147201A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
| CN112600563A (en) * | 2020-12-18 | 2021-04-02 | 北京字节跳动网络技术有限公司 | Application file configuration method and device, computer equipment and storage medium |
| CN113704206A (en) * | 2021-10-29 | 2021-11-26 | 苏州浪潮智能科技有限公司 | Metadata processing method and device, electronic equipment and storage medium |
| CN115118716A (en) * | 2022-06-27 | 2022-09-27 | 北京天融信网络安全技术有限公司 | Object data online compression method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108334791A (en) * | 2018-02-07 | 2018-07-27 | 冼钇冰 | Medical data storage method, apparatus based on Internet of Things and computer storage media |
| CN109582245A (en) * | 2018-12-06 | 2019-04-05 | 联想(北京)有限公司 | Data processing method, device and equipment |
-
2019
- 2019-04-25 CN CN201910337896.5A patent/CN110147201A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108334791A (en) * | 2018-02-07 | 2018-07-27 | 冼钇冰 | Medical data storage method, apparatus based on Internet of Things and computer storage media |
| CN109582245A (en) * | 2018-12-06 | 2019-04-05 | 联想(北京)有限公司 | Data processing method, device and equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
| CN112600563A (en) * | 2020-12-18 | 2021-04-02 | 北京字节跳动网络技术有限公司 | Application file configuration method and device, computer equipment and storage medium |
| CN113704206A (en) * | 2021-10-29 | 2021-11-26 | 苏州浪潮智能科技有限公司 | Metadata processing method and device, electronic equipment and storage medium |
| CN113704206B (en) * | 2021-10-29 | 2022-02-22 | 苏州浪潮智能科技有限公司 | Metadata processing method and device, electronic equipment and storage medium |
| CN115118716A (en) * | 2022-06-27 | 2022-09-27 | 北京天融信网络安全技术有限公司 | Object data online compression method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147201A (en) | Line compression method, apparatus, computer equipment and storage medium | |
| CN101405728B (en) | Relational database architecture with dynamic load capability | |
| CN106506587A (en) | A kind of Docker image download methods based on distributed storage | |
| CN104050216B (en) | For customizing the file system manager of resource allocation | |
| CN108897638A (en) | Data back up method, device, computer equipment and storage medium | |
| US11720515B2 (en) | Article, device, and techniques for serverless stack for streaming message processing | |
| CN110781214A (en) | Database reading and writing method and device, computer equipment and storage medium | |
| CN103414762B (en) | cloud backup method and device | |
| CN103593215A (en) | System and method for information handling system image network communication | |
| CN110062028A (en) | Data synchronous method, apparatus, computer equipment and computer storage medium | |
| CN107689976A (en) | A kind of document transmission method and device | |
| CN111209090A (en) | Method and assembly for creating virtual machine in cloud platform and server | |
| CN105049524B (en) | A method of the large-scale dataset based on HDFS loads | |
| US11223528B2 (en) | Management of cloud-based shared content using predictive cost modeling | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| US20150163326A1 (en) | Approaches for remotely unzipping content | |
| CN104408178A (en) | Device and method for WEB control loading | |
| CN110008284A (en) | Method for synchronizing data of database and equipment based on data page preloading and rollback | |
| CN108287663A (en) | A kind of date storage method and device based on EMMC | |
| EP2686791B1 (en) | Variants of files in a file system | |
| CN111339054A (en) | Storage system hierarchical management method, device, equipment and storage medium | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN110019056A (en) | Container separated from meta-data for cloud layer | |
| US7703106B2 (en) | Discovering and monitoring process executions | |
| CN112131025A (en) | Component transmission method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |