CN110019008A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN110019008A CN110019008A CN201711072800.4A CN201711072800A CN110019008A CN 110019008 A CN110019008 A CN 110019008A CN 201711072800 A CN201711072800 A CN 201711072800A CN 110019008 A CN110019008 A CN 110019008A
- Authority
- CN
- China
- Prior art keywords
- log
- fragment
- affiliated
- access
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The embodiment of the invention provides a data storage method and a data storage device, which are applied to a server, and the method comprises the following steps: obtaining an access log file to be stored; performing fragmentation processing on the logs contained in the access log file to obtain logs contained in fragments to which each log of the access log file belongs; and storing the logs contained in the obtained fragments to which each log belongs to the storage device in a parallel mode. According to the technical scheme provided by the embodiment of the invention, the access log file is subjected to fragmentation processing to obtain the logs contained in the fragments to which the logs belong, and the obtained logs are stored in a parallel mode, so that the efficiency of storing the access log file by the server is improved.
Description
Technical field
The present invention relates to technical field of data storage, more particularly to a kind of date storage method and device.
Background technique
Access log file is the log comprising the client operation information that client generates, and access log file can be with
Much information is recorded, for example, when user's login on the client and log-off message, firewall information, client failures
Fault message etc..Access log file can be sent to server by client, for example, above-mentioned server, which can be, to be had
The server of the high performance web container such as nginx or apache is stored after above-mentioned server receives access log file
Into storage equipment, in this way, server can be for statistical analysis to the access log file stored, and then available right
Answer the operating status of client.
However, server for the read-write of data is written and read based on file under normal conditions, therefore, server exists
After obtaining the access log file that client is sent, generally this access log file distributes a thread, by this
One thread is responsible for storing above-mentioned access log file to storage equipment.In this way, result in server to access log file into
The problem of low efficiency of row storage.
Summary of the invention
The embodiment of the present invention is designed to provide a kind of date storage method and device, is set with settlement server in storage
When standby middle storage access log file the problem of low efficiency.Specific technical solution is as follows:
In a first aspect, being applied to server, the method packet the embodiment of the invention provides a kind of date storage method
It includes:
Obtain access log file to be stored；
The log for being included to the access log file carries out fragment processing, obtains each of the access log file
The log that the affiliated fragment of log is included；
In a parallel fashion, the log that each affiliated fragment of log obtained is included is stored to storage equipment.
Optionally, described the step of fragment processing is carried out to the access log file, comprising:
According to the attribute of the included log of the access log file, the log that is included to the access log file into
The processing of row fragment.
Optionally, the attribute according to the included log of the access log file, to the access log file into
The step of row fragment processing, comprising:
Detect the corresponding relationship that whether there is log to be processed in mapping table, wherein the corresponding relationship is day
Corresponding relationship between the attribute value of will attribute and the number of the affiliated fragment of log, the mapping table record pair of each log
It should be related to, each corresponding storage location of the affiliated fragment of log；
If it exists, according to corresponding relationship, the log for being included to the access log file carries out fragment processing, obtains institute
State the log that the affiliated fragment of each log of access log file is included, wherein the corresponding attribute value of each log, and
And the affiliated fragment of log corresponding to log is corresponding with the attribute value of the log.
Optionally, the method also includes:
If detecting the corresponding relationship that log to be processed is not present in mapping table, determined using following formula to be processed
The number of the affiliated fragment of log corresponding to log:
P=k%n
Wherein, p is the number of the affiliated fragment of log, and k is attribute value, and n is the quantity of the affiliated fragment of log；
According to the number and the attribute value of log to be processed of the affiliated fragment of identified log, pair of log to be processed is determined
It should be related to；
According to identified corresponding relationship, the log for being included to the access log file carries out fragment processing, obtains
The log that the affiliated fragment of each log of the access log file is included, wherein the corresponding attribute value of each log, and
And the affiliated fragment of log corresponding to log is corresponding with the attribute value of the log.
Optionally, the corresponding relationship according to determined by, the log for being included to the access log file divide
Piece processing, before the step of obtaining the log that the affiliated fragment of each log of the access log file is included, further includes:
Obtain the corresponding storage location of the affiliated fragment of log；
With the presence or absence of abnormal storage location in the acquired storage location of detection；
If it does, determining the corresponding affiliated fragment of log of the abnormal storage location, and will be belonging to identified log
The corresponding attribute value of fragment is distributed to preset reservation fragment, wherein the reservation fragment is the day for being not involved in distributive property value
The affiliated fragment of will, and the corresponding storage location of the reservation fragment is normal；
If it does not, executing the corresponding relationship according to determined by, the day for being included to the access log file
The step of will carries out fragment processing, obtains the log that the affiliated fragment of each log of the access log file is included.
Optionally, the corresponding relationship according to determined by, the log for being included to the access log file divide
Piece processing, after the step of obtaining the log that the affiliated fragment of each log of the access log file is included, the method
Further include:
Identified corresponding relationship is updated in the mapping table.
Optionally, before the detection whether there is the step of corresponding relationship of log to be processed in mapping table,
The method also includes:
Detect whether the storage location for having newly-increased；
If it does not, executing the step of detection whether there is the corresponding relationship of log to be processed in mapping table；
If it does, the step of executing the number for determining the affiliated fragment of log corresponding to log to be processed.
Optionally, the log for being included to the access log file carries out fragment processing, obtains the access day
After the step of log that the affiliated fragment of each log of will file is included, the method also includes:
The log for being included to each affiliated fragment of log obtained carries out data cleansing；
It is described in a parallel fashion, by the log that each affiliated fragment of log obtained is included store to storage equipment
Step, comprising:
In a parallel fashion, the log that the affiliated fragment of each log after progress data cleansing is included is stored to storage and is set
It is standby.
Second aspect, the embodiment of the invention provides a kind of data storage devices, are applied to server, described device packet
It includes:
Module is obtained, for obtaining access log file to be stored；
Fragment processing module, log for being included to the access log file carry out fragment processing, described in acquisition
The log that the affiliated fragment of each log of access log file is included；
Memory module, in a parallel fashion, by the log that each affiliated fragment of log obtained is included store to
Store equipment.
Optionally, the fragment processing module includes:
Fragment handles submodule, for the attribute according to the included log of the access log file, to the access day
The log that will file is included carries out fragment processing.
Optionally, the fragment processing submodule includes:
First detection unit, for detecting the corresponding relationship that whether there is log to be processed in mapping table, wherein
The corresponding relationship is the corresponding relationship between the attribute value of log properties and the number of the affiliated fragment of log, the corresponding relationship
Table records the corresponding relationship of each log, each corresponding storage location of the affiliated fragment of log；
First fragment processing unit in the presence of being for the testing result when the first detection unit, is closed according to corresponding
System, the log for being included to the access log file carry out fragment processing, obtain each log of the access log file
The log that affiliated fragment is included, wherein the corresponding attribute value of each log, also, divide belonging to log corresponding to log
Piece is corresponding with the attribute value of the log.
Optionally, the fragment handles submodule further include:
First determination unit, it is true using following formula in the presence of being for the testing result when the first detection unit
The number of the affiliated fragment of log corresponding to fixed log to be processed:
P=k%n
Wherein, p is the number of the affiliated fragment of log, and k is attribute value, and n is the quantity of the affiliated fragment of log；
Second determination unit, for the number and the attribute value of log to be processed according to the affiliated fragment of identified log,
Determine the corresponding relationship of log to be processed；
Second fragment processing unit, for being included to the access log file according to identified corresponding relationship
Log carries out fragment processing, obtains the log that the affiliated fragment of each log of the access log file is included, wherein each
Log corresponds to an attribute value, also, the affiliated fragment of log corresponding to log is corresponding with the attribute value of the log.
Optionally, the fragment handles submodule further include:
Acquiring unit, for obtaining the corresponding storage location of the affiliated fragment of log；
Second detection unit, for detecting with the presence or absence of abnormal storage location in acquired storage location, when detection institute
There is no when abnormal storage location in the storage location of acquisition, the second fragment processing unit is triggered；
Third determination unit determines that the exception is deposited in the presence of being for the testing result when the second detection unit
Storage space sets the corresponding affiliated fragment of log, and the corresponding attribute value of the affiliated fragment of identified log is distributed to preset reservation
Fragment, wherein the reservation fragment is the affiliated fragment of log for being not involved in distributive property value, and the reservation fragment is corresponding deposits
Storage space is set normally.
Optionally, the fragment handles submodule further include:
Updating unit, for identified corresponding relationship to be updated in the mapping table.
Optionally, the fragment handles submodule further include:
Third detection unit, for detecting whether there is newly-increased storage location, when there is no newly-increased storage positions for detection
When setting, the first detection unit is triggered, when detection has newly-increased storage location, triggers first determination unit.
Optionally, described device further include:
Data cleansing module, the log for being included to each affiliated fragment of log obtained carry out data cleansing；
The memory module includes:
Sub-module stored, in a parallel fashion, being included by the affiliated fragment of each log after progress data cleansing
Log is stored to storage equipment.
The third aspect, the embodiment of the invention provides a kind of electronic equipment, including processor, communication interface, memory and
Communication bus, wherein processor, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes a kind of any of the above-described data storage
The step of method.
Fourth aspect, the embodiment of the invention provides a kind of computer readable storage medium, the computer-readable storage
Dielectric memory contains computer program, and the computer program realizes a kind of any of the above-described data when being executed by processor
The step of storage method.
In technical solution provided in an embodiment of the present invention, access log file to be stored is obtained；To access log file
The log for being included carries out fragment processing, obtains the log that the affiliated fragment of each log of access log file is included；With simultaneously
Line mode stores the log that each affiliated fragment of log obtained is included to storage equipment.Through the embodiment of the present invention
Access log file is carried out fragment processing, obtains the log that the affiliated fragment of log is included, and pass through by the technical solution of offer
Parallel mode stores obtained log, in this way, improving the efficiency that server stores access log file.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is a kind of a kind of flow chart of date storage method provided in an embodiment of the present invention；
Fig. 2 is a kind of another flow chart of date storage method provided in an embodiment of the present invention；
Fig. 3 is a kind of another flow chart of date storage method provided in an embodiment of the present invention；
Fig. 4 is a kind of a kind of structural schematic diagram of data storage device provided in an embodiment of the present invention；
Fig. 5 is a kind of another structural schematic diagram of data storage device provided in an embodiment of the present invention；
Fig. 6 is a kind of another structural schematic diagram of data storage device provided in an embodiment of the present invention；
Fig. 7 is a kind of structural schematic diagram of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Nominal explanation, each reality of the definition and explanation of following word suitable for the embodiment of the present invention are carried out first below
Apply mode.
Access log file is the file of the log generated for recording operating main body access operation object, for one
For a access log file, wherein can wrap containing a log, also it can wrap containing a plurality of log, the application is not to this
It is defined.
For example, aforesaid operations main body can be client, operation object can be Website server, application program service
Device, http-server etc., then for storing the behaviour such as client access Website server, apps server, http-server
The file of generated log is referred to as access log file when making object, certainly, can also store in the access log file
The log for having client to access other operation objects and generating, will not enumerate here.
Fragment processing are as follows: according to preset classifying rules, the process classified to the log in access log file.Often
One classification can be understood as a fragment of access log file.
For example, attribute value can classify corresponding to the attribute according to log, specifically, it is identical to can be attribute value
Log as one classify；It can also be in the way of attribute value complementation, at this point, corresponding to the identical attribute value of remainder
Log is classified as one.Certainly, it is not limited in above two, no longer one one action herein according to the mode that attribute value is classified
Example.
For example, can classify according to the size of log, specifically, can set log size [100kb,
200kb] conduct one classification in range, conduct of the log size in [200kb, 300kb] range one classification, remaining
Classify as one.
It should be noted that [x, y] is indicated to be greater than or equal to x and is less than or equal to the numerical value of y, the embodiment of the present invention
[] of middle appearance indicates the meaning.
Certainly, the example above is not limited in the mode that log is classified, it can also be according to other classifying rules pair
The classification that log carries out, will not enumerate herein.
In order to improve the efficiency that server stores access log file, the embodiment of the invention provides a kind of data
Storage method and device are applied to server, wherein date storage method includes:
Obtain access log file to be stored；
The log for being included to access log file carries out fragment processing, obtains belonging to each log of access log file
The log that fragment is included；
In a parallel fashion, the log that each affiliated fragment of log obtained is included is stored to storage equipment.
Access log file is carried out fragment processing, obtains log institute by the technical solution provided through the embodiment of the present invention
Belong to the log that fragment is included, and obtained log is stored by parallel mode, in this way, improving server to visit
Ask the efficiency that journal file is stored.
It describes in detail first below to a kind of date storage method provided in an embodiment of the present invention, as shown in Figure 1, this
A kind of date storage method that inventive embodiments provide is applied to server, includes the following steps:
S101 obtains access log file to be stored.
Wherein, access log file is log caused by the client of server communication connection, generates access log
The client of file can be mobile phone, computer, plate, smartwatch etc..It include client operation information in access log file
Log.
The access log file can be sent to server, to this after obtaining access log file by client
Access log file is stored.That is, the access log text that the client that server can receive communication connection is sent
Part.
S102, the log for being included to access log file carry out fragment processing, obtain each day of access log file
The log that the affiliated fragment of will is included.
It, can be right according to preset classifying rules when the log for being included to access log file carries out fragment processing
Log in access log file carries out fragment processing.Wherein, classifying rules can be customized setting, for example, can be by
According to log size classified, can be classified according to the attribute value of log properties, can also be generated according to log
Time sequencing is classified etc..
In a kind of implementation, classifying rules can be with are as follows: classifies according to the size of log.For example, day can be set
The size [500,599] of will is one kind, i.e. an affiliated fragment of log；It [600,699] is the affiliated fragment of a log.Access log
It include log A, log B, log C, log D, log E in file, wherein log A is 550kb, and log B is 530kb, log C
It is 610kb for 650kb, log D, log E is 680kb；Divide it is possible to which log A and log B is determined to belong to belonging to same log
Piece, log C, log D and log E belong to the affiliated fragment of same log.
With the classifying rules that the attribute value according to log properties is classified, it is specifically described in embodiment below,
Herein without being described in detail.
S103 in a parallel fashion stores the log that each affiliated fragment of log obtained is included to storage equipment.
Wherein, storage equipment can correspond to uniform machinery with server, for example, as the storage in the machine of server
Hard disk；It can also be that storage equipment belongs to different machines from server, for example, based on open source full-text search engine Lucene
Search server (Elastic Search, abbreviation es)；The Hadoop distributed file system that can be deployed on each machine
(Hadoop Distributed File System, abbreviation hdfs).
Wherein, the transmission path in parallel mode can be distributed thread, transmission path and the affiliated fragment of log
Corresponding relationship can be customized setting, that is, and it can be a transmission path and correspond to the affiliated fragment of multiple logs, it can be with one
The corresponding affiliated fragment of log of a transmission path.
In a kind of implementation, parallel mode can using transmission path and the one-to-one mode of the affiliated fragment of log,
That is, the quantity of transmission path is identical with the quantity of the affiliated fragment of log, also, the affiliated fragment of each log corresponding one is exclusive
Transmission path.For example, the affiliated fragment A of log corresponds to transmission path A, then, it is only possible by the log that transmission path A is transmitted
Included log in the affiliated fragment A of log, and be transmitted to storage location corresponding to the affiliated fragment A of log and stored.
It is transmitted by the log that parallel mode is included to each affiliated fragment of log obtained, improves service
Transmission and storage efficiency of the device to access log file.
In the first embodiment, to access log file carry out fragment processing (S102) the step of, can specifically include:
According to the attribute of the included log of access log file, the log for being included to access log file carries out fragment processing.
It, can customized selection any attribute when log included in access log file has multiple attributes simultaneously
Foundation as fragment processing.
For example, the attribute of log can be general exclusive identification code (Universally Unique Identifier, letter
Claim uuid), that is to say, that the log for being included to access log file carries out fragment processing can be according to the general unique of log
Identification code carries out fragment processing.
In second of embodiment, the log for being included to the access log file carries out fragment processing, described in acquisition
After the step of log (S102) that the affiliated fragment of each log of access log file is included, this method can also include:
The log for being included to each affiliated fragment of log obtained carries out data cleansing.
Wherein, data cleansing may include various ways: the consistency check of data, the inspection of data invalid value, data
Inspection of missing values etc..
The mode of data cleansing can carry out customized selection, can choose and only carry out one of data cleansing, for example,
Only carry out the consistency check of data, the inconsistent situation of data if it exists, i.e. data, then can be by numbers there are when multiple format
According to being converted into unified format.
It is also an option that a variety of data cleansing modes are carried out simultaneously, for example, carrying out the consistency check of data, data simultaneously
The inspection of invalid value, when checking data, there are consistency problems, then convert the data into unified format；When checking that data deposit
In invalid value, then invalid value is deleted.
In the embodiment, after carrying out data cleansing, in a parallel fashion, it will divide belonging to each log obtained
The log that piece is included is stored to the step of storing equipment (S103), can specifically include:
In a parallel fashion, the log that the affiliated fragment of each log after progress data cleansing is included is stored to storage and is set
It is standby.
In this embodiment, what is transmitted in a parallel fashion is the log carried out after data cleansing, is stored in storage equipment
In be also carry out data cleansing after log.In this way, invalid or missing data can be disposed, improves and stored
The ratio of effective log.
The step can be with reference to the S103 in above-described embodiment, and details are not described herein.
In technical solution provided in an embodiment of the present invention, access log file to be stored is obtained；To access log file
The log for being included carries out fragment processing, obtains the log that the affiliated fragment of each log of access log file is included；With simultaneously
Line mode stores the log that each affiliated fragment of log obtained is included to storage equipment.Through the embodiment of the present invention
Access log file is carried out fragment processing, obtains the log that the affiliated fragment of log is included, and pass through by the technical solution of offer
Parallel mode stores obtained log, in this way, improving the efficiency that server stores access log file.
The embodiment of the present invention also provides a kind of date storage method, is applied to server, as shown in Fig. 2, this method includes
Following steps:
S201 obtains access log file to be stored.
In the present embodiment, S201 is identical as the S101 of above-described embodiment, and details are not described herein.
S202 detects the corresponding relationship that whether there is log to be processed in mapping table；If so, executing step
S203。
Wherein, corresponding relationship are as follows: the corresponding relationship between the attribute value of log properties and the number of the affiliated fragment of log.Day
Will attribute is the foundation as fragment processing of customized selection, for example, when log properties are uuid, at this point, corresponding close
System are as follows: the corresponding relationship between value corresponding to uuid and the number of the affiliated fragment of log.
Wherein, the affiliated fragment of log can be preset, the number of the affiliated fragment of log can be it is preset, and
The number of each affiliated fragment of log is unique.
In addition, each corresponding storage location of the affiliated fragment of log.Therefore, the quantity of the affiliated fragment of log is by setting
What the quantity of storage location was determined.For example, the storage location of setting has 4, then, the quantity of the corresponding affiliated fragment of log
It also is 4, also, the affiliated fragment of each log only corresponds to a storage location.
Storage location can be extended according to demand, in a kind of implementation, when increase store equipment when, be adapted to
The quantity of the increase storage location of property.For example, the quantity of storage location is increased to 8 from 4, correspondingly, divide belonging to log
The quantity of piece also increases to 8 from 4.
It is believed that by a channel between the affiliated fragment of a log and the corresponding storage location of the affiliated fragment of the log
It is connected.Channel is used to carry out log transmission included in the affiliated fragment of the log connected to the storage location connected
Storage.
Wherein it is possible to think, the corresponding transmission path of the affiliated fragment of log includes the corresponding channel of the affiliated fragment of the log.
Specifically, when the corresponding transmission path of the affiliated fragment of a log, it may be considered that, the corresponding channel of the affiliated fragment of the log
The as transmission path；And when a transmission path corresponds to the affiliated fragment of multiple logs, then the affiliated fragment of the log is corresponding
Channel is one of them in multiple channels included by the transmission path.
In the embodiment of the present invention, log included in access log file can be respectively allocated to each log by server
In affiliated fragment, then, each affiliated fragment of log passes through respectively corresponding channel respectively, and respectively included log is sent
It is stored to corresponding storage location.
For example, the affiliated fragment A of log includes log A, log B, also, the corresponding storage location of the affiliated fragment A of log are as follows:
Channel between storage location A, log affiliated fragment A and storage location A is channel A.In this way, log A and log B pass through channel
A is transmitted to storage location A and is stored.
Wherein, mapping table can be preset, and the corresponding relationship of each log is recorded in mapping table.When will to institute
When the access log file of acquisition carries out fragment processing, can be inquired in mapping table first with the presence or absence of log to be processed
Corresponding relationship executes step S203 when it is present；When it be not present, phase is carried out in following figure 3 and the corresponding embodiment of Fig. 3
It introduces with answering, it is not described here in detail.
S203, according to corresponding relationship, the log for being included to access log file carries out fragment processing, obtains access log
The log that the affiliated fragment of each log of file is included.
Wherein, the corresponding attribute value of each log, also, the category of log corresponding to log affiliated fragment and the log
Property value is corresponding.
Wherein, an attribute value corresponding to each log are as follows: the category of the selected attribute as fragment processing foundation
Property value.For example, then each log can correspond to the value of a uuid using the attribute of the uuid foundation handled as fragment, the value
An as corresponding attribute value.
Included each log in access log file corresponds to the attribute value of an attribute, is then note in corresponding relationship
The corresponding relationship of attribute value and the affiliated fragment of log corresponding to log is recorded, each attribute value, which only corresponds to belonging to a log, to be divided
Piece, the affiliated fragment of each log can then correspond to multiple attribute values.
For example, the preset affiliated fragment of log is respectively as follows: fragment 0, fragment 1, fragment 2, recorded in mapping table
Corresponding relationship includes: that the attribute value 0 of log is corresponding with fragment 0, attribute value of log 1 and fragment 1 are corresponding, log attribute
Value 2 is corresponding with fragment 2, log attribute value 3 and fragment 0 are corresponding, log attribute value 4 and fragment 1 are corresponding.
Also, the log for including in acquired access log file are as follows: log A, log B, log C, log D, log
E, wherein attribute value corresponding to log A is 0, and attribute value corresponding to log B is 1, and attribute value corresponding to log C is 2,
Attribute value corresponding to log D is 3, and attribute value corresponding to log E is 4.So, according to the corresponding pass in mapping table
System, can determine: log included by fragment 0 is log A and log D, and log included by fragment 1 is log B and log E,
Log included by fragment 2 is log C.
S204 in a parallel fashion stores the log that each affiliated fragment of log obtained is included to storage equipment.
In the present embodiment, S204 is identical as the S103 of above-described embodiment, and details are not described herein.
The technical solution provided through the embodiment of the present invention, each log according to recorded in preset mapping table
Access log file is carried out fragment processing by corresponding relationship, obtains the log that the affiliated fragment of log is included, and pass through parallel side
Formula stores obtained log, in this way, improving the efficiency that server stores access log file.
The embodiment of the present invention also provides a kind of date storage method, is applied to server, as shown in figure 3, this method includes
Following steps:
S301 obtains access log file to be stored.
In the present embodiment, S301 is identical as the S101 of above-described embodiment, and details are not described herein.
S302 detects the corresponding relationship that whether there is log to be processed in mapping table；If so, executing step
S303, if not, executing step S304.
Wherein, corresponding relationship are as follows: the corresponding relationship between the attribute value of log properties and the number of the affiliated fragment of log；It is right
Relation table is answered to record the corresponding relationship of each log, each corresponding storage location of the affiliated fragment of log.
S303, according to corresponding relationship, the log for being included to access log file carries out fragment processing, obtains access log
The log that the affiliated fragment of each log of file is included.
Wherein, the corresponding attribute value of each log, also, the category of log corresponding to log affiliated fragment and the log
Property value is corresponding.
In the present embodiment, S302, S303 are identical as S202, S203 of above-described embodiment, and details are not described herein.
S304 determines the number of the affiliated fragment of log corresponding to log to be processed using following formula:
P=k%n
Wherein, p is the number of the affiliated fragment of log, and k is attribute value, and n is the quantity of the affiliated fragment of log.
In the present embodiment, if there is no pairs for the log to be processed being partitioned into access log file in mapping table
It should be related to.A kind of embodiment, can be by the method to attribute value complementation corresponding to log to be processed, and then determining should be to
Handle the affiliated fragment of log corresponding to log.
For example, the quantity of the preset affiliated fragment of log is 3, it is respectively as follows: fragment 0, fragment 1, fragment 2, corresponding number
Respectively 0,1,2.The attribute value of log to be processed is 10, then, attribute value k is 10, and the quantity n of the affiliated fragment of log is 3, root
According to above-mentioned complementation formula, the number p=10%3=1 of the available affiliated fragment of log.The attribute of the log to be processed known to then
Value is corresponding with fragment 1.
S305 determines day to be processed according to the number and the attribute value of log to be processed of the affiliated fragment of identified log
The corresponding relationship of will.
According to the number of the affiliated fragment of the available log of formula in above-mentioned steps S304, which is by day to be processed
What the attribute value of will was calculated, therefore, the attribute value of log to be processed and the number of the affiliated fragment of calculated log are phases
Corresponding, the corresponding relationship obtained in this way is the corresponding relationship of log to be processed.
S306, according to identified corresponding relationship, the log for being included to access log file carries out fragment processing, obtains
The log that the affiliated fragment of each log of access log file is included.
Wherein, the corresponding attribute value of each log, also, the category of log corresponding to log affiliated fragment and the log
Property value is corresponding.
Wherein, corresponding relationship based on is by the determined corresponding relationship of above-mentioned steps S305.
In the present embodiment, S306 can be with reference to the S203 in above-described embodiment, and details are not described herein.
S307 in a parallel fashion stores the log that each affiliated fragment of log obtained is included to storage equipment.
In the present embodiment, S307 is identical as the S103 of above-described embodiment, and this will not be repeated here.
In the third embodiment, according to identified corresponding relationship, the log for being included to access log file is carried out
Fragment processing may be used also before the step of obtaining log (S306) that the affiliated fragment of each log of access log file is included
To include the following steps:
Firstly, obtaining the corresponding storage location of the affiliated fragment of log；Whether deposited in the acquired storage location of detection
In abnormal storage location.
Wherein, the affiliated fragment of log and storage location are one-to-one, therefore, can be obtained by the affiliated fragment of each log
To the corresponding storage location of the affiliated fragment of each log.
Wherein, storage location includes: that cannot write data into storage location, cannot read data from storage location extremely
Deng.As long as detecting that storage location meets at least one of above-mentioned abnormal conditions, it can determine that the storage location is deposited to be abnormal
Storage space is set.For example, data cannot be written in storage location detected, then the storage location can be determined for abnormal storage location.
There is abnormal storage location when detecting in acquired storage location, determine exception storage location in a kind of situation
The corresponding affiliated fragment of log, and the corresponding attribute value of the affiliated fragment of identified log is distributed to preset reservation fragment.
Wherein, reservation fragment is preset, and retaining fragment is the affiliated fragment of log for being not involved in distributive property value, and
It is normal to retain the corresponding storage location of fragment.Also, the quantity for retaining fragment can be preparatory customized setting, a kind of realization
In mode, a fragment can be only set as reservation fragment, at this point, when there is abnormal storage location, it can be by abnormal storage
The attribute value that the position affiliated fragment of corresponding log is distributed, reassigns to the reservation fragment.In this way, guaranteeing subsequent fragment
Processing is normally carried out.
For example, the corresponding storage location of fragment 0 is position 0, the corresponding storage location of fragment 1 is position 1, and fragment 2 is corresponding
Storage location be position 2, preset reservations fragment is fragment 3, also, the corresponding relationship recorded in mapping table are as follows: category
Property value 10 is corresponding with fragment 0, and attribute value 15 is corresponding with fragment 1, and attribute value 20 is corresponding with fragment 2.Detect 0 He of position
There is exception in position 1, is determined as abnormal storage location, it is possible to which attribute value 10 and attribute value 15 are distributed to fragment
3, in this way, the corresponding log of attribute value 10 and the corresponding log of attribute value 15 are respectively assigned to fragment 3 when carrying out fragment processing
In.
In another implementation, multiple fragments can be set as fragment is retained, for example, 2 fragments of setting are as guarantor
Stay fragment.In this way, when there is abnormal storage location, the category that the affiliated fragment of the corresponding log of abnormal storage location can be distributed
Property value, reassign to as retain fragment multiple fragments.
For example, the corresponding storage location of fragment 0 is position 0, the corresponding storage location of fragment 1 is position 1, and fragment 2 is corresponding
Storage location be position 2, also, the corresponding relationship recorded in mapping table are as follows: attribute value 10 is corresponding with fragment 0, category
Property value 15 is corresponding with fragment 1, and attribute value 20 is corresponding with fragment 2.Preset reservation fragment is fragment 3 and fragment 4, is detected
Position 0 and position 1 occur abnormal, are determined as abnormal storage location, can be with it is possible to attribute value 10 is distributed to fragment 3
Attribute value 15 is distributed into fragment 4.In this way, the corresponding log of attribute value 10 is distributed into fragment 3 when carrying out fragment processing,
The corresponding log of attribute value 15 is distributed into fragment 4.
Another situation, when detecting in acquired storage location there is no when abnormal storage location, i.e., each log institute
Belong to the storage location that storage location corresponding to fragment can be normally written and be read, then executes above-mentioned steps S306.
Each storage location is carried out to detect whether exception, is being detected before carrying out fragment processing by the embodiment
To when there is abnormal storage location, the corresponding relationship of abnormal storage location can be adjusted in time, it is ensured that fragment
Treatment process is gone on smoothly, also, improves the efficiency of fragment processing.
In 4th kind of embodiment, according to identified corresponding relationship, the log for being included to access log file is carried out
Fragment processing may be used also after the step of obtaining log (S306) that the affiliated fragment of each log of access log file is included
To include the following steps:
Identified corresponding relationship is updated in mapping table.
Wherein, the corresponding relationship determined is unwritten new corresponding relationship in mapping table, also, this is right
It should be related to and can be what the method in the embodiment as corresponding to above-mentioned Fig. 3 obtained, it is, of course, also possible to be obtained by other methods
New corresponding relationship.
By the embodiment, new corresponding relationship can be increased in mapping table and be stored, in this way, for
The log of same attribute value can inquire the corresponding relationship of needs, from mapping table directly so as to avoid repeating
The operation for determining corresponding relationship, improves the efficiency of fragment processing, and then improves the efficiency of data storage.
In 5th kind of embodiment, the corresponding relationship (S202) that whether there is log to be processed in mapping table is detected
The step of before, can also include the following steps:
Detect whether the storage location for having newly-increased；If the storage location not increased newly, thens follow the steps S302.If
When detecting the presence of newly-increased storage location, S304 is thened follow the steps.
Wherein, newly-increased storage location may is that the storage location increaseing accordingly according to the increase of storage equipment.Storage
Position is one-to-one relationship with the affiliated fragment of log, and therefore, storage location increases, then the number of the affiliated fragment of corresponding log
Amount also increases, then, need to redefine corresponding relationship, i.e. execution step S304.
By the embodiment, storage equipment and storage location can be extended according to demand, and can phase
It extends the transmission path in parallel mode with answering, so that flow shunt, and then improves the efficiency of fragment processing and data are deposited
The throughput of storage.
In technical solution provided in an embodiment of the present invention, access log file to be stored is obtained；To access log file
The log for being included carries out fragment processing, obtains the log that the affiliated fragment of each log of access log file is included；With simultaneously
Line mode stores the log that each affiliated fragment of log obtained is included to storage equipment.Through the embodiment of the present invention
Access log file is carried out fragment processing, obtaining the affiliated fragment of log is included according to hash algorithm by the technical solution of offer
Log, and obtained log is stored by parallel mode, in this way, improve server to access log file into
The efficiency of row storage.
Corresponding to the corresponding embodiment of the method for above-mentioned Fig. 1 and Fig. 1, the embodiment of the present invention also provides a kind of data storage dress
It sets, is applied to server, as shown in figure 4, the device includes:
Module 410 is obtained, for obtaining access log file to be stored；
Fragment processing module 420, the log for being included to access log file carry out fragment processing, obtain access day
The log that the affiliated fragment of each log of will file is included；
Memory module 430, in a parallel fashion, the log for being included by each affiliated fragment of log obtained to store
To storage equipment.
In a kind of embodiment, fragment processing module 420 may include:
Fragment handles submodule, for the attribute according to the included log of access log file, to access log file institute
The log for including carries out fragment processing.
In a kind of embodiment, device can also include:
Data cleansing module, the log for being included to each affiliated fragment of log obtained carry out data cleansing；
Memory module 430 can also include:
Sub-module stored, in a parallel fashion, being included by the affiliated fragment of each log after progress data cleansing
Log is stored to storage equipment.
In technical solution provided in an embodiment of the present invention, access log file to be stored is obtained；To access log file
The log for being included carries out fragment processing, obtains the log that the affiliated fragment of each log of access log file is included；With simultaneously
Line mode stores the log that each affiliated fragment of log obtained is included to storage equipment.Through the embodiment of the present invention
Access log file is carried out fragment processing, obtains the log that the affiliated fragment of log is included, and pass through by the technical solution of offer
Parallel mode stores obtained log, in this way, improving the efficiency that server stores access log file.
Corresponding to the corresponding embodiment of the method for above-mentioned Fig. 2 and Fig. 2, the embodiment of the present invention also provides a kind of data storage dress
It sets, is applied to server, as shown in figure 5, fragment handles submodule also on the basis of Fig. 4 and Fig. 4 corresponding Installation practice
May include:
First detection unit 421, for detecting the corresponding relationship that whether there is log to be processed in mapping table,
In, the corresponding relationship is the corresponding relationship between the attribute value of log properties and the number of the affiliated fragment of log, the correspondence
Relation table records the corresponding relationship of each log, each corresponding storage location of the affiliated fragment of log；
First fragment processing unit 422, in the presence of being for the testing result when the first detection unit 421, according to
Corresponding relationship, the log for being included to the access log file carry out fragment processing, obtain the every of the access log file
The log that the affiliated fragment of one log is included, wherein the corresponding attribute value of each log, also, log corresponding to log
Affiliated fragment is corresponding with the attribute value of the log.
The technical solution provided through the embodiment of the present invention, each log according to recorded in preset mapping table
Access log file is carried out fragment processing by corresponding relationship, obtains the log that the affiliated fragment of log is included, and pass through parallel side
Formula stores obtained log, in this way, improving the efficiency that server stores access log file.
Corresponding to the corresponding embodiment of the method for above-mentioned Fig. 3 and Fig. 3, the embodiment of the present invention also provides a kind of data storage dress
It sets, is applied to server, as shown in fig. 6, fragment handles submodule also on the basis of Fig. 5 and Fig. 5 corresponding Installation practice
May include:
First determination unit 423 utilizes following formula in the presence of being for the testing result when the first detection unit
Determine the number of the affiliated fragment of log corresponding to log to be processed:
P=k%n
Wherein, p is the number of the affiliated fragment of log, and k is attribute value, and n is the quantity of the affiliated fragment of log；
Second determination unit 424, for the number and the attribute of log to be processed according to the affiliated fragment of identified log
Value, determines the corresponding relationship of log to be processed；
Second fragment processing unit 425, for being included to the access log file according to identified corresponding relationship
Log carry out fragment processing, obtain the log that the affiliated fragment of each log of the access log file is included, wherein every
One log corresponds to an attribute value, also, the affiliated fragment of log corresponding to log is corresponding with the attribute value of the log.
The technical solution provided through the embodiment of the present invention carries out access log file at fragment according to hash algorithm
Reason, obtains the log that the affiliated fragment of log is included, and store to obtained log by parallel mode, in this way, mentioning
The efficiency that high server stores access log file.
In a kind of embodiment, fragment processing submodule can also include:
Acquiring unit, for obtaining the corresponding storage location of the affiliated fragment of log；
Second detection unit, for detecting with the presence or absence of abnormal storage location in acquired storage location, when detection institute
There is no when abnormal storage location in the storage location of acquisition, the second fragment processing unit is triggered；
Third determination unit determines that the exception is deposited in the presence of being for the testing result when the second detection unit
Storage space sets the corresponding affiliated fragment of log, and the corresponding attribute value of the affiliated fragment of identified log is distributed to preset reservation
Fragment, wherein the reservation fragment is the affiliated fragment of log for being not involved in distributive property value, and the reservation fragment is corresponding deposits
Storage space is set normally.
In a kind of embodiment, fragment processing submodule can also include:
Updating unit, for identified corresponding relationship to be updated in the mapping table.
In a kind of embodiment, fragment processing submodule can also include:
Third detection unit, for detecting whether there is newly-increased storage location, when there is no newly-increased storage positions for detection
When setting, the first detection unit is triggered, when detection has newly-increased storage location, triggers first determination unit.
The technical solution provided through the embodiment of the present invention carries out access log file at fragment according to hash algorithm
Reason, obtains the log that the affiliated fragment of log is included, and store to obtained log by parallel mode, in this way, mentioning
The efficiency that high server stores access log file.
The embodiment of the invention also provides a kind of electronic equipment, as shown in fig. 7, comprises processor 710, communication interface 720,
Memory 730 and communication bus 740, wherein processor 710, communication interface 720, memory 730 are complete by communication bus 740
At mutual communication；
Memory 730, for storing computer program；
Processor 710 when for executing the program stored on memory 730, realizes following steps:
Obtain access log file to be stored；
The log for being included to access log file carries out fragment processing, obtains belonging to each log of access log file
The log that fragment is included；
In a parallel fashion, the log that each affiliated fragment of log obtained is included is stored to storage equipment.
In technical solution provided in an embodiment of the present invention, access log file to be stored is obtained；To access log file
The log for being included carries out fragment processing, obtains the log that the affiliated fragment of each log of access log file is included；With simultaneously
Line mode stores the log that each affiliated fragment of log obtained is included to storage equipment.Through the embodiment of the present invention
Access log file is carried out fragment processing, obtains the log that the affiliated fragment of log is included, and pass through by the technical solution of offer
Parallel mode stores obtained log, in this way, improving the efficiency that server stores access log file.
Certainly, a kind of electronic equipment provided in an embodiment of the present invention can also be performed any described one in above-described embodiment
Kind date storage method.It is specifically shown in embodiment corresponding to Fig. 1, Fig. 2 and Fig. 3, which is not described herein again.
In another embodiment provided by the invention, a kind of computer readable storage medium is additionally provided, which can
It reads to be stored with instruction in storage medium, when run on a computer, so that computer executes above-mentioned Fig. 1, Fig. 2 and Fig. 3
Any a kind of date storage method in corresponding embodiment.
The communication bus that above-mentioned electronic equipment is mentioned can be Peripheral Component Interconnect standard (Peripheral Component
Interconnect, PCI) bus or expanding the industrial standard structure (Extended Industry Standard
Architecture, EISA) bus etc..The communication bus can be divided into address bus, data/address bus, control bus etc..For just
It is only indicated with a thick line in expression, figure, it is not intended that an only bus or a type of bus.
Communication interface is for the communication between above-mentioned electronic equipment and other equipment.
Memory may include random access memory (Random Access Memory, RAM), also may include non-easy
The property lost memory (Non-Volatile Memory, NVM), for example, at least a magnetic disk storage.Optionally, memory may be used also
To be storage device that at least one is located remotely from aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit,
CPU), network processing unit (Network Processor, NP) etc.；It can also be digital signal processor (Digital Signal
Processing, DSP), it is specific integrated circuit (Application Specific Integrated Circuit, ASIC), existing
It is field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete
Door or transistor logic, discrete hardware components.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.
Each embodiment in this specification is all made of relevant mode and describes, same and similar portion between each embodiment
Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for device reality
For applying example, electronic equipment and computer readable storage medium, since it is substantially similar to the method embodiment, so description
Fairly simple, the relevent part can refer to the partial explaination of embodiments of method.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (10)
1. a kind of date storage method, which is characterized in that be applied to server, which comprises
Obtain access log file to be stored；
The log for being included to the access log file carries out fragment processing, obtains each log of the access log file
The log that affiliated fragment is included；
In a parallel fashion, the log that each affiliated fragment of log obtained is included is stored to storage equipment.
2. the method according to claim 1, wherein described carry out fragment processing to the access log file
Step, comprising:
According to the attribute of the included log of the access log file, the log for being included to the access log file divides
Piece processing.
3. according to the method described in claim 2, it is characterized in that, described according to the included log of the access log file
Attribute, to the access log file carry out fragment processing the step of, comprising:
Detect the corresponding relationship that whether there is log to be processed in mapping table, wherein the corresponding relationship is log category
Property the affiliated fragment of attribute value and log number between corresponding relationship, the mapping table records the corresponding of each log and closes
System, each corresponding storage location of the affiliated fragment of log；
If it exists, according to corresponding relationship, the log for being included to the access log file carries out fragment processing, obtains the visit
Ask the log that the affiliated fragment of each log of journal file is included, wherein the corresponding attribute value of each log, also, day
The affiliated fragment of log corresponding to will is corresponding with the attribute value of the log.
4. according to the method described in claim 3, it is characterized in that, the method also includes:
If detecting the corresponding relationship that log to be processed is not present in mapping table, log to be processed is determined using following formula
The number of the affiliated fragment of corresponding log:
P=k%n
Wherein, p is the number of the affiliated fragment of log, and k is attribute value, and n is the quantity of the affiliated fragment of log；
According to the number and the attribute value of log to be processed of the affiliated fragment of identified log, determine that the corresponding of log to be processed is closed
System；
According to identified corresponding relationship, the log for being included to the access log file carries out fragment processing, described in acquisition
The log that the affiliated fragment of each log of access log file is included, wherein the corresponding attribute value of each log, also,
The affiliated fragment of log corresponding to log is corresponding with the attribute value of the log.
5. according to the method described in claim 4, it is characterized in that, the corresponding relationship according to determined by, to the access
The log that journal file is included carries out fragment processing, and the affiliated fragment of each log for obtaining the access log file is included
Log the step of before, further includes:
Obtain the corresponding storage location of the affiliated fragment of log；
With the presence or absence of abnormal storage location in the acquired storage location of detection；
If it does, determine the corresponding affiliated fragment of log of the abnormal storage location, and by the affiliated fragment of identified log
Corresponding attribute value is distributed to preset reservation fragment, wherein the reservation fragment is the log institute for being not involved in distributive property value
Belong to fragment, and the corresponding storage location of the reservation fragment is normal；
If it does not, execute the corresponding relationship according to determined by, the log that is included to the access log file into
The processing of row fragment, the step of obtaining the log that the affiliated fragment of each log of the access log file is included.
6. according to the method described in claim 4, it is characterized in that, the corresponding relationship according to determined by, to the access
The log that journal file is included carries out fragment processing, and the affiliated fragment of each log for obtaining the access log file is included
Log the step of after, the method also includes:
Identified corresponding relationship is updated in the mapping table.
7. according to the method described in claim 4, it is characterized in that, the detection is in mapping table with the presence or absence of to be processed
Before the step of corresponding relationship of log, the method also includes:
Detect whether the storage location for having newly-increased；
If it does not, executing the step of detection whether there is the corresponding relationship of log to be processed in mapping table；
If it does, the step of executing the number for determining the affiliated fragment of log corresponding to log to be processed.
8. the method according to claim 1, wherein the log for being included to the access log file into
The processing of row fragment, it is described after the step of obtaining the log that the affiliated fragment of each log of the access log file is included
Method further include:
The log for being included to each affiliated fragment of log obtained carries out data cleansing；
It is described in a parallel fashion, by the log that each affiliated fragment of log obtained is included store to storage equipment step
Suddenly, comprising:
In a parallel fashion, the log that the affiliated fragment of each log after progress data cleansing is included is stored to storage equipment.
9. a kind of data storage device, which is characterized in that be applied to server, described device includes:
Module is obtained, for obtaining access log file to be stored；
Fragment processing module, the log for being included to the access log file carry out fragment processing, obtain the access
The log that the affiliated fragment of each log of journal file is included；
Memory module, in a parallel fashion, the log that each affiliated fragment of log obtained is included to be stored to storage
Equipment.
10. a kind of electronic equipment, which is characterized in that including processor, communication interface, memory and communication bus, wherein processing
Device, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes any method and step of claim 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711072800.4A CN110019008A (en) | 2017-11-03 | 2017-11-03 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711072800.4A CN110019008A (en) | 2017-11-03 | 2017-11-03 | Data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019008A true CN110019008A (en) | 2019-07-16 |
Family
ID=67186783
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711072800.4A Pending CN110019008A (en) | 2017-11-03 | 2017-11-03 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019008A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181965A (en) * | 2020-09-29 | 2021-01-05 | 成都商通数治科技有限公司 | MYSQL-based big data cleaning system and method for writing bottleneck into MYSQL-based big data cleaning system |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011248537A (en) * | 2010-05-25 | 2011-12-08 | Nippon Telegr & Teleph Corp <Ntt> | Distributed media processing device, method and program |
| CN102880658A (en) * | 2012-08-31 | 2013-01-16 | 电子科技大学 | Distributed file management system based on seismic data processing |
| CN103699495A (en) * | 2013-12-27 | 2014-04-02 | 乐视网信息技术（北京）股份有限公司 | Transmission device and transmission system for splitting data |
| CN104008780A (en) * | 2013-02-26 | 2014-08-27 | 中芯国际集成电路制造(上海)有限公司 | Repair method and device for memory cell |
| CN104598563A (en) * | 2015-01-08 | 2015-05-06 | 北京京东尚科信息技术有限公司 | High concurrency data storage method and device |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
| US20170242732A1 (en) * | 2015-02-19 | 2017-08-24 | Netapp, Inc. | Efficient recovery of erasure coded data |
-
2017
- 2017-11-03 CN CN201711072800.4A patent/CN110019008A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011248537A (en) * | 2010-05-25 | 2011-12-08 | Nippon Telegr & Teleph Corp <Ntt> | Distributed media processing device, method and program |
| CN102880658A (en) * | 2012-08-31 | 2013-01-16 | 电子科技大学 | Distributed file management system based on seismic data processing |
| CN104008780A (en) * | 2013-02-26 | 2014-08-27 | 中芯国际集成电路制造(上海)有限公司 | Repair method and device for memory cell |
| CN103699495A (en) * | 2013-12-27 | 2014-04-02 | 乐视网信息技术（北京）股份有限公司 | Transmission device and transmission system for splitting data |
| CN104598563A (en) * | 2015-01-08 | 2015-05-06 | 北京京东尚科信息技术有限公司 | High concurrency data storage method and device |
| US20170242732A1 (en) * | 2015-02-19 | 2017-08-24 | Netapp, Inc. | Efficient recovery of erasure coded data |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112866319B (en) * | 2019-11-28 | 2023-10-13 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112181965A (en) * | 2020-09-29 | 2021-01-05 | 成都商通数治科技有限公司 | MYSQL-based big data cleaning system and method for writing bottleneck into MYSQL-based big data cleaning system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210326885A1 (en) | Method and Apparatus of Identifying a Transaction Risk | |
| US10262145B2 (en) | Systems and methods for security and risk assessment and testing of applications | |
| CN110377667B (en) | Correlation map display method, device, computer equipment and storage medium | |
| US20190065738A1 (en) | Detecting anomalous entities | |
| US20180165258A1 (en) | Methods for improved auditing of web sites and devices thereof | |
| EP3152653A2 (en) | Techniques to isolating a portion of an online computing service | |
| CN104346365A (en) | Device and method for determining specific service associated logs | |
| CN109189758B (en) | Operation and maintenance flow design method, device and equipment, operation method, device and host | |
| CN110096363A (en) | A kind of correlating method and device of network event and process | |
| CN107729897A (en) | A kind of text maninulation method, apparatus and terminal | |
| CN114095567A (en) | Data access request processing method and device, computer equipment and medium | |
| CN104468222A (en) | Method, device and system for reporting log information | |
| CN106649210B (en) | Data conversion method and device | |
| CN108200023A (en) | Unaware authentication method and device | |
| CN110019008A (en) | Data storage method and device | |
| CN109377383A (en) | Product data synchronous method, device, computer equipment and storage medium | |
| CN110943887B (en) | Probe scheduling method, device, equipment and storage medium | |
| CN111143526B (en) | Method and device for generating and controlling configuration information of counsel service control | |
| CN111083173B (en) | Dynamic defense method in network communication based on openflow protocol | |
| CN110659096B (en) | Method, device, system and storage medium for displaying network topological graph | |
| CN107453950A (en) | A kind of information processing method and monitoring system | |
| CN106293536A (en) | input/output I/O dispatching method and device | |
| CN111753162A (en) | Data crawling method, device, server and storage medium | |
| CN112749159A (en) | Form generation method and device, electronic equipment and storage medium | |
| CN109542798A (en) | A kind of method, apparatus and electronic equipment dynamically distributing physical address |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |