CN110019054A - Log De-weight method and system, content distribution network system - Google Patents
Log De-weight method and system, content distribution network system Download PDFInfo
- Publication number
- CN110019054A CN110019054A CN201711487741.7A CN201711487741A CN110019054A CN 110019054 A CN110019054 A CN 110019054A CN 201711487741 A CN201711487741 A CN 201711487741A CN 110019054 A CN110019054 A CN 110019054A
- Authority
- CN
- China
- Prior art keywords
- data set
- log
- log data
- title
- daily record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
This application provides a kind of log De-weight method and system, content distribution network system, wherein log De-weight method, comprising: obtain log data set；In the daily record data set identifier of the log data set, log data set title and serial number are determined；Wherein, the serial number is used to distinguish the different log data sets with same log dataset name；If in the corresponding storage state mark array of the log data set title, storage state corresponding with serial number mark expression storing daily record data collection, refusing to store the log data set.The application no longer executes deduplication operation one by one to daily record data, but carries out log deduplication operation with log data set.Log data set at most may include 4096 daily record datas, it is possible to disposably concentrate a plurality of daily record data to execute deduplication operation daily record data.The cpu resource consumption to log collection equipment can be reduced in this way, and improves log deduplication operation efficiency.
Description
Technical field
This application involves field of communication technology more particularly to log De-weight methods and system, content distribution network system.
Background technique
Currently, most of system includes log collection equipment and multiple log generating devices, log generating device can be with
The daily record data of itself generation is collected, and sends daily record data to log collecting device, so as to log collection equipment utilization log
Data execute other treatment processes.
But it is likely to result in log generating device due to network jitter etc. and repeats to send asking for same daily record data
Topic, this will lead in log collection equipment comprising repeating daily record data.In order to enable subsequent can be used accurately and effectively log
Data can execute deduplication operation to daily record data.
Currently, the scheme of log deduplication operation are as follows: log generating device is every daily record data addition mark (daily record data
ID), so that log collection equipment can carry out deduplication operation one by one for the daily record data of duplicate marking.
Since the daily record data scale of construction is huge in log collection equipment, so execution journal deduplication operation scheme one by one, not only
A large amount of cpu resources of log collection equipment can be consumed, and will lead to the efficiency of log collection equipment execution journal deduplication operation
It is lower.
Summary of the invention
In consideration of it, the application provides log De-weight method and system, deduplication operation can be carried out to log data set, thus
It reduces and the cpu resource of log collection equipment is consumed, and improve log deduplication operation efficiency.
To achieve the goals above, this application provides following technical characteristics:
A kind of log De-weight method is applied to log collection equipment, which comprises
Obtain log data set；Wherein, the log data set includes the same process of log generating device at the first time
The daily record data generated in section, and, daily record data set identifier；
In the daily record data set identifier of the log data set, log data set title and serial number are determined；Wherein, institute
Serial number is stated for distinguishing the different log data sets with same log dataset name；
In the corresponding storage state mark array of the log data set title, storage corresponding with the serial number is determined
Status indicator；
If storage state mark corresponding with the serial number indicates storing daily record data collection state, refuse described in storage
Log data set.
Optionally, described in the corresponding storage state mark array of the log data set title, the determining and sequence
Number corresponding storage state mark, comprising:
In log data set name list, the log data set title has been searched whether；
If finding the log data set title, number is identified in the corresponding storage state of the log data set title
In group, storage state mark corresponding with the serial number is determined.
Optionally, further includes:
If not finding the log data set title, the log is added in the log data set name list
Dataset name；
It constructs storage state corresponding with the log data set title and identifies array, and, storage state identifies in array
Storage state mark indicate not stored log data set state；
The log data set is stored, and changes storage state mark corresponding with serial number in the storage state mark array
Know, so that storage state mark indicates storing daily record data collection state.
Optionally, the corresponding log data set of a first time period, several first time period form second time period,
And the corresponding log data set title of several first time period is consistent in second time period；
The then serial number in the daily record data set identifier has same log data set name for distinguishing in second time period
The log data set of title.
Optionally, the log data set title in the daily record data set identifier, comprising:
The device identification of the log generating device；
The process identification (PID) of process described in the log generating device；
The corresponding system timestamp of the second time period；
Wherein, which is the product of integer and the second time period, and the integer is to device log number
It is obtained according to collecting after quotient of the corresponding system timestamp with the second time period is rounded downwards.
Optionally, the serial number in the daily record data set identifier includes:
The log generating device is the log data set 1 to the numerical value determined between greatest measure, and, described the
Different log data sets correspond to different serial numbers in two periods；
Wherein, the greatest measure rounds up to obtain for the quotient of the second time period and the first time period.
Optionally, the storage state mark array includes bit array, and the bit in bit array is corresponding with serial number,
And the data value of bit indicates storage state mark.
A kind of log De-weight method is applied to log generating device, which comprises
Obtain this equipment log data set that same process generates in first time period；
Daily record data set identifier is added for the log data set；Wherein, the daily record data set identifier includes log number
According to collection title, and, for distinguishing the serial number of the different log data sets with same log dataset name；
Log data set is sent to log collecting device, so that log collection equipment is in the log number of the log data set
According to log data set title and serial number in set identifier, is determined, in the corresponding storage state mark of the log data set title
In array, storage state mark corresponding with the serial number is determined；If storage state mark corresponding with the serial number indicates
Storing daily record data collection state is then refused to store the log data set.
It is optionally, described to add daily record data set identifier for the log data set, comprising:
Determine the corresponding log data set title of the log data set；
Determine the corresponding serial number of the log data set；
The character string that the log data set title and the serial number are formed, is determined as the daily record data set identifier；
The daily record data set identifier is added for the log data set.
Optionally, the corresponding log data set title of the determination log data set, comprising:
Obtain the device identification of this equipment, the process identification (PID) of the process and system timestamp；
The quotient for calculating the system timestamp Yu the second time period obtains whole after being rounded downwards to the quotient
Number, calculates the product of the integer Yu the second time period；
The character string that the device identification, the process identification (PID) and the product are formed, is determined as the daily record data
Collect title.
Optionally, the corresponding serial number of the determination log data set, comprising:
In the second time period, by the first initial count value, it is every to generate log data set and be incremented by 1, directly
Increase preset times to count value；Alternatively,
In the second time period, by the second initial count value, one log data set of every generation successively decreases 1, directly
Successively decrease preset times up to count value；
Wherein, the preset times are rounded up to obtain by the quotient of the second time period and the first time period.
A kind of content distribution network system, comprising: central node and the fringe node being connected with the central node, it is described
Fringe node includes several servers；
Server is the log data set for collecting the log data set that same process generates in first time period
Daily record data set identifier is added, sends log data set to streaming message system；Wherein, the daily record data set identifier includes more
The log data set title that a log data set is all made of, and, for distinguish with same log dataset name not on the same day
The serial number of will data set；
Central node, for obtaining log data set from the streaming message system, in the log of the log data set
In data set identification, log data set title and serial number are determined；In the corresponding storage state mark of the log data set title
Know in array, determines storage state mark corresponding with the serial number；If storage state mark corresponding with the serial number indicates
Storing daily record data collection state is then refused to store the log data set.
A kind of log machining system, comprising:
Log generating device is the log for collecting the log data set that same process generates in first time period
Data set adds daily record data set identifier, sends log data set to log collecting device；Wherein, the daily record data set identifier
Including the log data set title that multiple log data sets are all made of, and, for distinguishing with same log dataset name
The serial number of different log data sets；
Log collection equipment, for obtaining log data set, in the daily record data set identifier of the log data set, really
Make log data set title and serial number；In the log data set title corresponding storage state mark array, determine with
The corresponding storage state mark of the serial number；If storage state mark corresponding with the serial number indicates storing daily record data collection
State is then refused to store the log data set.
A kind of log De-weight method is applied to log collection equipment, which comprises
Obtain log data set；Wherein, the log data set includes the same process of log generating device at the first time
The daily record data and log data set title generated in section；
Determine the log data set title of the log data set；
In existing log data set name list, the log data set title has been searched whether；
If finding the log data set title, refuse to store the log data set.
Optionally, further includes:
If not finding the log data set title, the log data set is stored；
In the log data set name list, the log data set title is added.
Optionally, the log data set title of the determination log data set, comprising: concentrated in the daily record data
Extract the log data set title；
The wherein log data set title, comprising:
The device identification of the log generating device；
The process identification (PID) of process described in the log generating device；
The log generating device generates the system timestamp during log data set.
A kind of log De-weight method is applied to log generating device, which comprises
Obtain this equipment log data set that same process generates in first time period；
Log data set title is generated, and adds log data set title for the log data set；
It sends comprising log data set to log collecting device, so that log collection equipment is in existing daily record data set name
Claim to have searched whether the log data set title in list, if finding the log data set title, refuses to store institute
State log data set.
It is optionally, described to add log data set title for the log data set, comprising:
Obtain the device identification of described equipment, the process identification (PID) of the process and system timestamp；
The character string that the device identification, the process identification (PID) and the system timestamp are formed, is determined as the day
Will dataset name；
The log data set title is added for log data set.
A kind of log machining system, comprising:
Log generating device is the log for collecting the log data set that same process generates in first time period
Data set adds log data set title；The log data set is sent to log collecting device；
Log collection equipment sends log data set for obtaining log generating device；In existing daily record data set name
Claim to search whether the log data set title in the log data set in list；If so, refusal stores the log number
According to collection.
A kind of content distribution network system, comprising: central node and the fringe node being connected with the central node, it is described
Fringe node includes several servers；
Server is the log data set for collecting the log data set that same process generates in first time period
Log data set title is added, sends log data set to streaming message system；
Central node, for obtaining log data set from streaming message system；In existing log data set name list
In, search whether the log data set title in the log data set；If so, refusal stores the log data set.
A kind of log duplicate removal device, is integrated in log collection equipment, comprising:
First acquisition unit, for obtaining log data set；Wherein, the log data set includes that log generating device is same
The daily record data and log data set title that one process generates in first time period；
Determination unit, for determining the log data set title of the log data set；
Searching unit, for having searched whether the log data set name in existing log data set name list
Claim；
Refuse storage unit, if refusing to store the log data set for finding the log data set title.
A kind of log duplicate removal device, is integrated in log generating device, comprising:
Second acquisition unit, for obtaining this equipment log data set that same process generates in first time period；
First addition mark unit, for generating log data set title and adding daily record data for the log data set
Collect title；
First transmission unit is set for sending log data set to the log collection comprising the log data set title
It is standby, so that log collection equipment has searched whether the log data set title in log data set name list, if searching
To the log data set title, then refuse to store the log data set.
A kind of log duplicate removal device, is integrated in log collection equipment, comprising:
Third acquiring unit, for obtaining log data set；Wherein, the log data set includes that log generating device is same
The daily record data that one process generates in first time period, and, daily record data set identifier；
Determination unit, for determining log data set title in the daily record data set identifier of the log data set
And serial number；Wherein, the serial number is used to distinguish the different log data sets with same log dataset name；
Determine that storage state identifies unit, for identifying array in the corresponding storage state of the log data set title
In, determine storage state mark corresponding with the serial number；
Refuse storage unit, if indicating storing daily record data collection shape for storage state corresponding with serial number mark
State is then refused to store the log data set.
A kind of log duplicate removal device, is integrated in log generating device, comprising:
4th acquiring unit, for obtaining this equipment log data set that same process generates in first time period；
Second addition mark unit, for adding daily record data set identifier for the log data set；Wherein, the log
Data set identification includes log data set title, and, for distinguishing the different daily record datas with same log dataset name
The serial number of collection；
Second transmission unit, for sending log data set to log collecting device, so that log collection equipment is described
In the daily record data set identifier of log data set, log data set title and serial number are determined, in the log data set title
In corresponding storage state mark array, storage state mark corresponding with the serial number is determined；If corresponding with the serial number
Storage state mark indicates storing daily record data collection state, then refuses to store the log data set.Pass through the above technology hand
Section, may be implemented it is following the utility model has the advantages that
The application no longer executes deduplication operation one by one to daily record data, but carries out log duplicate removal behaviour with log data set
Make.Log data set at most may include 4096 daily record datas, it is possible to disposably concentrate a plurality of log to daily record data
Data execute deduplication operation.The cpu resource consumption to log collection equipment can be reduced in this way, and improves log deduplication operation effect
Rate.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of application for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is a kind of structural schematic diagram of log machining system disclosed in the embodiment of the present application；
Fig. 2 is a kind of flow chart of log De-weight method disclosed in the embodiment of the present application；
Fig. 3 is the flow chart of another log De-weight method disclosed in the embodiment of the present application；
Fig. 4 is the flow chart of another log De-weight method disclosed in the embodiment of the present application；
Fig. 5 is the schematic diagram of bit array disclosed in the embodiment of the present application；
Fig. 6 is the structural schematic diagram of the embodiment of the present application disclosure distributing network system.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of embodiments of the present application, instead of all the embodiments.It is based on
Embodiment in the application, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall in the protection scope of this application.
00 divides 00 second (Beijing time 1970 01 when system timestamp refers to Greenwich Mean Time 1970 01 month 01 day 00
00 divides 00 second when the moon 01 day 08) it rises to present total number of seconds.For example, 1501925027 seconds, being converted into Beijing time is 2017/8/
5 17:23:47。
The application no longer executes deduplication operation one by one to daily record data, but carries out log duplicate removal behaviour with log data set
Make.Log data set at most may include 4096 daily record datas, it is possible to disposably concentrate a plurality of log to daily record data
Data execute deduplication operation.The cpu resource consumption to log collection equipment can be reduced in this way, and improves log deduplication operation effect
Rate.
For the ease of skilled in the art realises that, provide a kind of log deduplication operation system, including multiple logs are raw
Forming apparatus 100 and log collection equipment 200.
According to one embodiment provided by the present application, a kind of log De-weight method embodiment one is provided.Referring to fig. 2, have
Body the following steps are included:
Step S201: log generating device obtains this equipment daily record data that a process generates in first time period
Collection.
Log generating device can open multiple processes simultaneously, be consistent for the implementation procedure of each process, because
This is illustrated by taking a process as an example.
The daily record data of process generation can constantly be collected and be cached to log generating device, reach at the first time in acquisition time
Duan Hou, then a plurality of daily record data by the process in first time period forms log data set.Log data set at most can be with
Including 4096 logs.
For example, then log generating device can collect day of the process in 30 seconds so that first time period is 30 seconds as an example
Will data, and a plurality of daily record data in 30 seconds is formed into log data set.
Step S202: log generating device 100 is that log data set constructs daily record data set identifier.Wherein, the log
Data set identification includes log data set title.
Since the process of log generating device is different in the daily record data of different first time period, in order to uniquely indicate this
The log data set that the first time period of log generating device 100 under the process generates, log generating device 100 can obtain
The device identification of this equipment, the process identification (PID) of process and current system timestamp.
Then, the character string device identification, the process identification (PID) and the system timestamp formed, is determined as institute
State log data set title.
For example, device identification is indicated using H, process identification (PID) is indicated using P, system timestamp is indicated using T, then
Log data set title name=CONCAT (H, P, T)；Wherein, CONCAT is composition character string function.
Step S203: log generating device 100 is that the log data set adds daily record data set identifier.
Determining daily record data set identifier is added to daily record data and concentrated by log generating device 100.
Step S204: log generating device 100 sends the log data set comprising the daily record data set identifier to log
Collecting device 200.
Step S205: log collection equipment 200 has searched whether the daily record data in log data set name list
Collect title, if so, S206 is entered step, if otherwise entering step S207.
Log collection equipment 200 record can be somebody's turn to do stored log data set in log data set name list
Log data set title.That is, log data set name list includes a log data set title, then it represents that log collection equipment
200 stored corresponding log data sets of log data set title.
Therefore, log collection equipment 200 has searched whether the log data set name in log data set name list
Claim, if so, then indicating log collection equipment stored mistake log data set, enters step S206.If not, then it represents that log
The collecting device not stored mistake log data set, enters step S207.
Step S206: the refusal of log collection equipment 200 stores the log data set.
In order to avoid repeating to store identical log data set, log collection equipment 200 can refuse storage log collection and set
It is standby.For example, log collection equipment, which can execute, abandons the log data set, specifically handled alternatively, deleting the log data set etc.
Operation, to refuse storing daily record data collection.
Step S207: the log data set corresponding with the log data set title is stored, and in log data set
The log data set title is added in name list.
In the case where the not stored log data set of log collection equipment 200, log collection equipment can store log number
According to collection.The log data set is stored again in order to avoid subsequent, the log number can be added in log data set name list
According to collection title.
The present embodiment carries out log deduplication operation with log data set, and log data set at most may include 4096 logs
Data, so the application disposably can concentrate a plurality of daily record data to execute deduplication operation daily record data.It can reduce in this way
Cpu resource consumption to log collection equipment, and improve log deduplication operation efficiency.
The application also provides a kind of log duplicate removal device, is integrated in log collection equipment 200, comprising:
First acquisition unit, for obtaining log data set；Wherein, the log data set includes that the same process of equipment exists
The daily record data and log data set title generated in first time period；
Determination unit, for determining the log data set title of the log data set；
Searching unit, for having searched whether the log data set name in existing log data set name list
Claim；
Refuse storage unit, if refusing to store the log data set for finding the log data set title.
Storage unit, if storing the log data set for not finding the log data set title；Described
In log data set name list, the log data set title is added.
Wherein, the log data set title, comprising: the device identification of the equipment；Process described in the equipment
Process identification (PID)；The equipment generates the system timestamp during log data set.
The application also provides a kind of log duplicate removal device, comprising:
Second acquisition unit, for obtaining the equipment log data set that same process generates in first time period；
First addition mark unit, for generating log data set title and adding daily record data for the log data set
Collect title；
First transmission unit, for sending the log data set comprising the log data set title.
Wherein, the first addition mark unit, be specifically used for: obtaining the device identification of the equipment, the process
Process identification (PID) and system timestamp；The character string that the device identification, the process identification (PID) and the system timestamp are formed,
It is determined as the log data set title；The log data set title is added for log data set.
Specific implementation about log duplicate removal device may refer to embodiment shown in Fig. 2, and details are not described herein.
Log collection equipment 200 needs to safeguard log data set name list in example 1, due to log data set
Enormous amount, so the enormous amount of log data set title, this causes log data set name list to occupy biggish storage
Space.
In order to reduce the memory space of log data set name list occupancy, this application provides a kind of log De-weight methods
Embodiment two.
Referring to Fig. 3, specifically includes the following steps:
Step S301: log generating device 100 obtains this equipment log number that a process generates in first time period
According to collection.This step is consistent with the process of step S201, and details are not described herein.
Step S302: log generating device 100 is that the log data set adds daily record data set identifier；Wherein, described
Daily record data set identifier includes log data set title and serial number, and the serial number is for distinguishing the equipment in second time period
Log data set with same log dataset name.Wherein, the second time period includes several first time period.
Each first time period corresponds to a log data set title in embodiment one, in order to reduce daily record data set name
Second time period is arranged in the quantity of title, the present embodiment, and second time period includes several first time period；It is each in the present embodiment
Second time period corresponds to a log data set title, that is to say, that: several first time period in second time period have
There is identical log data set title.
For example, second time period is for 600 seconds, when being directed to second in example 1 with first time period for 30 seconds
Between section 600 seconds, need to be arranged 600/30=20 log data set title.In the present embodiment, 1 is arranged in second time period
A log data set title, 20 log data sets are all made of the log data set title.
In order to enable each log data set is all made of the purpose of same log data set title, this reality in second time period
It applies example and provides the process of determining log data set title.
Referring to fig. 4, comprising the following steps:
Step S401: log generating device 100 obtains the device identification of the equipment, the process identification (PID) of the process and is
System timestamp.
Log generating device obtains device identification, the process identification (PID) of process in equipment, and obtains after obtaining log data set
Take current system timestamp.
Step S402: log generating device 100 calculates the quotient of the system timestamp Yu the second time period, to institute
The integer obtained after quotient is rounded downwards is stated, the product of the integer Yu the second time period is calculated.
Current system timestamp is no longer directly used in the present embodiment two, but, computing system timestamp and when second
Between section quotient, and to quotient downwards be rounded obtain integer, the integer in second time period can be made after being rounded to quotient
It is consistent.
Then, in the product for calculating integer and second time period.Using product as when system unified in second time period
Between stab.Above-mentioned text is organized into formula i.e. are as follows: [T/D] * D, wherein T is system timestamp, D is second time period, and [x] is downward
The function of rounding.
For example, by taking first time period is 30 seconds, second time period is 15019200-15019799 this 600 seconds as an example
(15019200=25032*600,15019799=25033*600-1) can be executed repeatedly raw in 30 seconds in this 600 seconds
At the process of log data set.
It is assumed that the corresponding current system timestamp of log data set is 15019248 seconds (25032*600 < 15019248
< 25033*600), then [15019248/600] * 600=25032*600=15019200.That is, in 15019200-15019799
In this 600 seconds, no matter system timestamp is how many, 15019200 (25032*600) are obtained after calculating according to this step；This when
Between stamp be the unified system timestamp of the second time period.
Step S403: the word that log generating device 100 forms the device identification, the process identification (PID) and the product
Symbol string, is determined as the log data set title.
For example, by taking device identification is H, process identification (PID) P, product are [T/D] * D as an example, then log data set title name
=CONCAT (H, P, [T/D] * D)；Wherein, CONCAT is composition character string function.
Since device identification is consistent, process identification (PID) is consistent, and product is also consistent, so true according to this step
Fixed log data set title is also consistent.That is, the title of each log data set is consistent in second time period.
Since the title of log data sets multiple in second time period is consistent, in order to distinguish each daily record data
Collection adds an attribute: serial number again in daily record data set identifier.Serial number is used to distinguish having of generating in second time period
The log data set of same log dataset name.
Multiple log data sets can be generated in second time period, particular number can be to second time period and first time period
Quotient round up to obtain.Subsequent log collection equipment can construct storage state mark number according to the number of log data set
Group.The quantity of storage state mark is greater than or equal to log data set number.
Storage state mark array includes multiple storage state marks.There are two state, a shapes for storage state mark tool
Storing daily record data collection state, another state indicate not stored status data collection state to state.
Under initial situation, storage state identifies the not stored state of each corresponding storage state mark expression of serial number in array
Data set.
It can determine whether out log data set title and serial number after receiving log data set；And it is corresponding in log data set title
Storage state mark array in, determine that corresponding with serial number storage state identifies；If storage corresponding with the serial number
Status indicator indicates storing daily record data collection state, then refuses to store the log data set.
Storage state mark array can realize that the bit quantity of bit array is greater than or equal to using bit array
Log data set number.It is subsequent by taking bit array as an example, to storage state mark array specific implementation be described.About than
The use of special array may refer to step S306-S308.
For example, for it can produce 20 log data sets in second time period, then in subsequent builds bit array
Bit quantity minimum value is 20, it can bit arrays more than 20 and 20 bits of building.
The application provides two kinds of implementations for determining the corresponding serial number of the log data set:
The first implementation: in the second time period, by the first initial count value, one log of every generation
Data set is incremented by 1, until count value increases preset times.Wherein, when preset times are by the second time period and described first
Between the quotient of section round up to obtain.
For example, then the first initial count value is every to generate one since 0 by taking the bit array for constructing 20 bits as an example
A log data set serial number is incremented by 1, and serial number is distributed to log data set；After generating log data set again, serial number is incremented by again
1, serial number is distributed into log data set, and so on.
Due to being 600 seconds in second time period, first time period is 30 seconds, 20 can be generated in first time period
Therefore a log data set after serial number is incremented by 20 times, then is completed to distribute not in the second time period for each log data set
With the purpose of serial number.
That is, assigning serial number 1 for log data set 1, serial number 2 ... ... Wie log data set 20 is assigned for log data set 2
Serial number 20 is assigned, so that each log data set has different serial numbers.
Second of implementation: in the second time period, by the second initial count value, one log of every generation
Data set successively decreases 1, and through count value is successively decreased preset times.Wherein, when preset times are by the second time period and described first
Between the quotient of section round up to obtain.
It is log number for example, assigning serial number 20 by taking the bit array for constructing 20 bits as an example for log data set 1
Serial number 1 is assigned as log data set 20 according to 2 imparting serial numbers 19 ... ... are integrated, so that each log data set is with different
Serial number.
The method is similar with the first implementation, the difference is that being counted with decreasing fashion, which is repeated no more.
Certainly, other than above two implementation, serial number can also be realized using other way, as long as can be with area
Divide the log data set with same log dataset name generated in second time period.
It is understood that is, after initial count value counting preset times, just being opened after a second time period
Begin another second time period.Serial number can restart to count from initial count value again in another second time period,
Different serial numbers is assigned to assign different log data sets for each log data set in another second time period.
Log generating device 100 is determining the corresponding log data set title of the log data set and is determining the log
After the corresponding serial number of data set, the character string that the log data set title and the serial number are formed is determined as the log
Data set identification.By taking serial number is using seq as an example, daily record data set identifier=CONCAT (name, Seq).
Be then returned to Fig. 3 and enter step S303: log generating device 100 sends the day comprising the daily record data set identifier
Will data set is to log collecting device 200.
Step S304: log collection equipment 200 obtains the log data set comprising daily record data set identifier, and parses log
Data set obtains log data set title and serial number.
Log collection equipment 200 obtains the log data set comprising daily record data set identifier, and daily record data set identifier includes
Log data set title name and serial number seq.
Step S305: log collection equipment 200 searches log data set title in log data set name list, if searching
To S306 is entered step, S307 is entered step if not finding.
Log collection equipment 200 searches log data set title in log data set name list, if not finding,
Illustrate the not stored mistake of log data set title, enters step S306；If finding, the log data set title is illustrated
It stored, and entered step S307.
Step S306: if not there is bit array corresponding with the log data set title, the daily record data is stored
Collection constructs bit array corresponding with the daily record data title；Each bit initial value indicates not deposit in bit array
Log data set state is stored up, the numerical value of serial number corresponding bit position described in the bit array is modified, so that modified numerical value
Indicate storing daily record data collection state.
If not there is bit array corresponding with the log data set title, illustrate also not stored mistake and log data set
Therefore the corresponding log data set of title stores the log data set.
Then, according to preset number of bits, (the bit quantity of bit array is greater than or equal to second time period
Interior log data set number), bit array corresponding with log data set title is constructed, and assign expression for each bit
The not stored initial value for crossing log data set state.
For example, building is corresponding with log data set title name, the bit array of 20 bits, and by 20 bits
Position assigns " 0 "；Wherein, " 1 " indicates storing daily record data collection state, and " 0 " indicates not stored log data set state.
The serial number in daily record data set identifier is obtained, the number of serial number corresponding bit position described in the bit array is set
Value, so that modified numerical value indicates storing daily record data collection state.For example, by taking serial number 1 as an example, in order to indicate the log
The corresponding log data set of the serial number is stored under dataset name, is set as " 1 " for the bit array the 1st.
Step S307: log collection equipment 200 adds the log data set title in log data set name list, and
Construct the corresponding relationship of log data set title and bit array.
Log collection equipment can add log data set title from log data set name list, to indicate the log
Dataset name is stored excessively primary, has bit array corresponding with the log data set title.
The corresponding relationship for constructing log data set title and bit data group, can be based on daily record data set name so as to subsequent
Title finds bit array.
Step S308: log collection equipment 200 finds log data set title in log data set name list, then really
Fixed bit array corresponding with log data set title, and determine the numerical value of serial number corresponding bit position in bit array.
Log collection equipment 200 finds log data set title in log data set name list, then is based on preparatory structure
The corresponding relationship built determines bit array corresponding with log data set title.Then, the sequence in daily record data set identifier is obtained
Number, the determining numerical value with serial number corresponding bit position from bit array.
Step S309: in the case where the numerical value of serial number corresponding bit position indicates storing daily record data collection, refusal
Store the log data set.
For example, if the numerical value of serial number corresponding bit position is " 1 ", then it represents that storing daily record data collection state, in order to keep away
Exempt to repeat storing daily record data collection, refuses storing daily record data collection again.
Step S310: in the case where the numerical value of serial number corresponding bit position indicates not stored log data set, storage
The log data set；The numerical value of serial number corresponding bit position described in the bit array is modified, so that modified numerical tabular
Show storing daily record data collection state.
In order to make it easy to understand, being illustrated referring to Fig. 5 with an example:
First time period, second time period and the bit quantity for determining bit array are determined by technical staff in advance,
It is 30 seconds with first time period, second time period 600, for the bit quantity of bit array is 20:
Log generating device 100 can be in first 30 seconds generation log data sets 1, its log data set of second time period
Mark 1 is (name, seq=1), sends log data set 1 to log collecting device.
Log collection equipment 200 receives log data set 1, finds after checking not corresponding with log data set title
Therefore bit array constructs 20 bit arrays, and establish corresponding relationship with log data set title, in bit array
The initial value of each bit is " 0 ", for indicating that each daily record data set in second time period is not stored.
Then, log collection equipment storing daily record data collection 1, and (1 is log data set 1 by the 1st in bit array
Serial number) it is set as " 1 ", indicate that the log data set 1 in second time period has stored.
If log collection equipment 200 receives log data set 1 again, number corresponding with serial number 1 in bit array is determined
Value is " 1 ", then log collection equipment 200 just learns stored log data set 1, therefore no longer storing daily record data collection 1 excessively.
Log generating device 100 is understood in next 30 seconds generation log data sets 2, its daily record data set identifier 2
(name, seq=2) ... is (name, seq in the last one 30 seconds generation log data sets 20, its daily record data set identifier 20
=20).Log generating device can send log data set 1- log data set 20 one by one.
It is subsequent to be set as " 1 " for the 2nd of bit array after receiving log data set 2, receiving log data set 3
Afterwards, be set as " 1 " for the 3rd of bit array ... ... receive log data set 20 after, the 20th of bit array is set as
“1”。
A kind of content distribution network system is described below, referring to Fig. 6, specifically includes: central node 300 and with the center
The connected fringe node 400 of node 300, the fringe node 400 include several servers 401.
Two kinds of implementations of the log De-weight method in content distribution network system are described below:
The first implementation:
Server 401 is the daily record data for collecting the log data set that a process generates in first time period
Collection addition daily record data set identifier；Wherein, the daily record data set identifier includes log data set title；Sending includes the day
The log data set of will data set identification is to streaming message system；
Central node 300, for obtaining the log data set comprising daily record data set identifier from streaming message system；?
In some log data set name lists, the log data set title has been searched whether；If so, refusal stores the log
Data set.
Server is equivalent to log generating device in embodiment one shown in Fig. 2 in the present embodiment, and central node is equivalent to
Log collection equipment.Therefore, the specific implementation of the first implementation may refer to retouch with embodiment one shown in Fig. 2
Content is stated, details are not described herein.
Second of implementation:
Server 401 is the daily record data for collecting the log data set that same process generates in first time period
Collection addition daily record data set identifier, sends log data set to streaming message system；Wherein, the daily record data set identifier includes
The log data set title that multiple log data sets are all made of, and, for distinguishing the difference with same log dataset name
The serial number of log data set；
Central node 300, for obtaining log data set from the streaming message system, in the day of the log data set
In will data set identification, log data set title and serial number are determined；In the corresponding storage state of the log data set title
It identifies in array, determines storage state mark corresponding with the serial number；If storage state corresponding with the serial number identifies table
Show storing daily record data collection state, then refuses to store the log data set.
Server is equivalent to log generating device in embodiment two shown in Fig. 3 in the present embodiment, and central node is equivalent to
Log collection equipment.Therefore, the specific implementation of second of implementation may refer to retouch with embodiment two shown in Fig. 3
Content is stated, details are not described herein.
This application provides a kind of log duplicate removal devices, comprising:
Third acquiring unit, for obtaining log data set；Wherein, the log data set includes that log generating device is same
The daily record data that one process generates in first time period, and, daily record data set identifier；
Determination unit, for determining log data set title in the daily record data set identifier of the log data set
And serial number；Wherein, the serial number is used to distinguish the different log data sets with same log dataset name；
Determine that storage state identifies unit, for identifying array in the corresponding storage state of the log data set title
In, determine storage state mark corresponding with the serial number；
Refuse storage unit, if indicating storing daily record data collection shape for storage state corresponding with serial number mark
State is then refused to store the log data set.Wherein it is determined that storage state identifies unit, specifically include: searching unit is used for
In log data set name list, the log data set title has been searched whether；It determines that storage state identifies unit, is used for
If finding the log data set title, in the corresponding storage state mark array of the log data set title, really
Fixed storage state mark corresponding with the serial number.
Storage unit, if for not finding the log data set title, in the log data set name list
The middle addition log data set title；It constructs storage state corresponding with the log data set title and identifies array, and, it deposits
Storage state mark in storage status indicator array indicates not stored log data set state；Store the log data set；
And storage state mark corresponding with serial number in the storage state mark array is changed, so that storage state mark indicates to have deposited
Store up log data set state.
Wherein, the corresponding log data set of a first time period, several first time period form second time period,
And the corresponding log data set title of several first time period is consistent in second time period；Then in the daily record data set identifier
Serial number, for distinguish in second time period with same log dataset name log data set.
Wherein, the log data set title in the daily record data set identifier, comprising: the device identification of the equipment；Institute
State the process identification (PID) of process described in log generating device；The corresponding system timestamp of the second time period；Wherein, the system
Timestamp is the product of integer and the second time period, and the integer is to the corresponding system timestamp of device log data collection
It is obtained after being rounded downwards with the quotient of the second time period.
Serial number in the daily record data set identifier includes: that the equipment is the log data set 1 to greatest measure
Between the numerical value that determines, and, different log data sets correspond to different serial numbers in the second time period；Wherein, the maximum number
Value rounds up to obtain for the quotient of the second time period and the first time period.
The storage state mark array includes bit array, and the bit in bit array is corresponding with serial number, and, bit
The data value of position indicates storage state mark.
Present invention also provides a kind of log duplicate removal devices characterized by comprising
4th acquiring unit, for obtaining this equipment log data set that same process generates in first time period；
Second addition mark unit, for adding daily record data set identifier for the log data set；Wherein, the log
Data set identification includes log data set title, and, for distinguishing the different daily record datas with same log dataset name
The serial number of collection；
Second transmission unit, for sending log data set to log collecting device, so that log collection equipment is described
In the daily record data set identifier of log data set, log data set title and serial number are determined, in the log data set title
In corresponding storage state mark array, storage state mark corresponding with the serial number is determined；If corresponding with the serial number
Storage state mark indicates storing daily record data collection state, then refuses to store the log data set.
Wherein, the second addition mark unit, comprising:
Determine the corresponding log data set title of the log data set；Determine the corresponding serial number of the log data set；
The character string that the log data set title and the serial number are formed, is determined as the daily record data set identifier；For the day
Will data set adds the daily record data set identifier.
Wherein, the corresponding log data set title of the determination log data set, comprising:
Obtain the device identification of the equipment, the process identification (PID) of the process and system timestamp；When calculating the system
Between stamp and the quotient of the second time period, obtain integer after being rounded downwards to the quotient, calculate the integer and described the
The product of two periods；The character string that the device identification, the process identification (PID) and the product are formed, is determined as the day
Will dataset name.
Wherein, the corresponding serial number of the determination log data set, comprising: in the second time period, by first
Initial count value starts, every to generate log data set incremental 1, until count value increases preset times；Alternatively, described
In two periods, by the second initial count value, one log data set of every generation successively decreases 1, and through count value is successively decreased default
Number；
Wherein, the preset times are rounded up to obtain by the quotient of the second time period and the first time period.
Specific implementation about log duplicate removal device can be found in embodiment shown in Fig. 3, and details are not described herein.
If function described in the present embodiment method is realized in the form of SFU software functional unit and as independent product pin
It sells or in use, can store in a storage medium readable by a compute device.Based on this understanding, the embodiment of the present application
The part of the part that contributes to existing technology or the technical solution can be embodied in the form of software products, this is soft
Part product is stored in a storage medium, including some instructions are used so that calculating equipment (it can be personal computer,
Server, mobile computing device or network equipment etc.) execute all or part of step of each embodiment the method for the application
Suddenly.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), deposits at random
The various media that can store program code such as access to memory (RAM, Random Access Memory), magnetic or disk.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with it is other
The difference of embodiment, same or similar part may refer to each other between each embodiment.
The foregoing description of the disclosed embodiments makes professional and technical personnel in the field can be realized or use the application.
Various modifications to these embodiments will be readily apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the application.Therefore, the application
It is not intended to be limited to the embodiments shown herein, and is to fit to and the principles and novel features disclosed herein phase one
The widest scope of cause.
Claims (24)
1. a kind of log De-weight method, which is characterized in that be applied to log collection equipment, which comprises
Obtain log data set；Wherein, the log data set includes the same process of log generating device in first time period
The daily record data of generation, and, daily record data set identifier；
In the daily record data set identifier of the log data set, log data set title and serial number are determined；Wherein, the sequence
Number for distinguishes have same log dataset name different log data sets；
In the corresponding storage state mark array of the log data set title, storage state corresponding with the serial number is determined
Mark；
If storage state mark corresponding with the serial number indicates storing daily record data collection state, refuse to store the log
Data set.
2. the method as described in claim 1, which is characterized in that described in the corresponding storage state of the log data set title
It identifies in array, determines storage state mark corresponding with the serial number, comprising:
In log data set name list, the log data set title has been searched whether；
If finding the log data set title, array is identified in the corresponding storage state of the log data set title
In, determine storage state mark corresponding with the serial number.
3. method according to claim 2, which is characterized in that further include:
If not finding the log data set title, the daily record data is added in the log data set name list
Collect title；
It constructs storage state corresponding with the log data set title and identifies array, and, depositing in storage state mark array
Storage status indicator indicates not stored log data set state；
The log data set is stored, and changes storage state mark corresponding with serial number in the storage state mark array,
So that storage state mark indicates storing daily record data collection state.
4. the method as described in claim 1, which is characterized in that
The corresponding log data set of one first time period, several first time period form second time period, and, the second time
The corresponding log data set title of several first time period is consistent in section；
The then serial number in the daily record data set identifier has same log dataset name for distinguishing in second time period
Log data set.
5. method as claimed in claim 4, which is characterized in that the log data set title in the daily record data set identifier,
Include:
The device identification of the log generating device；
The process identification (PID) of process described in the log generating device；
The corresponding system timestamp of the second time period；
Wherein, which is the product of integer and the second time period, and the integer is to device log data collection
Corresponding system timestamp and the quotient of the second time period obtain after being rounded downwards.
6. method as claimed in claim 4, which is characterized in that the serial number in the daily record data set identifier includes:
The log generating device is the log data set 1 to the numerical value determined between greatest measure, and, when described second
Between in section different log data sets correspond to different serial numbers；
Wherein, the greatest measure rounds up to obtain for the quotient of the second time period and the first time period.
7. the method as described in claim 1, which is characterized in that the storage state mark array includes bit array, bit
Bit in array is corresponding with serial number, and, the data value of bit indicates storage state mark.
8. a kind of log De-weight method, which is characterized in that be applied to log generating device, which comprises
Obtain this equipment log data set that same process generates in first time period；
Daily record data set identifier is added for the log data set；Wherein, the daily record data set identifier includes log data set
Title, and, for distinguishing the serial number of the different log data sets with same log dataset name；
Log data set is sent to log collecting device, so that log collection equipment is in the log data set of the log data set
In mark, log data set title and serial number are determined, identify array in the corresponding storage state of the log data set title
In, determine storage state mark corresponding with the serial number；If storage state mark corresponding with the serial number indicates to have stored
Log data set state is then refused to store the log data set.
9. method according to claim 8, which is characterized in that described to add log data set mark for the log data set
Know, comprising:
Determine the corresponding log data set title of the log data set；
Determine the corresponding serial number of the log data set；
The character string that the log data set title and the serial number are formed, is determined as the daily record data set identifier；
The daily record data set identifier is added for the log data set.
10. method as claimed in claim 9, which is characterized in that the corresponding daily record data of the determination log data set
Collect title, comprising:
Obtain the device identification of this equipment, the process identification (PID) of the process and system timestamp；
The quotient for calculating the system timestamp Yu the second time period obtains integer after being rounded downwards to the quotient, meter
Calculate the product of the integer Yu the second time period；
The character string that the device identification, the process identification (PID) and the product are formed, is determined as the daily record data set name
Claim.
11. method as claimed in claim 9, which is characterized in that the corresponding serial number of the determination log data set, packet
It includes:
In the second time period, by the first initial count value, it is every to generate log data set and be incremented by 1, Zhi Daoji
Numerical value increases preset times；Alternatively,
In the second time period, by the second initial count value, one log data set of every generation successively decreases 1, and go directly meter
Number decrements preset times；
Wherein, the preset times are rounded up to obtain by the quotient of the second time period and the first time period.
12. a kind of content distribution network system characterized by comprising central node and the side being connected with the central node
Edge node, the fringe node include several servers；
Server is added for collecting the log data set that same process generates in first time period for the log data set
Daily record data set identifier sends log data set to streaming message system；Wherein, the daily record data set identifier includes multiple days
The log data set title that will data set is all made of, and, for distinguishing the different log numbers with same log dataset name
According to the serial number of collection；
Central node, for obtaining log data set from the streaming message system, in the daily record data of the log data set
In set identifier, log data set title and serial number are determined；Number is identified in the corresponding storage state of the log data set title
In group, storage state mark corresponding with the serial number is determined；If storage state mark corresponding with the serial number indicates to have deposited
Log data set state is stored up, then refuses to store the log data set.
13. a kind of log machining system characterized by comprising
Log generating device is the daily record data for collecting the log data set that same process generates in first time period
Collection addition daily record data set identifier, sends log data set to log collecting device；Wherein, the daily record data set identifier includes
The log data set title that multiple log data sets are all made of, and, for distinguishing the difference with same log dataset name
The serial number of log data set；
Log collection equipment in the daily record data set identifier of the log data set, is determined for obtaining log data set
Log data set title and serial number；In the log data set title corresponding storage state mark array, it is determining with it is described
The corresponding storage state mark of serial number；If storage state mark corresponding with the serial number indicates storing daily record data collection shape
State is then refused to store the log data set.
14. a kind of log De-weight method, which is characterized in that be applied to log collection equipment, which comprises
Obtain log data set；Wherein, the log data set includes the same process of log generating device in first time period
The daily record data and log data set title of generation；
Determine the log data set title of the log data set；
In existing log data set name list, the log data set title has been searched whether；
If finding the log data set title, refuse to store the log data set.
15. method as claimed in claim 14, which is characterized in that further include:
If not finding the log data set title, the log data set is stored；
In the log data set name list, the log data set title is added.
16. method as claimed in claim 15, which is characterized in that the daily record data set name of the determination log data set
Claim, comprising: concentrated in the daily record data and extract the log data set title；
The wherein log data set title, comprising:
The device identification of the log generating device；
The process identification (PID) of process described in the log generating device；
The log generating device generates the system timestamp during log data set.
17. a kind of log De-weight method, which is characterized in that be applied to log generating device, which comprises
Obtain this equipment log data set that same process generates in first time period；
Log data set title is generated, and adds log data set title for the log data set；
It sends comprising log data set to log collecting device, so that log collection equipment is in existing log data set name column
The log data set title has been searched whether in table, if finding the log data set title, refuses to store the day
Will data set.
18. method as claimed in claim 17, which is characterized in that described to add daily record data set name for the log data set
Claim, comprising:
Obtain the device identification of described equipment, the process identification (PID) of the process and system timestamp；
The character string that the device identification, the process identification (PID) and the system timestamp are formed, is determined as the log number
According to collection title；
The log data set title is added for log data set.
19. a kind of log machining system characterized by comprising
Log generating device is the daily record data for collecting the log data set that same process generates in first time period
Collection addition log data set title；The log data set is sent to log collecting device；
Log collection equipment sends log data set for obtaining log generating device；In existing log data set name column
In table, the log data set title in the log data set is searched whether；If so, refusal stores the daily record data
Collection.
20. a kind of content distribution network system characterized by comprising central node and the side being connected with the central node
Edge node, the fringe node include several servers；
Server is added for collecting the log data set that same process generates in first time period for the log data set
Log data set title sends log data set to streaming message system；
Central node, for obtaining log data set from streaming message system；In existing log data set name list, look into
The log data set title whether having in the log data set looked for；If so, refusal stores the log data set.
21. a kind of log duplicate removal device, which is characterized in that be integrated in log collection equipment, comprising:
First acquisition unit, for obtaining log data set；Wherein, the log data set include log generating device it is same into
The daily record data and log data set title that journey generates in first time period；
Determination unit, for determining the log data set title of the log data set；
Searching unit, for having searched whether the log data set title in existing log data set name list；
Refuse storage unit, if refusing to store the log data set for finding the log data set title.
22. a kind of log duplicate removal device, which is characterized in that be integrated in log generating device, comprising:
Second acquisition unit, for obtaining this equipment log data set that same process generates in first time period；
First addition mark unit, for generating log data set title and adding daily record data set name for the log data set
Claim；
First transmission unit, for send include the log data set title log data set to log collecting device, with
The log data set title is searched whether in log data set name list for log collection equipment, if finding described
Log data set title is then refused to store the log data set.
23. a kind of log duplicate removal device, which is characterized in that be integrated in log collection equipment, comprising:
Third acquiring unit, for obtaining log data set；Wherein, the log data set include log generating device it is same into
The daily record data that journey generates in first time period, and, daily record data set identifier；
Determination unit, for determining log data set title and sequence in the daily record data set identifier of the log data set
Number；Wherein, the serial number is used to distinguish the different log data sets with same log dataset name；
Determine that storage state identifies unit, for identifying in array in the corresponding storage state of the log data set title, really
Fixed storage state mark corresponding with the serial number；
Refuse storage unit, if indicating storing daily record data collection state for storage state corresponding with serial number mark,
Then refuse to store the log data set.
24. a kind of log duplicate removal device, which is characterized in that be integrated in log generating device, comprising:
4th acquiring unit, for obtaining this equipment log data set that same process generates in first time period；
Second addition mark unit, for adding daily record data set identifier for the log data set；Wherein, the daily record data
Set identifier includes log data set title, and, for distinguishing the different log data sets with same log dataset name
Serial number；
Second transmission unit, for sending log data set to log collecting device, so that log collection equipment is in the log
In the daily record data set identifier of data set, log data set title and serial number are determined, it is corresponding in the log data set title
Storage state mark array in, determine that corresponding with serial number storage state identifies；If storage corresponding with the serial number
Status indicator indicates storing daily record data collection state, then refuses to store the log data set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711487741.7A CN110019054B (en) | 2017-12-29 | 2017-12-29 | Log duplicate removal method and system, and content distribution network system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711487741.7A CN110019054B (en) | 2017-12-29 | 2017-12-29 | Log duplicate removal method and system, and content distribution network system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019054A true CN110019054A (en) | 2019-07-16 |
| CN110019054B CN110019054B (en) | 2023-01-31 |
Family
ID=67187220
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711487741.7A Active CN110019054B (en) | 2017-12-29 | 2017-12-29 | Log duplicate removal method and system, and content distribution network system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019054B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200137086A1 (en) * | 2018-10-31 | 2020-04-30 | EMC IP Holding Company LLC | Generating Unique Virtual Process Identifiers for Use in Network Security Mechanisms |
| CN112597138A (en) * | 2020-12-10 | 2021-04-02 | 浙江岩华文化科技有限公司 | Data deduplication method and device, computer equipment and computer-readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011123752A (en) * | 2009-12-11 | 2011-06-23 | Fujitsu Ltd | Device for collecting log, program for collecting log, method for collecting log, and system for collecting log |
| CN102567503A (en) * | 2010-12-16 | 2012-07-11 | 微软公司 | Extensible pipeline for data deduplication |
| JP2013037403A (en) * | 2011-08-03 | 2013-02-21 | Canon Inc | Log management system, log management method, application server, and log server |
| CN103270499A (en) * | 2011-12-21 | 2013-08-28 | 华为技术有限公司 | Log storage method and system |
| CN103593442A (en) * | 2013-11-15 | 2014-02-19 | 北京国双科技有限公司 | Duplication eliminating method and device for log data |
| CN106789303A (en) * | 2016-12-29 | 2017-05-31 | 北京奇艺世纪科技有限公司 | A kind of container log collection method and device |
-
2017
- 2017-12-29 CN CN201711487741.7A patent/CN110019054B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011123752A (en) * | 2009-12-11 | 2011-06-23 | Fujitsu Ltd | Device for collecting log, program for collecting log, method for collecting log, and system for collecting log |
| CN102567503A (en) * | 2010-12-16 | 2012-07-11 | 微软公司 | Extensible pipeline for data deduplication |
| JP2013037403A (en) * | 2011-08-03 | 2013-02-21 | Canon Inc | Log management system, log management method, application server, and log server |
| CN103270499A (en) * | 2011-12-21 | 2013-08-28 | 华为技术有限公司 | Log storage method and system |
| CN103593442A (en) * | 2013-11-15 | 2014-02-19 | 北京国双科技有限公司 | Duplication eliminating method and device for log data |
| CN106789303A (en) * | 2016-12-29 | 2017-05-31 | 北京奇艺世纪科技有限公司 | A kind of container log collection method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200137086A1 (en) * | 2018-10-31 | 2020-04-30 | EMC IP Holding Company LLC | Generating Unique Virtual Process Identifiers for Use in Network Security Mechanisms |
| US11050766B2 (en) * | 2018-10-31 | 2021-06-29 | EMC IP Holding Company LLC | Generating unique virtual process identifiers for use in network security mechanisms |
| CN112597138A (en) * | 2020-12-10 | 2021-04-02 | 浙江岩华文化科技有限公司 | Data deduplication method and device, computer equipment and computer-readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019054B (en) | 2023-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106484875B (en) | MOLAP-based data processing method and device | |
| CN102054000B (en) | Data querying method, device and system | |
| CN105528407B (en) | Method and device for acquiring L users with optimal propagation influence | |
| CN108415845A (en) | AB tests computational methods, device and the server of system index confidence interval | |
| CN103345508A (en) | Data storage method and system suitable for social network graph | |
| CN104346458B (en) | Date storage method and storage device | |
| CN104063501B (en) | copy balance method based on HDFS | |
| CN110222074A (en) | It indexes lookup method, search device, electronic equipment and storage medium | |
| CN110716950A (en) | Method, device and equipment for establishing aperture system and computer storage medium | |
| CN110209714A (en) | Report form generation method, device, computer equipment and computer readable storage medium | |
| CN110019054A (en) | Log De-weight method and system, content distribution network system | |
| CN110809066A (en) | IPv6 address generation model creation method, device and address generation method | |
| CN109033173A (en) | It is a kind of for generating the data processing method and device of multidimensional index data | |
| CN105242873B (en) | The acquisition of the performance data of cloud computing system and storage method and device | |
| CN111694793A (en) | Log storage method and device and log query method and device | |
| CN106789147A (en) | A kind of flow analysis method and device | |
| CN102724290A (en) | Method, device and system for getting target customer group | |
| CN103455491A (en) | Method and device for classifying search terms | |
| CN104301182B (en) | A kind of querying method and device of the exception information of website visiting at a slow speed | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN109150819B (en) | A kind of attack recognition method and its identifying system | |
| CN107070987B (en) | Data acquisition method and system for distributed object storage system | |
| CN104636384B (en) | A kind of method and device handling document | |
| CN104850658B (en) | A kind of data filling method and system | |
| CN105468603B (en) | Data selecting method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40010862 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |