CN109992469A - A kind of method and device merging log - Google Patents
A kind of method and device merging log Download PDFInfo
- Publication number
- CN109992469A CN109992469A CN201711489592.8A CN201711489592A CN109992469A CN 109992469 A CN109992469 A CN 109992469A CN 201711489592 A CN201711489592 A CN 201711489592A CN 109992469 A CN109992469 A CN 109992469A
- Authority
- CN
- China
- Prior art keywords
- caching
- log
- persistence
- persistentization
- same field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0778—Dumping, i.e. gathering error/state information after a fault for later diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0787—Storage of error reports, e.g. persistent data storage, storage using memory protection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a kind of method and devices for merging log, wherein, this method includes in the non-persistentization caching that the daily record reproduction received write-in is arranged and persistence caching, wherein non-persistentization caching caches the caching duration to log to the caching duration of log less than persistence.The log with same field is searched in non-persistentization caching, and is obtained from the log of the same field found and is met a variety of logs of preset backtracking condition and merge, by the log buffer after merging into persistence caching.The embodiment of the present invention passes through the non-persistentization caching of the write-in setting for the daily record reproduction that will be received and persistence caches, so as to avoid it is subsequent when obtaining log by a large amount of acquisition request be assigned directly to persistence caching in, with reduce persistence caching log processing pressure.Also, the memory space of log can also effectively be saved by merging the log with same field, and facilitate the subsequent centralized management to log.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of method and device for merging log.
Background technique
Log is a very extensive concept in computer systems, and (such as operating system nucleus various is answered for any program
With server etc.) it is likely to output journal.Although the content of log, scale and purposes are different, overall
Function is logging software operating status, the event information that storage system generates.
With the continuous development of computer technology, current log amount is more and more very big, especially recalls in log and advises
When then very much (such as the backtracking rule of same log includes returning every 5 minutes or 1 hour or progress log in 5 hours
Trace back), it needs constantly to obtain log from the cache database of storage log, since the performance of persistence caching is poor,
This can cause very big pressure to persistence cache database, to be easy to influence log processing system to the treatability of log
Energy.Therefore, how for the caching of log and processing provides sufficiently large flexibility and good retractility faces at present
Biggish challenge.
Summary of the invention
In view of the above problems, it proposes on the present invention overcomes the above problem or at least be partially solved in order to provide one kind
State the method and device of the merging log of problem.
One side according to the present invention provides a kind of method for merging log, comprising:
It will be in non-persistentization caching and persistence caching that daily record reproduction write-in setting be received, wherein described non-persistent
Change caching and persistence caching is less than to the caching duration of log to the caching duration of log；
The log with same field is searched in non-persistentization caching；
It is obtained from the log of the same field found and meets a variety of logs of preset backtracking condition and merge, will merged
During log buffer afterwards to the persistence caches.
Optionally, if not finding the log with same field in non-persistentization caching, continue from described
The log with same field is searched in persistence caching.
Optionally, described by before the non-persistentization caching and persistence caching that receive daily record reproduction write-in setting, it goes back
Include: that setting first order caching and second level caching are used as non-persistentization to cache, the third level is set and is cached as persistence caching.
Optionally, the log with same field is searched in non-persistentization caching, comprising:
Successively the log with same field is searched from first order caching, second level caching.
Optionally, the first order caching includes redis cluster, and the second level caching includes aerospike, institute
Stating third level caching includes hbase.
Optionally, the method is applied to kafka system, non-persistentization that will receive daily record reproduction write-in setting
In caching and persistence caching, comprising:
By receiving in log there is the log of same field to be assigned to based on the same worker row in the kafka system
Cheng Zhong, wherein the field includes User ID or other unique identifications of user key；
The non-persistentization caching is written in the daily record reproduction being assigned to each worker stroke and the persistence caches
In.
Optionally, there is described will receive in log the log of same field to be assigned to based in the kafka system
In same worker stroke, comprising:
It is operated using flink keyby and the log with same field is assigned to based on same in the kafka system
In one worker stroke.
Optionally, the log with same field is searched in non-persistentization caching, comprising:
Start back tracking operation when reaching the preset backtracking time, and searches in non-persistentization caching with same field
Log.
Optionally, the method also includes:
When reaching the preset backtracking time and starting back tracking operation, judge to recall time and log write time difference whether be more than
The caching duration of the non-persistentization caching；
If so, directly searching the log with same field from persistence caching.
Optionally, during the non-persistentization caching that will receive daily record reproduction write-in setting and persistence cache, comprising:
It receives log and adds timestamp for the log, by non-the holding of the daily record reproduction for carrying timestamp write-in setting
In longization caching and persistence caching.
Optionally, described when reaching preset backtracking time and starting back tracking operation, judge that backtracking time and log write are fashionable
Between difference whether be more than non-persistentization cache caching duration, comprising:
When reaching the preset backtracking time and starting back tracking operation, according to the timestamp judgement backtracking time of log and log write
Angle of incidence difference whether be more than non-persistentization caching caching duration.
Optionally, a variety of logs for meeting preset backtracking condition include:
Log corresponding to multiple composition objects in complete session movement, wherein in the complete session movement
Multiple composition objects according to business demand determination obtain.
Optionally, the method also includes: if can not be obtained from the log of the same field found satisfaction put back into advance
The a variety of logs for condition of tracing back, topic is established in the log according to the same field found, and maximum is arranged for the topic established
Recall the time；
Search whether exist from non-persistentization caching and persistence caching when reaching the maximum traceback time
A variety of logs that are identical as the log field in topic and meeting preset backtracking condition；
If so, merging a variety of logs that there is same field and meet preset backtracking condition, and the log after merging is delayed
It deposits into persistence caching.
Another aspect according to the present invention additionally provides a kind of device for merging log, comprising:
Writing module, suitable for that will receive in the non-persistentization caching and persistence caching that daily record reproduction write-in is arranged,
In, the non-persistentization caching is less than persistence caching to the caching duration of log to the caching duration of log；
Searching module, suitable for searching the log with same field in non-persistentization caching；
Merging module, suitable for obtaining a variety of logs for meeting preset backtracking condition from the log of the same field found
And merge, by the log buffer after merging into persistence caching.
Optionally, the searching module is further adapted for:
If not finding the log with same field in non-persistentization caching, continue slow from the persistence
Deposit the middle log searched and there is same field.
Optionally, described device further include:
Setup module, suitable for non-persistentization for receiving daily record reproduction write-in setting is cached and held in the write module
Before longization caching, the setting first order is cached and the second level caches and caches as non-persistentization, and the setting third level caches to be used as and hold
Longization caching.
Optionally, the searching module is further adapted for:
Successively the log with same field is searched from first order caching, second level caching.
Optionally, the first order caching includes redis cluster, and the second level caching includes aerospike, institute
Stating third level caching includes hbase.
Optionally, described device is applied to kafka system, and the write module is further adapted for:
By receiving in log there is the log of same field to be assigned to based on the same worker row in the kafka system
Cheng Zhong, wherein the field includes User ID or other unique identifications of user key；
The non-persistentization caching is written in the daily record reproduction being assigned to each worker stroke and the persistence caches
In.
Optionally, the write module is further adapted for:
It is operated using flink keyby and the log with same field is assigned to based on same in the kafka system
In one worker stroke.
Optionally, the searching module is further adapted for:
Start back tracking operation when reaching the preset backtracking time, and searches in non-persistentization caching with same field
Log.
Optionally, described device further include:
Judgment module when suitable for reaching the preset backtracking time and start back tracking operation, judges that recalling time and log is written
Time difference whether be more than non-persistentization caching caching duration；
If so, the searching module directly searches the log with same field from persistence caching.
Optionally, the write module is further adapted for:
It receives log and adds timestamp for the log, by non-the holding of the daily record reproduction for carrying timestamp write-in setting
In longization caching and persistence caching.
Optionally, the judgment module is further adapted for:
When reaching the preset backtracking time and starting back tracking operation, according to the timestamp judgement backtracking time of log and log write
Angle of incidence difference whether be more than non-persistentization caching caching duration.
Optionally, a variety of logs for meeting preset backtracking condition include:
Log corresponding to multiple composition objects in complete session movement, wherein in the complete session movement
Multiple composition objects according to business demand determination obtain.
Optionally, described device further include:
Establish module, if can not be obtained from the log of the same field found suitable for the merging module meet it is preset
A variety of logs of backtracking condition, topic is established in the log according to the same field found, and is arranged most for the topic established
The big backtracking time；
The searching module is when reaching the maximum traceback time from non-persistentization caching and persistence caching
Search whether there are a variety of logs identical and meeting preset backtracking condition with the log field in topic；
If so, the merging module merges a variety of logs for having same field and meeting preset backtracking condition, and will close
Log buffer after and is into persistence caching.
According to the present invention in another aspect, additionally providing a kind of electronic equipment, comprising:
Processor；And
It is arranged to the memory of storage computer executable instructions, the executable instruction makes the place when executed
Device is managed to execute according to the method for merging log in any embodiment above.
Another aspect according to the present invention, additionally provides a kind of computer storage medium, wherein described computer-readable to deposit
Storage media stores one or more programs, and one or more of programs are executed when the electronic equipment for being included multiple application programs
When, so that the electronic equipment executes the method according to the merging log in any embodiment above.
In embodiments of the present invention, when receiving log, by the non-persistent of the daily record reproduction received write-in setting
Change in caching and persistence caching, wherein non-persistentization caching is less than persistence to the caching duration of log and caches to log
Cache duration.The log with same field can also be searched in non-persistentization caching by being written after log, and from finding
Same field log in obtain a variety of logs for meeting preset backtracking condition and merge, by the log buffer after merging to holding
In longization caching.The embodiment of the present invention is cached by setting non-persistentization as a result, and persistence caches, and the log that will be received
Replicability write-in setting non-persistentization caching and persistence caching, so as to avoid it is subsequent will be a large amount of when obtaining log
Acquisition request be assigned directly to persistence caching in, with reduce persistence caching log processing pressure.Further, will have
The memory space of log can also effectively be saved by having the log of same field to merge, and facilitate the subsequent concentration to log
Management.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
According to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will be brighter
The above and other objects, advantages and features of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows the flow diagram of the method according to an embodiment of the invention for merging log；
Fig. 2 shows the flow diagrams of the method in accordance with another embodiment of the present invention for merging log；
Fig. 3 shows the structural schematic diagram of the device according to an embodiment of the invention for merging log；
Fig. 4 shows the structural schematic diagram of the device in accordance with another embodiment of the present invention for merging log；
Fig. 5 shows the block diagram of the calculating equipment for executing the method according to the present invention for merging log；And
Fig. 6 shows depositing for the program code for keeping or carrying the method for realizing merging log according to the present invention
Storage unit.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
In order to solve the above technical problems, the embodiment of the invention provides a kind of method for merging log, this method can be answered
In kafka system, Kafka is that a kind of distributed post of high-throughput subscribes to message system, it can handle consumer
Everything flow data in the website of scale, the main target of kafka exploitation are building one and are used to handle massive logs, use
The data processing shelf of statistics etc. is runed in family behavior and website.Fig. 1 shows merging log according to an embodiment of the invention
Method flow diagram.Referring to Fig. 1, this method includes at least step S102 to step S106.
Step S102, will be in the non-persistentization caching and persistence caching that receive daily record reproduction write-in setting, wherein non-
Persistence, which is cached, is less than persistence caching to the caching duration of log to the caching duration of log.
In this step, it will be referred in the non-persistentization caching and persistence caching of the write-in setting of daily record reproduction
After a log received, non-persistentization which can be not only written to setting is cached, and can also the log be written and is arranged
Persistence caching in.Write-in log mode can be and meanwhile be written two classes caching (i.e. non-persistentization cache with persistence delays
Deposit), or a kind of caching of first write-in, then be written in another kind of caching, the embodiment of the present invention is not specifically limited in this embodiment.
Step S104 searches the log with same field in non-persistentization caching.
In this step, although the log with same field refers to that log content is different (or identical), it has
There is identical field.Wherein, field may include User ID, can also include other unique identifications of user key etc..
Step S106 is obtained from the log of the same field found and is met a variety of logs of preset backtracking condition and close
And by the log buffer after merging into persistence caching.
In embodiments of the present invention, when receiving log by non-persistentization of the daily record reproduction received write-in setting
In caching and persistence caching, wherein non-persistentization is cached delays the caching duration of log less than persistence caching to log
Deposit duration.The log with same field can also be searched in non-persistentization caching by being written after log, and from finding
It is obtained in the log of same field and meets a variety of logs of preset backtracking condition and merge, by the log buffer after merging to lasting
Change in caching.The embodiment of the present invention is cached by setting non-persistentization as a result, and persistence caches, and the log received is answered
Property processed write-in setting non-persistentization caching and persistence caching, so as to avoid it is subsequent will be a large amount of when obtaining log
Acquisition request is assigned directly in persistence caching, to reduce the log processing pressure of persistence caching.Further, will have
The log of same field, which merges, effectively to save the memory space of log, and facilitates the subsequent concentrate tube to log
Reason.
Step S102 is seen above, in an embodiment of the present invention, non-the holding of daily record reproduction write-in setting will received
It also needs first to be arranged non-persistentization caching before longization caching and persistence caching and persistence caches two classes caching.For example, can
It is cached so that first order caching and second level caching is arranged as non-persistentization, and it is slow as persistence that third level caching is arranged
It deposits.Wherein, first order caching can use redis cluster (i.e. Redis cluster), and Redis is the Key- of an open source
Value database, and the API (Application Program Interface, application programming interfaces) of multilingual is provided.
Second level caching can use aerospike database, be a distributed expandable NoSql (Not Only SQL, it is non-
Relevant database).Third level caching can use hbase database, be a distributed open source data towards column
Library.The database that cachings at different levels herein use is only schematical, can also include other kinds of database, the present invention
Embodiment is not specifically limited in this embodiment.
In embodiments of the present invention, non-persistentization caching can be far smaller than persistence caching pair to the caching duration of log
The caching duration of log.For example, a length of 1 hour, 2 hours etc. when the caching of non-persistentization caching, and persistence caching is slow
A length of 10 days, 20 days etc. when depositing, or can also to persistence cache caching duration be not provided with the specific time, that is, cache to
As long as the log in persistence caching is without artificial deletion, log can be saved always.If not persistence caching includes
First order caching and second level caching, then when can be respectively that first order caching caches accordingly with second level buffer setting
Long, the caching duration of this two-level cache may be the same or different.For example, the caching of first order caching and second level caching
Duration is 1 hour, then the log in two-level cache just will be updated once every 1 hour, that is, deletes the log cached originally,
And continue the log that caching newly receives.It is 10 days a length of when caching if persistence caching is cached comprising the third level, similarly, the
Log in three-level caching updated once every 10 days, still, if third level caching does not have allocating cache duration, then therein
Log can persistence, certain user can taking human as delete persistence caching in log.The embodiment of the present invention is to non-persistent
The cache-time for changing caching and persistence caching is not specifically limited.
It in embodiments of the present invention, is not Installed System Memory in databases at different levels by the log buffer received, it is main
If reason is wanted to be to save log in system memory, then the log when system restarting in memory is possible to lose
Lose, i.e., when the system restarts before the log that caches in memory all can miss fall, to only be left persistence caching (such as
Hbase database) in log.When it is subsequent have business usage log when can only from persistence caching in obtain log, this may
So that the process performance of persistence caching can not keep up with the demand of business.It is cached by setting non-persistentization and persistence caches,
If first order caching, second level caching and third level caching, log after system reboot in caching at different levels will not lose,
And the log in cachings at different levels can satisfy various businesses demand, avoid the log processing pressure for increasing persistence caching.
Step S104 is seen above, in an embodiment of the present invention, if not persistence caching includes first order caching and the
L2 cache.When so, searching the log with same field in non-persistentization caching, can successively be cached from the first order,
The log with same field is searched in the caching of the second level, i.e., first searches whether the day with same field from first order caching
Will, if so, a variety of logs for meeting preset backtracking condition can be obtained from the log of same field, if it is not, again from second
The log with same field is searched in grade caching.Certainly, if still not found with same field in the caching of the second level
Log, can also continue to cache the log searched in (such as the third level cache) and there is same field from persistence.
In an embodiment of the present invention, the process of the log with same field is searched in non-persistentization caching i.e. to slow
The trace-back process for the log deposited.When reaching the preset backtracking time and starting to start back tracking operation (i.e. join operation), non-
The log with same field is searched in persistence caching.Wherein, the preset backtracking time can be 5 minutes, 1 hour, 5 hours
Etc. duration, the embodiment of the present invention do not do specific restriction to this.
In this embodiment, when reaching the preset backtracking time and starting back tracking operation, can also first judge to recall the time
With log write time difference whether be more than non-persistentization caching caching duration, if so, can directly from persistence caching in
Search the log with same field.Since the cache-time of non-persistentization caching is shorter, aim at storage time therein within the next few days
It is shorter, it needs the log recalled that may not cache in non-persistentization when reaching the preset backtracking time and suffer, therefore, when sentencing
It, can be directly from persistence caching when disconnected backtracking time and log write time difference are more than the caching duration of non-persistentization caching
The log with same field is searched, to reduce unnecessary operating process, to improve backtracking efficiency.
It wherein, can when judging whether recall the time is more than the caching duration of non-persistentization caching with log write time difference
Judge to recall the time with the timestamp according to log in non-persistentization caching and whether log write time difference is more than non-persistentization
The caching duration of caching.The timestamp of log can be the timestamp added after receiving log for log, and then will take
In the non-persistentization caching and persistence caching of daily record reproduction write-in setting with timestamp.
Step S106 is seen above, in embodiments of the present invention, a variety of logs for meeting preset backtracking condition can be
Log corresponding to multiple composition objects in whole session movement.Wherein, multiple composition objects in complete session movement
It is obtained according to business demand determination.For example, primary complete session movement may include the competing of customer flow in business 1
Valence, displaying, clicking operation, wherein bid, show, clicking operation be the complete session of business 1 movement in three groups it is pairs of
As.In business 2, primary complete session movement may include bidding, showing for customer flow, without including clicking operation,
Wherein bid, show as the complete session of business 2 movement in two composition objects.Different business demands is corresponding as a result,
Complete session movement be also it is different.
In this embodiment, meet preset backtracking condition if can not obtain from the log of the same field found
A variety of logs.At this point it is possible to handled using a variety of logs for being unsatisfactory for preset backtracking condition as special log type, i.e. foundation
Topic is established in the log of the same field found, and the maximum traceback time is arranged for the topic established.And it is maximum reaching
Recall the time when from non-persistentization caching and persistence cache in search whether in the presence of with the log field in topic it is identical and
Meet a variety of logs of preset backtracking condition, and if it exists, then merging has same field and meet a variety of of preset backtracking condition
Log, and by the log buffer after merging into persistence caching.Wherein, kafka system can use topic and carry out to log
Classification Management, a business can apply for multiple topic, also, the log of a topic can also be shared by multiple business.
For example, in business 1 mentioned above, user's A flow can be only found in trace-back process and bids, show pair
The log answered, the corresponding log of clicking operation without finding user A, i.e., without recalling successfully.At this point it is possible to according to looking into
Topic is established in the log of the user A found, and maximum traceback time, such as the maximum traceback of setting is arranged for the topic established
Time is 5 hours, then searching whether that there are user A again from non-persistentization caching and persistence caching after 5 hours
The bidding, show of flow, the corresponding log of clicking operation, if it does, merge the log found, and by the log after merging
Caching is into persistence caching.
Herein it should be noted that being that second grade can recall completion, the maximum of setting for most back tracking operation
The backtracking time refers to the maximum traceback time that business end can tolerate, i.e. business end can tolerate that successfully log is recalled in acquisition
Most waiting time.Certainly, for no successful user journal of backtracking, in (such as 5 hours no arrival maximum traceback time
Deng) when, it is possible to preset backtracking condition is just met without recalling successful user journal before.For example, recalling before
User's A flow can only be found in journey to bid, show corresponding log, the clicking operation pair without finding user A
The log answered, and the maximum traceback time being arranged is 5 hours, if having received click logs in next 1 minute, that
The process of back tracking operation can be no longer executed when reaching the maximum traceback time of setting.
The maximum traceback time can actually be considered that one kind is revealed all the details guarantee as a result, be exactly in the maximum tolerance of business end
It is interior to go under attempting whether to recall successfully log.Because the rule of some business be even if the maximum tolerance time (i.e. most
The big backtracking time) do not recall success, then being also required to merge the successful log of partial backtracking.
It can directly be cached after log is merged into function to lasting in the present invention one is real-time with continued reference to step S106
Change in caching, for offline job processing.Of course, it is possible to directly be divided outward centered on kafka system after log merges successfully
Hair, for real-time job processing.
The embodiment of the invention also provides another methods for merging log.Fig. 2 shows another realities according to the present invention
Apply the flow diagram of the method for the merging log of example.Referring to fig. 2, this method includes at least step S202 to step S212.
Step S202 by receiving in log there is the log of same field to be assigned to based on same in kafka system
In worker stroke.Wherein, field may include User ID or other unique identifications of user key.
In this step, the log with same field can be assigned to using flink keyby operation (such as hash)
Based in the same worker stroke in kafka system.Flink is one similar spark " open source technology stack ", can be provided
The functions such as batch processing, streaming computing, figure calculating, interactive inquiry, machine learning.By that will have the log of same field distribution
Into same worker stroke, thereby may be ensured that it is subsequent more easily the log of same field can be merged, into
And effectively guarantee the consistency of log when system carries out dilatation or upgrading.For example, will by flink keyby operation
User ID is that the log of " 11 " is assigned in worker1, the log that User ID is " 12 " is assigned in worker2, by user
ID is that the log of " 13 " is assigned in worker3.
Step S204, each worker stroke cache the daily record reproduction being assigned to write-in non-persistentization caching and persistence
In.
In this step, since the log of same field being assigned in identical worker stroke above,
Each worker stroke can by write-in non-persistentization of the daily record reproduction of the same field assigned by itself cache and persistently
Change in caching.For example, being assigned to the log that User ID is " 11 " in conjunction with above-described embodiment worker1, worker2 is assigned to user
ID is the log of " 12 ", then, non-persistentization caching and persistence is written in the daily record reproduction that User ID is " 11 " by worker1
In caching, the daily record reproduction that User ID is " 12 " is written in non-persistentization caching and persistence caching worker2, also,
The daily record reproduction that User ID is " 13 " is written in non-persistentization caching and persistence caching worker3.
Step S206 searches whether the log for having with same field in non-persistentization caching, if so, executing step
S208；If it is not, executing step S210.
Step S208 is obtained from the log of the same field found and is met a variety of logs of preset backtracking condition and close
And by the log buffer after merging into persistence caching, and execute step S212.
Step S210 searches whether the log with same field in persistence caching, if so, executing step S208；
If it is not, executing step S212, terminate.
The embodiment of the present invention can carry out stream process to the log received based on kafka system and flink, and then can be with
Dilatation, the capacity reducing of system are realized by degree of parallelism configuration.
Based on the same inventive concept, the embodiment of the invention also provides a kind of device for merging log, Fig. 3 shows basis
The structural schematic diagram of the device of the merging log of one embodiment of the invention, referring to Fig. 3, the device 300 for merging log includes writing
Enter module 310, searching module 320 and merging module 330.
It now introduces between each composition of the device 300 of the merging log of the embodiment of the present invention or the function and each section of device
Connection relationship:
Writing module 310, suitable for that will receive in the non-persistentization caching and persistence caching that daily record reproduction write-in is arranged,
Wherein, non-persistentization caching is less than persistence caching to the caching duration of log to the caching duration of log；
Searching module 320 is coupled with writing module 310, has same field suitable for searching in non-persistentization caching
Log；
Merging module 330 is coupled with searching module 320, is met suitable for obtaining from the log of the same field found
A variety of logs of preset backtracking condition simultaneously merge, by the log buffer after merging into persistence caching.Wherein, meet and put back into advance
The a variety of logs for condition of tracing back may include log corresponding to multiple composition objects in complete session movement, wherein complete
Multiple composition objects in session movement are obtained according to business demand determination.
In an embodiment of the present invention, if searching module 320 is further adapted for not finding in non-persistentization caching with phase
With the log of field, then continue to search the log with same field from persistence caching.
In an embodiment of the present invention, the device for merging log can also be applied to kafka system, and writing module 310 is also
It is assigned to suitable for the log in log with same field will be received based in the same worker stroke in kafka system,
In, field includes User ID or other unique identifications of user key.The daily record reproduction being assigned to is written each worker stroke
In non-persistentization caching and persistence caching.
In an embodiment of the present invention, writing module 310 is further adapted for, and will have same word using flink keyby operation
The log of section is assigned to based in the same worker stroke in kafka system.
In an embodiment of the present invention, searching module 320 is further adapted for, and starts back tracking operation when reaching the preset backtracking time,
And the log with same field is searched in non-persistentization caching.
The embodiment of the invention also provides another devices for merging log, and referring to fig. 4, the device 300 for merging log removes
Comprising except above-mentioned each module, further including setup module 340, judgment module 350 and establishing module 360.
Setup module 340 is coupled with writing module 310, is set suitable for will receive daily record reproduction write-in in writing module 310
Before non-persistentization caching and the persistence caching set, the setting first order is cached and second level caching is cached as non-persistentization,
The third level is arranged to cache as persistence caching.
Judgment module 350 couples respectively with writing module 310 and searching module 320, suitable for that will connect in writing module 310
After the non-persistentization caching and persistence caching of receiving daily record reproduction write-in setting, reaches the preset backtracking time and starting backtracking
When operation, judge to recall whether time and log write time difference are more than caching duration that non-persistentization caches, if so, by searching
Module 320 directly searches the log with same field from persistence caching.
Module 360 is established, is coupled respectively with merging module 330 and searching module 320, if suitable for merging module 330 from looking into
The a variety of logs for meeting preset backtracking condition can not be obtained in the log of the same field found, then according to the same word found
Topic is established in the log of section, and the maximum traceback time is arranged for the topic established.Searching module 320 is when reaching maximum traceback
Between when from non-persistentization caching and persistence caching in search whether exist with the log field in topic it is identical and meet in advance
The a variety of logs for the condition of tracing back are put back into, if so, being merged by merging module 330 has same field and meet the more of preset backtracking condition
Kind log, and by the log buffer after merging into persistence caching.
In an embodiment of the present invention, searching module 320 is further adapted for successively searching from first order caching, second level caching
Log with same field.Wherein, first order caching may include redis cluster, and second level caching may include
Aerospike, third level caching may include hbase.
In an embodiment of the present invention, writing module 310 is further adapted for, and receives log and adds timestamp for log, will take
In the non-persistentization caching and persistence caching of daily record reproduction write-in setting with timestamp.
In an embodiment of the present invention, judgment module 350 is further adapted for, and reaches the preset backtracking time and starts back tracking operation
When, when the timestamp according to log judges whether recall the time is more than the caching of non-persistentization caching with log write time difference
It is long.
According to the combination of any one above-mentioned preferred embodiment or multiple preferred embodiments, the embodiment of the present invention can reach
It is following the utility model has the advantages that
In embodiments of the present invention, when receiving log, by the non-persistent of the daily record reproduction received write-in setting
Change in caching and persistence caching, wherein non-persistentization caching is less than persistence to the caching duration of log and caches to log
Cache duration.The log with same field can also be searched in non-persistentization caching by being written after log, and from finding
Same field log in obtain a variety of logs for meeting preset backtracking condition and merge, by the log buffer after merging to holding
In longization caching.The embodiment of the present invention is cached by setting non-persistentization as a result, and persistence caches, and the log that will be received
Replicability write-in setting non-persistentization caching and persistence caching, so as to avoid it is subsequent will be a large amount of when obtaining log
Acquisition request be assigned directly to persistence caching in, with reduce persistence caching log processing pressure.Further, will have
The memory space of log can also effectively be saved by having the log of same field to merge, and facilitate the subsequent concentration to log
Management.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself
All as a separate embodiment of the present invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment
Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any
Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed
All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments in this include institute in other embodiments
Including certain features rather than other feature, but the combination of the feature of different embodiment means in the scope of the present invention
Within and form different embodiments.For example, in detail in the claims, the one of any of embodiment claimed all may be used
Come in a manner of in any combination using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors
Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice
Microprocessor or digital signal processor (DSP) are some in the device according to an embodiment of the present invention for merging log to realize
Or some or all functions of whole components.The present invention is also implemented as one for executing method as described herein
Partly or completely device or device program (for example, computer program and computer program product).Such realization is originally
The program of invention can store on a computer-readable medium, or may be in the form of one or more signals.In this way
Signal can be downloaded from an internet website to obtain, be perhaps provided on the carrier signal or be provided in any other form.
According to the present invention in another aspect, additionally provide a kind of electronic equipment, including processor；And it is arranged to store
The memory of computer executable instructions, executable instruction execute processor according in any embodiment above
Merging log method.
Another aspect according to the present invention additionally provides a kind of computer storage medium, wherein computer storage medium is deposited
One or more programs are stored up, one or more programs are when the electronic equipment for being included multiple application programs executes, so that electronics
Equipment executes the method according to the merging log in any embodiment above.
For example, Fig. 5 shows the calculating equipment that may be implemented to merge the method for log.The calculating equipment conventionally comprises place
Manage the computer program product or computer-readable medium of device 510 and 520 form of memory.Memory 520 can be such as
The electronic memory of flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk or ROM etc.Storage
Device 520 has the memory space 530 stored for executing the program code 531 of any method and step in the above method.For example,
The memory space 530 of storage program code may include each program for being respectively used to realize the various steps in above method
Code 531.These program codes can read or be written to from one or more computer program product this or
In the multiple computer program products of person.These computer program products include such as hard disk, compact-disc (CD), storage card or soft
The program code carrier of disk etc.Such computer program product is usually for example shown in fig. 6 portable or fixation is deposited
Storage unit.The storage unit can have the memory paragraph of 520 similar arrangement of memory in the calculating equipment with Fig. 5, storage sky
Between etc..Program code can for example be compressed in a suitable form.In general, storage unit includes for executing method of the invention
The computer-readable code 531 ' of step, it can the code read by such as 510 etc processor, when these codes are by counting
When calculating equipment operation, the calculating equipment is caused to execute each step in method described above.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
So far, although those skilled in the art will appreciate that present invention has been shown and described in detail herein multiple shows
Example property embodiment still without departing from the spirit and scope of the present invention, still can according to the present disclosure directly
Determine or deduce out many other variations or modifications consistent with the principles of the invention.Therefore, the scope of the present invention is understood that and recognizes
It is set to and covers all such other variations or modifications.The embodiment of the invention provides A1, a kind of method for merging log, packets
It includes:
It will be in non-persistentization caching and persistence caching that daily record reproduction write-in setting be received, wherein described non-persistent
Change caching and persistence caching is less than to the caching duration of log to the caching duration of log；
The log with same field is searched in non-persistentization caching；
It is obtained from the log of the same field found and meets a variety of logs of preset backtracking condition and merge, will merged
During log buffer afterwards to the persistence caches.
A2, method according to a1, wherein further include:
If not finding the log with same field in non-persistentization caching, continue slow from the persistence
Deposit the middle log searched and there is same field.
A3, method according to a1 or a2, wherein described to delay non-persistentization for receiving daily record reproduction write-in setting
It deposits before being cached with persistence, further includes: the setting first order caches and second level caching is used as non-persistentization to cache, and third is arranged
Grade caching is cached as persistence.
A4, method according to a3, wherein the log with same field is searched in non-persistentization caching,
Include:
Successively the log with same field is searched from first order caching, second level caching.
A5, method according to a3, wherein the first order caching includes redis cluster, and the second level is slow
Bag deposit includes aerospike, and the third level caching includes hbase.
A6, method according to a1 or a2, wherein the method is applied to kafka system, described to answer reception log
In the non-persistentization caching and persistence caching of property write-in setting processed, comprising:
By receiving in log there is the log of same field to be assigned to based on the same worker row in the kafka system
Cheng Zhong, wherein the field includes User ID or other unique identifications of user key；
The non-persistentization caching is written in the daily record reproduction being assigned to each worker stroke and the persistence caches
In.
A7, the method according to A6, wherein described that the log received in log with same field is assigned to and is based on
In same worker stroke in the kafka system, comprising:
It is operated using flink keyby and the log with same field is assigned to based on same in the kafka system
In one worker stroke.
A8, method according to a1 or a2, wherein search the day with same field in non-persistentization caching
Will, comprising:
Start back tracking operation when reaching the preset backtracking time, and searches in non-persistentization caching with same field
Log.
A9, the method according to A8, wherein further include:
When reaching the preset backtracking time and starting back tracking operation, judge to recall time and log write time difference whether be more than
The caching duration of the non-persistentization caching；
If so, directly searching the log with same field from persistence caching.
A10, the method according to A9, wherein described to cache non-persistentization for receiving daily record reproduction write-in setting
In persistence caching, comprising:
It receives log and adds timestamp for the log, by non-the holding of the daily record reproduction for carrying timestamp write-in setting
In longization caching and persistence caching.
A11, the method according to A10, wherein it is described when reaching preset backtracking time and starting back tracking operation, judge
Whether backtracking time and log write time difference are more than caching duration that non-persistentization caches, comprising:
When reaching the preset backtracking time and starting back tracking operation, according to the timestamp judgement backtracking time of log and log write
Angle of incidence difference whether be more than non-persistentization caching caching duration.
A12, method according to a1 or a2, wherein a variety of logs for meeting preset backtracking condition include:
Log corresponding to multiple composition objects in complete session movement, wherein in the complete session movement
Multiple composition objects according to business demand determination obtain.
A13, method according to a1 or a2, wherein further include:
If can not obtain a variety of logs for meeting preset backtracking condition from the log of the same field found, foundation is looked into
Topic is established in the log of the same field found, and the maximum traceback time is arranged for the topic established；
Search whether exist from non-persistentization caching and persistence caching when reaching the maximum traceback time
A variety of logs that are identical as the log field in topic and meeting preset backtracking condition；
If so, merging a variety of logs that there is same field and meet preset backtracking condition, and the log after merging is delayed
It deposits into persistence caching.
B14, a kind of device for merging log, comprising:
Writing module, suitable for that will receive in the non-persistentization caching and persistence caching that daily record reproduction write-in is arranged,
In, the non-persistentization caching is less than persistence caching to the caching duration of log to the caching duration of log；
Searching module, suitable for searching the log with same field in non-persistentization caching；
Merging module, suitable for obtaining a variety of logs for meeting preset backtracking condition from the log of the same field found
And merge, by the log buffer after merging into persistence caching.
B15, device according to b14, wherein the searching module is further adapted for:
If not finding the log with same field in non-persistentization caching, continue slow from the persistence
Deposit the middle log searched and there is same field.
B16, the device according to B14 or B15, wherein further include:
Setup module, suitable for non-persistentization for receiving daily record reproduction write-in setting is cached and held in the write module
Before longization caching, the setting first order is cached and the second level caches and caches as non-persistentization, and the setting third level caches to be used as and hold
Longization caching.
B17, the device according to B16, wherein the searching module is further adapted for:
Successively the log with same field is searched from first order caching, second level caching.
B18, the device according to B16, wherein the first order caching includes redis cluster, the second level
Caching includes aerospike, and the third level caching includes hbase.
B19, the device according to B14 or B15, wherein described device is applied to kafka system, the write module
It is further adapted for:
By receiving in log there is the log of same field to be assigned to based on the same worker row in the kafka system
Cheng Zhong, wherein the field includes User ID or other unique identifications of user key；
The non-persistentization caching is written in the daily record reproduction being assigned to each worker stroke and the persistence caches
In.
B20, the device according to B19, wherein the write module is further adapted for:
It is operated using flink keyby and the log with same field is assigned to based on same in the kafka system
In one worker stroke.
B21, the device according to B14 or B15, wherein the searching module is further adapted for:
Start back tracking operation when reaching the preset backtracking time, and searches in non-persistentization caching with same field
Log.
B22, the device according to B21, wherein further include:
Judgment module when suitable for reaching the preset backtracking time and start back tracking operation, judges that recalling time and log is written
Time difference whether be more than non-persistentization caching caching duration；
If so, the searching module directly searches the log with same field from persistence caching.
B23, the device according to B22, wherein the write module is further adapted for:
It receives log and adds timestamp for the log, by non-the holding of the daily record reproduction for carrying timestamp write-in setting
In longization caching and persistence caching.
B24, the device according to B23, wherein the judgment module is further adapted for:
When reaching the preset backtracking time and starting back tracking operation, according to the timestamp judgement backtracking time of log and log write
Angle of incidence difference whether be more than non-persistentization caching caching duration.
B25, the device according to B14 or B15, wherein a variety of logs for meeting preset backtracking condition include:
Log corresponding to multiple composition objects in complete session movement, wherein in the complete session movement
Multiple composition objects according to business demand determination obtain.
B26, the device according to B14 or B15, wherein further include:
Establish module, if can not be obtained from the log of the same field found suitable for the merging module meet it is preset
A variety of logs of backtracking condition, topic is established in the log according to the same field found, and is arranged most for the topic established
The big backtracking time；
The searching module is when reaching the maximum traceback time from non-persistentization caching and persistence caching
Search whether there are a variety of logs identical and meeting preset backtracking condition with the log field in topic；
If so, the merging module merges a variety of logs for having same field and meeting preset backtracking condition, and will close
Log buffer after and is into persistence caching.
C27, a kind of electronic equipment, comprising:
Processor；And
It is arranged to the memory of storage computer executable instructions, the executable instruction makes the place when executed
Device is managed to execute according to the described in any item methods for merging log of A1-A13.
D28, a kind of computer storage medium, wherein the computer storage medium stores one or more programs, described
One or more programs are when the electronic equipment for being included multiple application programs executes, so that the electronic equipment executes basis
The described in any item methods for merging log of A1-A13.
Claims (10)
1. a kind of method for merging log, comprising:
It will be in non-persistentization caching and persistence caching that daily record reproduction write-in setting be received, wherein non-persistentization is slow
It deposits and persistence caching is less than to the caching duration of log to the caching duration of log；
The log with same field is searched in non-persistentization caching；
It is obtained from the log of the same field found and meets a variety of logs of preset backtracking condition and merge, after merging
During log buffer to the persistence caches.
2. according to the method described in claim 1, wherein, further includes:
If not finding the log with same field in non-persistentization caching, continue from persistence caching
Search the log with same field.
3. method according to claim 1 or 2, wherein non-persistentization that daily record reproduction write-in setting will be received
Before caching and persistence caching, further includes: the setting first order caches and second level caching is cached as non-persistentization, is arranged the
Three-level caching is cached as persistence.
4. according to the method described in claim 3, wherein, searching the day with same field in non-persistentization caching
Will, comprising:
Successively the log with same field is searched from first order caching, second level caching.
5. according to the method described in claim 3, wherein, the first order caching includes redis cluster, the second level
Caching includes aerospike, and the third level caching includes hbase.
6. method according to claim 1 or 2, wherein the method is applied to kafka system, described to receive log
In the non-persistentization caching and persistence caching of replicability write-in setting, comprising:
By receiving in log there is the log of same field to be assigned to based on the same worker stroke in the kafka system
In, wherein the field includes User ID or other unique identifications of user key；
The daily record reproduction being assigned to is written in the non-persistentization caching and persistence caching each worker stroke.
7. according to the method described in claim 6, wherein, there is described will receive in log the log of same field to be assigned to base
In same worker stroke in the kafka system, comprising:
It is operated using flink keyby and the log with same field is assigned to based on same in the kafka system
In worker stroke.
8. a kind of device for merging log, comprising:
Writing module, suitable for that will receive in the non-persistentization caching and persistence caching that daily record reproduction write-in is arranged, wherein institute
It states non-persistentization caching and persistence caching is less than to the caching duration of log to the caching duration of log；
Searching module, suitable for searching the log with same field in non-persistentization caching；
Merging module, suitable for from the log of the same field found obtain meet preset backtracking condition a variety of logs and close
And by the log buffer after merging into persistence caching.
9. a kind of electronic equipment, comprising:
Processor；And
It is arranged to the memory of storage computer executable instructions, the executable instruction makes the processor when executed
Execute the method according to claim 1-7 for merging log.
10. a kind of computer storage medium, wherein the computer storage medium stores one or more programs, one
Or multiple programs are wanted so that the electronic equipment is executed according to right when the electronic equipment for being included multiple application programs executes
Seek the described in any item methods for merging log of 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711489592.8A CN109992469B (en) | 2017-12-29 | 2017-12-29 | Method and device for merging logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711489592.8A CN109992469B (en) | 2017-12-29 | 2017-12-29 | Method and device for merging logs |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109992469A true CN109992469A (en) | 2019-07-09 |
| CN109992469B CN109992469B (en) | 2023-08-18 |
Family
ID=67110676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711489592.8A Active CN109992469B (en) | 2017-12-29 | 2017-12-29 | Method and device for merging logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109992469B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000698A (en) * | 2020-08-25 | 2020-11-27 | 青岛海尔科技有限公司 | Log recording method and device, storage medium and electronic device |
| CN112667686A (en) * | 2020-12-30 | 2021-04-16 | 中国农业银行股份有限公司 | Real-time streaming data splicing method and device |
| CN112988741A (en) * | 2021-02-04 | 2021-06-18 | 北京淇瑀信息科技有限公司 | Real-time service data merging method and device and electronic equipment |
| CN113656645A (en) * | 2020-05-12 | 2021-11-16 | 北京字节跳动网络技术有限公司 | Log consumption method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100725415B1 (en) * | 2005-12-24 | 2007-06-07 | 삼성전자주식회사 | Log compaction method and apparatus of database |
| CN105138481A (en) * | 2014-05-30 | 2015-12-09 | 华为技术有限公司 | Stored data processing method and apparatus and system |
| CN106502875A (en) * | 2016-10-21 | 2017-03-15 | 过冬 | A kind of daily record generation method and system based on cloud computing |
| CN106649627A (en) * | 2016-12-06 | 2017-05-10 | 杭州迪普科技股份有限公司 | Log searching method and device |
| CN106775498A (en) * | 2017-01-23 | 2017-05-31 | 深圳国泰安教育技术股份有限公司 | A kind of data cached synchronous method and system |
-
2017
- 2017-12-29 CN CN201711489592.8A patent/CN109992469B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100725415B1 (en) * | 2005-12-24 | 2007-06-07 | 삼성전자주식회사 | Log compaction method and apparatus of database |
| CN105138481A (en) * | 2014-05-30 | 2015-12-09 | 华为技术有限公司 | Stored data processing method and apparatus and system |
| CN106502875A (en) * | 2016-10-21 | 2017-03-15 | 过冬 | A kind of daily record generation method and system based on cloud computing |
| CN106649627A (en) * | 2016-12-06 | 2017-05-10 | 杭州迪普科技股份有限公司 | Log searching method and device |
| CN106775498A (en) * | 2017-01-23 | 2017-05-31 | 深圳国泰安教育技术股份有限公司 | A kind of data cached synchronous method and system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656645A (en) * | 2020-05-12 | 2021-11-16 | 北京字节跳动网络技术有限公司 | Log consumption method and device |
| CN112000698A (en) * | 2020-08-25 | 2020-11-27 | 青岛海尔科技有限公司 | Log recording method and device, storage medium and electronic device |
| CN112000698B (en) * | 2020-08-25 | 2023-09-19 | 青岛海尔科技有限公司 | Log recording method and device, storage medium and electronic device |
| CN112667686A (en) * | 2020-12-30 | 2021-04-16 | 中国农业银行股份有限公司 | Real-time streaming data splicing method and device |
| CN112988741A (en) * | 2021-02-04 | 2021-06-18 | 北京淇瑀信息科技有限公司 | Real-time service data merging method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109992469B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103902698B (en) | A kind of data-storage system and storage method | |
| US8364751B2 (en) | Automated client/server operation partitioning | |
| US9507821B2 (en) | Mail indexing and searching using hierarchical caches | |
| US11567681B2 (en) | Method and system for synchronizing requests related to key-value storage having different portions | |
| CN104123238A (en) | Data storage method and device | |
| CN110134648A (en) | Log processing method, device, equipment, system and computer readable storage medium | |
| CN110737682A (en) | cache operation method, device, storage medium and electronic equipment | |
| CN109992469A (en) | A kind of method and device merging log | |
| US11748357B2 (en) | Method and system for searching a key-value storage | |
| CN105574054A (en) | Distributed cache range query method, apparatus and system | |
| CN112463886A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN103353901A (en) | Orderly table data management method and system based on Hadoop distributed file system (HDFS) | |
| CN108875046A (en) | A kind of storage system access method, device and electronic equipment | |
| CN109947729A (en) | A kind of real-time data analysis method and device | |
| CN110196835A (en) | The processing method of metadata, the processing unit of metadata and electronic equipment | |
| CN105610881B9 (en) | Distributed cache range query method, device and system | |
| CN115423555A (en) | Commodity recommendation method and device, electronic equipment and storage medium | |
| CN107316245A (en) | Expense adjusts method and system | |
| US20090089071A1 (en) | Compressed non-indexed data storage | |
| CN114116908A (en) | Data management method and device and electronic equipment | |
| CN113139113A (en) | Search request processing method and device | |
| CN114168595A (en) | Data analysis method and device | |
| US10769110B2 (en) | Facilitating queries for interaction data with visitor-indexed data objects | |
| CN112328630A (en) | Data query method, device, equipment and storage medium | |
| CN106776772B (en) | Data retrieval method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230727 Address after: Room 03, 2nd Floor, Building A, No. 20 Haitai Avenue, Huayuan Industrial Zone (Huanwai), Binhai New Area, Tianjin, 300450 Applicant after: 3600 Technology Group Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Applicant before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |