CN107220287A - For the index managing method of log query, device, storage medium and equipment - Google Patents

For the index managing method of log query, device, storage medium and equipment Download PDF

Info

Publication number
CN107220287A
CN107220287A CN201710272219.0A CN201710272219A CN107220287A CN 107220287 A CN107220287 A CN 107220287A CN 201710272219 A CN201710272219 A CN 201710272219A CN 107220287 A CN107220287 A CN 107220287A
Authority
CN
China
Prior art keywords
index
target
temperature
ranking
cache
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710272219.0A
Other languages
Chinese (zh)
Inventor
吴擒龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Neusoft Corp
Original Assignee
Neusoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Neusoft Corp filed Critical Neusoft Corp
Priority to CN201710272219.0A priority Critical patent/CN107220287A/en
Publication of CN107220287A publication Critical patent/CN107220287A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2272Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2462Approximate or statistical queries

Abstract

This disclosure relates to a kind of for the index managing method of log query, device, storage medium and equipment, this method includes:The query pattern being related to by analyzing in current queries, it is determined that target corresponding with query pattern is indexed, the index temperature for then indexing target increases a unit temperature, obtains the index temperature after the renewal of target index;The current temperature ranking of target index is obtained further according to the index temperature after renewal;When current temperature ranking determines that target index meets preparatory condition, target is indexed into write cache.Mobile state renewal can be entered to the index in cache according to the search temperature of index, it is capable of the storage of long period in the caches so as to search for the higher index of temperature, so as to while ensureing that the higher index of search temperature can be by quick search, it also ensure that the index will not frequently be exchanged between cache and low speed caching again, it is to avoid the problem of being easily caused systematic function degradation.

Description

For the index managing method of log query, device, storage medium and equipment
Technical field
This disclosure relates to Internet technical field, more particularly to it is a kind of for the index managing method of log query, device, Storage medium and equipment.
Background technology
At present, the storage of journal file is typically to use distributed storage, and managed concentratedly, but be due to daily record Substantial amounts of information is all included in every daily record in file, the information of wherein most is all not the valuable information of tool, Therefore journal file progress centralized management can be taken into substantial amounts of memory space, causes the waste of storage resource, and it is directly right The content of a large amount of distributed information logs scans for that the degraded performance of system can be caused, and can not even realize.It is right in the prior art The centralized management that journal file is carried out generally includes data prediction, the operation of index is set up, at present in cache The way to manage of index is normally based on hit principle, and the way to manage lacks flexibility, easily causes index in cache Frequently exchanged between low speed caching, systematic function degradation can be caused when data volume is larger.
The content of the invention
The disclosure provides a kind of for the index managing method of log query, device, storage medium and equipment, for solving The managerial shortage flexibility of daily record index, is easily caused the problem of systematic function declines.
First aspect includes there is provided a kind of index managing method for log query, methods described:
The query pattern being related to by analyzing in current queries, it is determined that target corresponding with the query pattern is indexed;
The index temperature that the target is indexed increases a unit temperature, obtains the rope after the renewal of the target index Draw temperature;
The current temperature ranking that the target is indexed is obtained according to the index temperature after the renewal;
When determining that the target index meets preparatory condition according to the current temperature ranking, target index is write Enter cache.
Optionally, the query pattern being related to by analyzing in current queries, it is determined that corresponding with the query pattern Target is indexed, including:
The query pattern being related to by analyzing in current queries, obtains the aiming field to be inquired about of query pattern;
Target index corresponding with the query pattern is determined according to the aiming field.
Optionally, it is described to determine that target corresponding with the query pattern is indexed according to the aiming field, including:
The index for including the aiming field is determined whether there is according to the aiming field;
When there is one or more indexes comprising the aiming field, one or more of indexes are defined as institute State target index;
When in the absence of one or more indexes comprising the aiming field, by traveling through day according to the aiming field Will file indexes to create one or more indexes comprising the aiming field as the target.
Optionally, the index temperature according to after the renewal obtains the current temperature ranking that the target is indexed, bag Include:
According to the index temperature after the renewal, all indexes are arranged according to the order of index temperature from high to low Name, obtains the current temperature ranking of the target index;
Wherein, when there is multiple index temperature identicals indexes, by the multiple index temperature identical index according to Index the order of the renewal time of temperature from the front to the back and carry out ranking.
Optionally, it is described when determining that the target index meets preparatory condition according to the current temperature ranking, by institute Target index write cache is stated, including:
Determine whether the target index is located in cache;
When the target index be not in the cache when, determine the current temperature ranking whether be more than or Equal to default ranking;
When the current temperature ranking is more than or equal to the default ranking, determine that the target index meets described pre- If condition, and the target is indexed into write cache;
When the current temperature ranking is less than the default ranking, determine that the target index is unsatisfactory for the default bar The target is indexed write cache by part, refusal.
Optionally, it is described when the current temperature ranking is more than or equal to the default ranking, determine the target rope Draw and meet the preparatory condition, and the target is indexed into write cache, including:
When the current temperature ranking is more than or equal to the default ranking, determine that being used in the cache deposits Whether the buffer memory capacity for storing up index is enough;
When it is determined that the buffer memory capacity is enough, the target is indexed into write cache;
When it is determined that the buffer memory capacity is not enough, temperature ranking is most in all indexes that will be stored in the cache The index of latter position is deleted from the cache;
It is being after last index is deleted from the cache, to perform the temperature ranking again described true Being used in the fixed cache stores the whether enough steps of the buffer memory capacity indexed, until determining the buffer memory capacity foot The target is indexed into write cache when enough;
Wherein, when it is last index that target index, which is the temperature ranking, refuse the target rope Draw write cache.
Optionally, it is described that the target is indexed into write cache, including:
When target index is the index created first, the target index is set up in the cache;
When target index is present in low speed caching, read from low speed caching after the target index, will The target indexes write cache.
Optionally, methods described also includes:
When determining that the target index is unsatisfactory for preparatory condition according to the current temperature ranking, the target is indexed It is retained in low speed caching.
Optionally, methods described also includes:
When target index is located at the cache, is indexed and obtained using the target from the cache The content to be inquired about of the query pattern;
When target index is located at low speed caching, is indexed and obtained using the target from low speed caching The content to be inquired about of the query pattern.
Second aspect includes there is provided a kind of index management device for log query, described device:
Index determining module, for the query pattern being related to by analyzing in current queries, it is determined that with the query pattern Corresponding target index;
Temperature update module is indexed, the index temperature for the target to be indexed increases a unit temperature, obtains institute State the index temperature after the renewal of target index;
Temperature ranking acquisition module, for obtaining the current heat that the target is indexed according to the index temperature after the renewal Spend ranking;
Index management module, determines that the target index meets preparatory condition for working as according to the current temperature ranking When, the target is indexed into write cache.
Optionally, the index determining module, including:
Field acquisition submodule, for the query pattern being related to by analyzing in current queries, obtains the query pattern The aiming field to be inquired about;
Determination sub-module is indexed, for determining that target corresponding with the query pattern is indexed according to the aiming field.
Optionally, the index determination sub-module, including:
Field recognizes submodule, for determining whether there is the rope for including the aiming field according to the aiming field Draw;
Index creation submodule, for when there is one or more indexes comprising the aiming field, by described one Individual or multiple indexes are defined as the target index;
The index creation submodule, is additionally operable to when in the absence of one or more indexes comprising the aiming field, It is used as institute by traveling through journal file according to the aiming field to create one or more indexes comprising the aiming field State target index.
Optionally, the temperature ranking acquisition module, is used for:
According to the index temperature after the renewal, all indexes are arranged according to the order of index temperature from high to low Name, obtains the current temperature ranking of the target index;
Wherein, when there is multiple index temperature identicals indexes, by the multiple index temperature identical index according to Index the order of the renewal time of temperature from the front to the back and carry out ranking.
Optionally, the index management module, including:
Index position determination sub-module, for determining whether the target index is located in cache;
Temperature ranking determination sub-module, for when target index is not in the cache, determining institute State whether current temperature ranking is more than or equal to default ranking;
Cache management submodule, for when the current temperature ranking is more than or equal to the default ranking, determining institute State target index and meet the preparatory condition, and the target is indexed into write cache;
The cache management submodule, is additionally operable to, when the current temperature ranking is less than the default ranking, determine institute State target index and be unsatisfactory for the preparatory condition, the target is indexed write cache by refusal.
Optionally, the cache management submodule, including:
Buffer memory capacity determination sub-module, for when the current temperature ranking be more than or equal to the default ranking when, really Whether the buffer memory capacity that being used in the fixed cache stores index is enough;
Write-in management submodule, it is for when it is determined that the buffer memory capacity is enough, the write-in of target index is slow at a high speed Deposit;
Index deletes submodule, for when it is determined that the buffer memory capacity is not enough, by the institute stored in the cache It is that last index is deleted from the cache to have temperature ranking in index;
The buffer memory capacity determination sub-module, is additionally operable to be last index from the height by the temperature ranking After being deleted in speed caching, the buffer memory capacity for being used to store index determined in the cache is performed again whether enough The step of, until determining the target index write cache when buffer memory capacity is enough;
Said write manages submodule, is additionally operable to when target index is that the temperature ranking is last index When, the target is indexed write cache by refusal.
Optionally, the index management module, is used for:
When target index is the index created first, the target index is set up in the cache;
When target index is present in low speed caching, read from low speed caching after the target index, will The target indexes write cache.
Optionally, the index management module is additionally operable to:
When determining that the target index is unsatisfactory for preparatory condition according to the current temperature ranking, the target is indexed It is retained in low speed caching.
Optionally, described device also includes:Enquiry module, is used for:
When target index is located at the cache, is indexed and obtained using the target from the cache The content to be inquired about of the query pattern;
When target index is located at low speed caching, is indexed and obtained using the target from low speed caching The content to be inquired about of the query pattern.
The third aspect is stored thereon with computer program, the computer journey there is provided a kind of computer-readable recording medium The step of first aspect methods described is realized when sequence is executed by processor.
Fourth aspect there is provided a kind of electronic equipment, including:
Computer-readable recording medium described in the third aspect;And one or more processor, for performing State the computer program in computer-readable recording medium.
The disclosure provide be used for the index managing method of log query, device, storage medium and equipment, by analysis ought The query pattern being related in preceding inquiry, it is determined that target corresponding with the query pattern is indexed;The index that the target is indexed Temperature increases a unit temperature, obtains the index temperature after the renewal of the target index;According to the index after the renewal Temperature obtains the current temperature ranking of the target index;Met when determining that the target is indexed according to the current temperature ranking During preparatory condition, the target is indexed into write cache.It can carry out ranking to judge by the search temperature to index Whether target is indexed in write cache, the index in cache carried out so as to the search temperature according to index Dynamic updates, so that searching for the higher index of temperature is capable of the storage of long period in the caches, so as to ensure to search for While the higher index of temperature can be by quick search, it also ensure that the index will not be exchanged frequently in cache again Between low speed caching, it is to avoid the problem of being easily caused systematic function degradation.
Other feature and advantage of the disclosure will be described in detail in subsequent embodiment part.
Brief description of the drawings
Accompanying drawing is, for providing further understanding of the disclosure, and to constitute a part for specification, with following tool Body embodiment is used to explain the disclosure together, but does not constitute limitation of this disclosure.In the accompanying drawings:
Fig. 1 is a kind of flow chart of index managing method for log query according to an exemplary embodiment;
Fig. 2 is the flow of another index managing method for log query according to an exemplary embodiment Figure;
Fig. 3 be according to an exemplary embodiment another be used for log query index managing method flow Figure;
Fig. 4 be according to an exemplary embodiment another be used for log query index managing method flow Figure;
Fig. 5 be according to an exemplary embodiment another be used for log query index managing method flow Figure;
Fig. 6 is a kind of block diagram of index management device for log query according to an exemplary embodiment;
Fig. 7 is a kind of block diagram of index determining module according to an exemplary embodiment;
Fig. 8 is a kind of block diagram of index determination sub-module according to an exemplary embodiment;
Fig. 9 is a kind of block diagram of index management module according to an exemplary embodiment;
Figure 10 is a kind of block diagram of cache management submodule according to an exemplary embodiment;
Figure 11 is the block diagram of another index management device inquired about for day according to an exemplary embodiment;
Figure 12 is the block diagram of a kind of electronic equipment according to an exemplary embodiment;
Figure 13 is the block diagram of another electronic equipment according to an exemplary embodiment.
Embodiment
It is described in detail below in conjunction with accompanying drawing embodiment of this disclosure.It should be appreciated that this place is retouched The embodiment stated is merely to illustrate and explained the disclosure, is not limited to the disclosure.
Fig. 1 is a kind of flow chart of index managing method for log query according to an exemplary embodiment, As shown in figure 1, this method comprises the following steps:
Step 110, the query pattern being related to by analyzing in current queries, it is determined that target rope corresponding with query pattern Draw.
The file with certain rule generated when daily record is system operation, usual computer, the network equipment, system and clothes Business program etc., operationally can all produce historical events record, per a line daily record all recite the date, the time, user and The description of the associative operations such as action.After computer system generates daily record, day that can be by way of inquiry to having generated The content of will file (a plurality of daily record can be included in journal file) is transferred, so as to in computer system it is specific some Some certain content in journal file is re-read, so as to obtain the operation shape of the computer system under the log recording The relevant information of state.Illustratively, before inquiry is performed to log content, the daily record text of all distributed storages can be traveled through Part, and according to the corresponding daily record mapping table of Model Establishment of daily record, the content of wherein daily record mapping table includes:Day aims at mapping table In identity (such as ID), field information, the positional information where the daily record, and positional information includes the log path (example The routing informations such as main frame, disk, file as where daily record), the length of the offset of the daily record and the daily record.By upper The method for setting up mapping table stated, can reduce the information content of centralized management, so as to reduce empty to the storage needed for log management Between.Every corresponding index of mapping is set up further according to the daily record mapping table afterwards, and index temperature is set up for each index The ranking of the search temperature of current all indexes is have recorded in sequencing table, the index temperature sequencing table, in any one index It is queried and is once required for the search temperature for updating this index afterwards, so that the following index temperature to after renewal carries out temperature Again ranking, so that the ranking of the search temperature newest according to the index is managed to the index in the buffer in subsequent operation Reason.The above method reduction centralized management information content, reduction log management needed for memory space while, by index Dynamically managed, accelerate inquiry velocity, reduce the exchange expense between high speed/low speed caching.
Therefore in this step, query pattern can be understood as user-defined query statement, i.e. user based on being inquired about One section of sentence for inquiring about the content related to these aiming fields that certain several aiming field is write.Indexed on target It is determined that can include:Determine whether target index has been present according to the query pattern of daily record first, if directly right in the presence of if Target index carries out the operation of next step 120, conversely, when target index is not present, then needing a first newly-built target Index, and the index temperature of target index now, should be default initial temperature, for example, could be arranged to 0, enter again afterwards Row step 120.
Step 120, index temperature target indexed increases a unit temperature, obtains the rope after the renewal of target index Draw temperature.
After the index of the target corresponding to query pattern is determined in step 110, then need to search the target index Rope temperature number of times is increased.For example, a unit temperature can be represented with 1.When performing inquiry, find there is the target During index, then add 1 to the index temperature that the target is indexed, obtain the index temperature after the renewal of target index, then walked Rapid 130 carry out the renewal of temperature ranking;Or, when it is newly-built index in step 110 that the target, which is indexed, the now target The temperature of index should be 1.
Step 130, the current temperature ranking that target is indexed is obtained according to the index temperature after renewal.
Because the index temperature in the step 120 to corresponding target index in this inquiry operation is updated, because The current temperature ranking of this corresponding target index is also required to update the current temperature ranking for obtaining target index again again.
According to the index temperature after renewal, it will index and carry out ranking according to the order of index temperature from high to low, obtain mesh Mark the current temperature ranking of index.Wherein, when there is multiple index temperature identical indexes, by multiple index temperature identicals Index carries out ranking according to the order of the renewal time of temperature from the front to the back is indexed.When there is ranking arranged side by side in multiple indexes, position It is that ranking is carried out according to its renewal time for indexing temperature from the front to the back in all and column index orders of the juxtaposed position, The ranking of the index updated more afterwards more before, that is, when there is multiple index temperature identicals indexes, the index of final updating First sorted side by side is come, then ranking from the front to the back in chronological order successively.
Step 140, it is when determining that target index meets preparatory condition according to current temperature ranking, target index write-in is high Speed caching.
Wherein, cache is the spatial cache for storing the index being commonly used, similar to internal memory or SSD (English Text:Solid-State Disk, Chinese:Solid state hard disc), be used in usual cache storing the space of index have it is certain Capacity limit.
In the present embodiment, target index write cache is included:It is existing and positioned at vulgar in target cache When in caching, target index is cached into write cache from low speed, or in target cache in the absence of (now index temperature is arranged The target index just created has been added in sequence table, but has been not present in caching), it is when creating for the first time, then to need synchronous at a high speed The target cache is created in caching.But it is due to that each index needs to take certain memory space, thus it is slow at a high speed in write-in , it is necessary to be judged first before in depositing, whether the target cache meets the preparatory condition of write-in, then the operation write, Judge the explanation referring to the following examples the step of now whether meet the preparatory condition of write cache.So at a high speed The index stored in caching will be index in have higher thermal degree ranking index, by each index search temperature dynamically Index in cache is updated, so that the management of optimum indexing.
In summary, the index managing method for log query that the disclosure is provided, is related to by analyzing in current queries And query pattern, it is determined that target corresponding with query pattern is indexed, the index temperature for then indexing target increases a list Position temperature, obtains the index temperature after the renewal of target index;Working as target index is obtained further according to the index temperature after renewal Preceding temperature ranking;When current temperature ranking determines that target index meets preparatory condition, target is indexed into write cache.Energy It is enough to carry out ranking to judge whether to index target in write cache by the search temperature to index, so as to basis The search temperature of index enters Mobile state renewal to the index in cache, so as to search for the higher index of temperature when can be longer Between storage in the caches, so as to while ensureing that the higher index of search temperature can be by quick search, also ensure The index will not frequently be exchanged between cache and low speed caching again, it is to avoid it is serious to be easily caused systematic function The problem of decline.
Further, Fig. 2 is another index management side for log query according to an exemplary embodiment The flow chart of method, as shown in Fig. 2 the query pattern being related to by analyzing in current queries described in step 110 in Fig. 1, really Fixed target index corresponding with query pattern includes:
Step 111, the query pattern being related to by analyzing in current queries, obtains the target word to be inquired about of query pattern Section.
Illustratively, the determination of query pattern is built upon on the basis of the rule of generation daily record, should be only right for inquiry A certain partial content carries out a regular extraction, can be with filter condition or canonical.And query pattern institute The statement of corresponding aiming field should be unique, therefore to being that can determine that the aiming field to be inquired about according to query pattern, So as to carry out the operation of step 112, indexed with determining whether there is corresponding target.
Step 112, determine that target corresponding with query pattern is indexed according to aiming field.
Example, determine in step 111 after the aiming field to be inquired about determined further according to the aiming field be It is no to there is the index comprising aiming field, wherein in the present embodiment, search whether to refer in the presence of the index comprising aiming field Searched in above-mentioned index temperature sequencing table;Target index is determined afterwards, that is,:When (in index temperature sequencing table) has bag During one or more indexes containing aiming field, one or more indexes are defined as target index;Or, when (index temperature In sequencing table) when one or more indexes comprising aiming field are not present, by according to aiming field travel through journal file come One or more indexes comprising aiming field are created to index as target, and after target index is created, by the target rope Draw and be added in the index temperature sequencing table, the index created in addition includes the aiming field, the main frame road where journal file Offset in footpath, journal file, the information such as the length of daily record.
Example, it is assumed that need to search the daily record that ip is 10.4.45.129 in journal file, then can be in index temperature Inquiry is with the presence or absence of the index for including ip fields in sequencing table, and the wherein description of query pattern for example can be:^(<ip>\d +\.\d+\.\d+)[\s\S].When there is the index comprising ip fields, the index is exactly the target index that we need, when not , it is necessary to create the target index during in the presence of index comprising ip fields, establishment process can include:Travel through in specified range and own Distributed information log file, according to above-mentioned query pattern ^ (<ip>D+ d+ d+) [s S] extract every daily record It is worth and sets up index, index includes:Offset in host paths where ip value, journal file, journal file, daily record The information such as length.
Then, the index managing method described in step 141 to 145 is performed, no matter whether target index is located at height afterwards On speed caching, ip=10.4.45.129 daily record is all filtered on target index.
Optionally, Fig. 3 be according to an exemplary embodiment another be used for log query index managing method Flow chart, as shown in figure 3, described in step 140 in Fig. 1 when determining that target index meets default according to current temperature ranking During condition, target index write cache is included:
Step 141, determine whether target index is located in cache.
Judge whether target index is located in cache, when target index is had been positioned in cache, explanation Do not need in write cache again;Otherwise with regard to the operation of progress step 142.
Step 142, when target index is not in cache, it is determined that whether current temperature ranking is more than or equal to Default ranking.
When judging that target index is not in cache, further according to its current temperature ranking, step 143 is carried out Or the operation of step 144.
Step 143, when current temperature ranking is more than or equal to default ranking, judge whether target index meets default bar Part.
Example, when the current temperature ranking that the target is indexed is the 20th, and default ranking is 30, then explanation can be with The step of whether meeting preparatory condition is determined further, judges the step of target index meets preparatory condition referring to following Fig. 4 embodiment.
The judged result of this step there is also two kinds of situations, the first:When meet target index meet preparatory condition when, will Target indexes write cache.Wherein, target index write cache is also included:In the first case:Work as target When index is the index created first, target index is set up in cache;In second of situation:When target index is present in When low speed is cached, read from low speed caching after target index, target is indexed into write cache.That is, the inquiry Corresponding target is operated to index when being not present in caching, it is necessary to when newly-built, directly can set up target index in height On speed caching;Another situation is exactly that target index had built up, but index temperature before ranks behind, therefore It is stored in low speed caching, and now the target indexes the temperature ranking of temperature ranking after temperature ranking after correspondence updates Meet preparatory condition again, therefore write cache again after being read from low speed caching.
Second, when discontented foot-eye index meets preparatory condition, the operation of step 145 is carried out, that is, by the mesh Mark index is stayed in low speed caching.
Step 144, when current temperature ranking is less than default ranking, determine that target index is unsatisfactory for preparatory condition, refuse Target is indexed into write cache.
Example, when default ranking is 30, and the current temperature ranking of target index is when being 35, i.e. current temperature row Name is less than default ranking, it is determined that the target can not be indexed into write cache, then continues the operation of step 145.
Step 145, when determining that target index is unsatisfactory for preparatory condition according to current temperature ranking, target is indexed and retained In low speed caching.
When target index is unsatisfactory for the preparatory condition of write cache, two kinds of situations can also be divided into, when target rope Draw when being the index created first, then by target index creation in low speed caching, delay when target index is already present on low speed In depositing, then target index also retains in the original location, i.e., in low speed caching, until meeting preparatory condition according to its temperature ranking Afterwards, just can be with write cache.
Further, Fig. 4 be according to an exemplary embodiment another be used for log query index management side The flow chart of method, when judging whether target index meets preparatory condition, the current temperature ranking except considering target index, also It is contemplated that whether the buffer memory capacity for being used to store index in cache is enough, as shown in figure 4, described in the step 144 in Fig. 3 When current temperature ranking is more than or equal to default ranking, determine that target index meets preparatory condition, and target index is write Entering cache includes:
Step 1441, when current temperature ranking is more than or equal to default ranking, determine that being used in cache stores Whether the buffer memory capacity of index is enough.
Example, the buffer memory capacity can choose 30 percent of total capacity shared by all indexes and system distributed Minimum value in the two of largest buffered capacity limit.Judging that being used in cache now is stored in the caching of index is No stored full, whether its residual caching capacity also writes new index enough, then judges to carry out step 1442 or step Rapid 1443 operation.
Step 1442, when it is determined that buffer memory capacity is enough, target is indexed into write cache.
Step 1443, when it is determined that buffer memory capacity is not enough, it is by temperature ranking in all indexes stored in cache Last index is deleted from cache.
Judge the buffer memory capacity in the cache when not enough continuing to store, it is necessary to delete stored in step 1441 Index in cache, obtains enough buffer memory capacity to be stored in target index, it is the last that selection will index temperature Index is deleted.Carry out the judgement of step 1444 again afterwards.
Step 1444, it is determined whether buffer memory capacity enough indexes target in write cache.
After the last index is deleted in step 1443, it is determined that now there is enough capacity to carry out mesh in cache When marking the write-in of index, the deterministic process can be terminated, target is directly indexed into write cache.Otherwise need to be from step 1441 Restart, untill step 1444 determines to meet.It is, being that last index is slow from high speed by temperature ranking Deposit after deletion, perform determine that being used in cache stores the whether enough steps of the buffer memory capacity indexed again, until Determine when buffer memory capacity is enough target indexing write cache.Wherein, when target index is that temperature ranking is last position Index when, refusal target is indexed into write cache.
Because the space shared by different index is inconsistent and buffer memory capacity of cache is limited, therefore in ablation process Whether the middle buffer memory capacity that need to be determined in cache meets the requirement of the required amount of capacity of the target index that will be write, When being unsatisfactory for, then the cleaning of buffer memory capacity is carried out first, the last index of temperature is deleted successively, until buffer memory capacity size Untill meeting the shared space size of target index.For example, the buffer memory capacity in cache can store 30 indexes, Target indexes the 20th that ranking at this moment is the index temperature ranking of all indexes in cache, therefore the target to be write During index, it need to delete and have the index temperature rankings of 31 indexes including being indexed plus target altogether for the 31st, i.e., script temperature is arranged The index of entitled 30th.And if after the last index of index temperature is deleted, the buffer memory capacity of cache Still not enough needed for target index, then still need to proceed the operation of step 1443, delete temperature ranking at the 30th Index, by that analogy, untill it can be completely written in cache by target index.But, delete ranking if worked as After the target index of the 21st, still then also to stop write-in without enough buffer memory capacity in the caches, because this When the target index ranking be last position.
Optionally, Fig. 5 be according to an exemplary embodiment another be used for log query index managing method Flow chart, as shown in figure 5, further comprising the steps of after step shown in embodiment in Fig. 1:
Step 150, when target index is located at cache, query pattern is obtained using target index from cache The content to be inquired about.
Step 160, when target index is located at low speed caching, query pattern is obtained using target index from low speed caching The content to be inquired about.
In addition, for the inquiry of data statistics type, the scene such as inquiring about the amount of money can be previously according to related field Content creating is indexed, and statistical result can be extracted from index in inquiry, without the former daily record of inquiry, it is possible to increase inquiry effect Rate.
In summary, the index managing method for log query that the disclosure is provided, is related to by analyzing in current queries And query pattern, it is determined that target corresponding with query pattern is indexed, the index temperature for then indexing target increases a list Position temperature, obtains the index temperature after the renewal of target index;Working as target index is obtained further according to the index temperature after renewal Preceding temperature ranking;When current temperature ranking determines that target index meets preparatory condition, target is indexed into write cache.Energy It is enough to carry out ranking to judge whether to index target in write cache by the search temperature to index, so as to basis The search temperature of index enters Mobile state renewal to the index in cache, so as to search for the higher index of temperature when can be longer Between storage in the caches, so as to while ensureing that the higher index of search temperature can be by quick search, also ensure The index will not frequently be exchanged between cache and low speed caching again, it is to avoid it is serious to be easily caused systematic function The problem of decline.
Fig. 6 is a kind of block diagram of index management device for log query according to an exemplary embodiment, should Device 600 can be used for performing any described methods of Fig. 1-Fig. 5.Referring to Fig. 6, the device 600 includes:
Index determining module 610, for the query pattern being related to by analyzing in current queries, it is determined that with query pattern pair The target index answered.
Temperature update module 620 is indexed, the index temperature for target to be indexed increases a unit temperature, obtains target Index temperature after the renewal of index.
Temperature ranking acquisition module 630, for obtaining the current temperature row that target is indexed according to the index temperature after renewal Name.
Index management module 640, for when determining that target index meets preparatory condition according to current temperature ranking, by mesh Mark index write cache.
Optionally, Fig. 7 is a kind of block diagram of index determining module according to an exemplary embodiment, as shown in fig. 7, The index determining module 610, including:
Field acquisition submodule 611, for the query pattern being related to by analyzing in current queries, obtains query pattern institute The aiming field to be inquired about.
Determination sub-module 612 is indexed, for determining that target corresponding with query pattern is indexed according to aiming field.
Optionally, Fig. 8 is a kind of block diagram of index determination sub-module according to an exemplary embodiment, such as Fig. 8 institutes Show, the index determination sub-module 612, including:
Field recognizes submodule 6121, for determining whether there is the index comprising aiming field according to aiming field.
Index creation submodule 6122, for when there is one or more indexes comprising aiming field, by one or Multiple indexes are defined as target index.
The index creation submodule 6122, is additionally operable to when in the absence of one or more indexes comprising aiming field, It is used as target creating one or more indexes comprising aiming field by traveling through journal file according to aiming field to index.
Optionally, temperature ranking acquisition module 630, is used for:
According to the index temperature after renewal, all indexes are subjected to ranking according to the order of index temperature from high to low, obtained The current temperature ranking indexed to target.
Wherein, when there is multiple index temperature identical indexes, by multiple index temperature identical indexes according to index The order of the renewal time of temperature from the front to the back carries out ranking.
Optionally, Fig. 9 is a kind of block diagram of index management module according to an exemplary embodiment, as shown in figure 9, The index management module 640, including:
Index position determination sub-module 641, for determining whether target index is located in cache.
Temperature ranking determination sub-module 642, for when target index is not in cache, it is determined that current temperature Whether ranking is more than or equal to default ranking.
Cache management submodule 643, for when current temperature ranking is more than or equal to default ranking, determining that target is indexed Preparatory condition is met, and target is indexed into write cache.
Cache management submodule 643, is additionally operable to, when current temperature ranking is less than default ranking, determine that target index is discontented Target is indexed write cache by sufficient preparatory condition, refusal.
Optionally, Figure 10 is a kind of block diagram of cache management submodule according to an exemplary embodiment, such as Figure 10 It is shown, the cache management submodule 643, including:
Buffer memory capacity determination sub-module 6431, for when current temperature ranking is more than or equal to default ranking, it is determined that high Whether the buffer memory capacity that being used in speed caching stores index is enough.
Write-in management submodule 6432, for when it is determined that buffer memory capacity is enough, target to be indexed into write cache.
Index deletes submodule 6433, for when it is determined that buffer memory capacity is not enough, by all ropes stored in cache Draw temperature ranking to delete from cache for last index.
Buffer memory capacity determination sub-module 6431, for being that last index is deleted from cache by temperature ranking After removing, perform determine that being used in cache stores the whether enough steps of the buffer memory capacity indexed again, until determining slow Deposit capacity it is enough when target is indexed into write cache.
Write-in management submodule 6432, is additionally operable to when it is last index that target index, which is temperature ranking, refusal Target is indexed into write cache.
Optionally, index management module 640, are used for:
When target index is the index created first, target index is set up in cache.
When target index is present in low speed caching, is read from low speed caching after target index, target is indexed and write Cache.
Optionally, the index management module 640 is additionally operable to:
When determining that target index is unsatisfactory for preparatory condition according to current temperature ranking, target index is retained in low speed and delayed Deposit.
Optionally, Figure 11 is a kind of frame of index management device inquired about for day according to an exemplary embodiment Figure, as shown in figure 11, the device 600 also include enquiry module 650, are used for:
When target index is located at cache, obtain query pattern to be inquired about using target index from cache Content;And when target index is located at low speed caching, utilizes target index to obtain query pattern from low speed caching and want The content of inquiry.
In summary, the index management device for log query that the disclosure is provided, is related to by analyzing in current queries And query pattern, it is determined that target corresponding with query pattern is indexed, the index temperature for then indexing target increases a list Position temperature, obtains the index temperature after the renewal of target index;Working as target index is obtained further according to the index temperature after renewal Preceding temperature ranking;When current temperature ranking determines that target index meets preparatory condition, target is indexed into write cache.Cause This, the technical scheme that the disclosure is provided can carry out ranking by the search temperature to index, it is determined whether index target In write cache, the index in cache is updated in real time, realized in the operating process of log query to rope The dynamic management drawn.
Figure 12 is the block diagram of a kind of electronic equipment 1200 according to an exemplary embodiment.As shown in figure 12, the electricity Sub- equipment 1200 can include:Processor 1201, memory 1202, multimedia groupware 1203, input/output (I/O) interface 1204, and communication component 1205.
Wherein, processor 1201 is used to control the integrated operation of the electronic equipment 1200, to complete above-mentioned to be used for daily record All or part of step in the index managing method of inquiry.Memory 1202 is used to store various types of data to support The operation of the electronic equipment 1200, these data can for example include any application for being used to operate on the electronic equipment 1200 The instruction of program or method, and the related data of application program, such as contact data, the message of transmitting-receiving, picture, audio, Video etc..The memory 1202 can be real by any kind of volatibility or non-volatile memory device or combinations thereof It is existing, such as static RAM (Static Random Access Memory, abbreviation SRAM), electric erasable can be compiled Journey read-only storage (Electrically Erasable Programmable Read-Only Memory, abbreviation EEPROM), Erasable Programmable Read Only Memory EPROM (Erasable Programmable Read-Only Memory, abbreviation EPROM), can be compiled Journey read-only storage (Programmable Read-Only Memory, abbreviation PROM), read-only storage (Read-Only Memory, abbreviation ROM), magnetic memory, flash memory, disk or CD.Multimedia groupware 1203 can include screen and sound Frequency component.Wherein screen for example can be touch-screen, and audio-frequency assembly is used to export and/or input audio signal.For example, audio group Part can include a microphone, and microphone is used to receive external audio signal.The audio signal received can be further It is stored in memory 1202 or is sent by communication component 1205.Audio-frequency assembly also includes at least one loudspeaker, for exporting Audio signal.I/O interfaces 1204 is provide interface between processor 1201 and other interface modules, other above-mentioned interface modules can To be keyboard, mouse, button etc..These buttons can be virtual push button or entity button.Communication component 1205 is used for the electricity Wired or wireless communication is carried out between sub- equipment 1200 and other equipment.Radio communication, such as Wi-Fi, bluetooth, near-field communication (Near Field Communication, abbreviation NFC), 2G, 3G or 4G, or one or more of combinations in them, therefore The corresponding communication component 1205 can include:Wi-Fi module, bluetooth module, NFC module.
In one exemplary embodiment, electronic equipment 1200 can be by one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviation ASIC), digital signal processor (Digital Signal Processor, abbreviation DSP), digital signal processing appts (Digital Signal Processing Device, Abbreviation DSPD), PLD (Programmable Logic Device, abbreviation PLD), field programmable gate array (Field Programmable Gate Array, abbreviation FPGA), controller, microcontroller, microprocessor or other electronics member Part realization, the index managing method for log query above-mentioned for performing.
In a further exemplary embodiment, a kind of computer-readable recording medium including programmed instruction, example are additionally provided Such as include the memory 1202 of programmed instruction, said procedure instruction can be performed to complete by the processor 1201 of electronic equipment 1200 The above-mentioned index managing method for log query.
Figure 13 is the block diagram of another electronic equipment 1300 according to an exemplary embodiment.For example, electronic equipment 1300 may be provided in a server.Reference picture 13, electronic equipment 1300 includes processor 1322, and its quantity can be one Or it is multiple, and memory 1332, for storing the computer program that can be performed by processor 1322.Stored in memory 1332 Computer program can include it is one or more each correspond to the module of one group of instruction.In addition, processor 1322 can be configured as performing the computer program, to perform the above-mentioned index managing method for log query.
In addition, electronic equipment 1300 can also include power supply module 1326 and communication component 1350, the power supply module 1326 The power management of electronic equipment 1300 can be configured as performing, the communication component 1350 can be configured as realizing electronic equipment 1300 communication, for example, wired or wireless communication.In addition, the electronic equipment 1300 can also connect including input/output (I/O) Mouth 1358.Electronic equipment 1300 can be operated based on the operating system for being stored in memory 1332, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM etc..
In a further exemplary embodiment, a kind of computer-readable recording medium including programmed instruction, example are additionally provided Such as include the memory 1332 of programmed instruction, said procedure instruction can be performed to complete by the processor 1322 of electronic equipment 1300 The above-mentioned index managing method for log query.
The preferred embodiment of the disclosure is described in detail above in association with accompanying drawing, still, the disclosure is not limited to above-mentioned reality The detail in mode is applied, in the range of the technology design of the disclosure, a variety of letters can be carried out with technical scheme of this disclosure Monotropic type, these simple variants belong to the protection domain of the disclosure.
It is further to note that each particular technique feature described in above-mentioned embodiment, in not lance In the case of shield, it can be combined by any suitable means.In order to avoid unnecessary repetition, the disclosure to it is various can The combination of energy no longer separately illustrates.
In addition, can also be combined between a variety of embodiments of the disclosure, as long as it is without prejudice to originally Disclosed thought, it should equally be considered as disclosure disclosure of that.

Claims (10)

1. a kind of index managing method for log query, it is characterised in that methods described includes:
The query pattern being related to by analyzing in current queries, it is determined that target corresponding with the query pattern is indexed;
The index temperature that the target is indexed increases a unit temperature, obtains the index heat after the renewal of the target index Degree;
The current temperature ranking that the target is indexed is obtained according to the index temperature after the renewal;
It is when determining that the target index meets preparatory condition according to the current temperature ranking, target index write-in is high Speed caching.
2. according to the method described in claim 1, it is characterised in that the inquiry mould being related to by analyzing in current queries Formula, it is determined that target corresponding with the query pattern is indexed, including:
The query pattern being related to by analyzing in current queries, obtains the aiming field to be inquired about of query pattern;
Target index corresponding with the query pattern is determined according to the aiming field.
3. method according to claim 2, it is characterised in that described to be determined and the inquiry mould according to the aiming field The corresponding target index of formula, including:
The index for including the aiming field is determined whether there is according to the aiming field;
When there is one or more indexes comprising the aiming field, one or more of indexes are defined as the mesh Mark index;
When in the absence of one or more indexes comprising the aiming field, by traveling through daily record text according to the aiming field Part indexes to create one or more indexes comprising the aiming field as the target.
4. according to the method described in claim 1, it is characterised in that described in the index temperature according to after the renewal is obtained The current temperature ranking of target index, including:
According to the index temperature after the renewal, all indexes are subjected to ranking according to the order of index temperature from high to low, obtained The current temperature ranking indexed to the target;
Wherein, when there is multiple index temperature identical indexes, by the multiple index temperature identical index according to index The order of the renewal time of temperature from the front to the back carries out ranking.
5. according to the method described in claim 1, it is characterised in that described the mesh to be determined according to the current temperature ranking When mark index meets preparatory condition, the target is indexed into write cache, including:
Determine whether the target index is located in cache;
When target index is not in the cache, determine whether the current temperature ranking is more than or equal to Default ranking;
When the current temperature ranking is more than or equal to the default ranking, determine that the target index meets the default bar Part, and the target is indexed into write cache;
When the current temperature ranking is less than the default ranking, determine that the target index is unsatisfactory for the preparatory condition, The target is indexed write cache by refusal.
6. method according to claim 5, it is characterised in that described when the current temperature ranking is more than or equal to described During default ranking, determine that the target index meets the preparatory condition, and the target is indexed into write cache, bag Include:
When the current temperature ranking is more than or equal to the default ranking, determine that being used in the cache stores rope Whether the buffer memory capacity drawn is enough;
When it is determined that the buffer memory capacity is enough, the target is indexed into write cache;
When it is determined that the buffer memory capacity is not enough, temperature ranking is last in all indexes that will be stored in the cache The index of position is deleted from the cache;
It is being that after last index is deleted from the cache, the determination institute is performed again by the temperature ranking The whether enough step of buffer memory capacity that being used in cache store index is stated, until when determining that the buffer memory capacity is enough The target is indexed into write cache;
Wherein, when it is last index that target index, which is the temperature ranking, refusal writes target index Enter cache.
7. the method according to claim 5 or 6, it is characterised in that described that the target is indexed into write cache, bag Include:
When target index is the index created first, the target index is set up in the cache;
When target index is present in low speed caching, read from low speed caching after the target index, will be described Target indexes write cache.
8. a kind of index management device for log query, it is characterised in that described device includes:
Determining module is indexed, for the query pattern being related to by analyzing in current queries, it is determined that corresponding with the query pattern Target index;
Temperature update module is indexed, the index temperature for the target to be indexed increases a unit temperature, obtains the mesh Mark the index temperature after the renewal of index;
Temperature ranking acquisition module, for obtaining the current temperature row that the target is indexed according to the index temperature after the renewal Name;
Index management module, for when determining that the target index meets preparatory condition according to the current temperature ranking, inciting somebody to action The target indexes write cache.
9. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the computer program quilt The step of method any one of claim 1-7 is realized during computing device.
10. a kind of electronic equipment, it is characterised in that including:
Computer-readable recording medium described in claim 9;And
One or more processor, for performing the computer program in the computer-readable recording medium.
CN201710272219.0A 2017-04-24 2017-04-24 For the index managing method of log query, device, storage medium and equipment Pending CN107220287A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710272219.0A CN107220287A (en) 2017-04-24 2017-04-24 For the index managing method of log query, device, storage medium and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710272219.0A CN107220287A (en) 2017-04-24 2017-04-24 For the index managing method of log query, device, storage medium and equipment

Publications (1)

Publication Number Publication Date
CN107220287A true CN107220287A (en) 2017-09-29

Family

ID=59943917

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710272219.0A Pending CN107220287A (en) 2017-04-24 2017-04-24 For the index managing method of log query, device, storage medium and equipment

Country Status (1)

Country Link
CN (1) CN107220287A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209969A (en) * 2019-05-08 2019-09-06 口碑(上海)信息技术有限公司 The caching pre-heating mean and device of menu information
CN110704436A (en) * 2019-09-26 2020-01-17 郑州阿帕斯科技有限公司 Hbase-based index generation method and device
CN112000885A (en) * 2020-08-14 2020-11-27 广州市百果园信息技术有限公司 List service system, list generation method, server and storage medium
CN113918606A (en) * 2021-12-15 2022-01-11 湖北亿咖通科技有限公司 Log query method and device
WO2023179414A1 (en) * 2022-03-22 2023-09-28 阿里云计算有限公司 Hotspot recognition method and rate limiting method

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102508913A (en) * 2011-11-17 2012-06-20 张真 Cloud computing system with data cube storage index structure
CN102779185A (en) * 2012-06-29 2012-11-14 浙江大学 High-availability distribution type full-text index method
CN102929936A (en) * 2012-09-26 2013-02-13 东软集团股份有限公司 Log recording method, log inquiring method and system
CN104092670A (en) * 2014-06-25 2014-10-08 北京蓝汛通信技术有限责任公司 Method for utilizing network cache server to process files and device for processing cache files
CN105912696A (en) * 2016-04-25 2016-08-31 中国互联网络信息中心 DNS (Domain Name System) index creating method and query method based on logarithm merging

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102508913A (en) * 2011-11-17 2012-06-20 张真 Cloud computing system with data cube storage index structure
CN102779185A (en) * 2012-06-29 2012-11-14 浙江大学 High-availability distribution type full-text index method
CN102929936A (en) * 2012-09-26 2013-02-13 东软集团股份有限公司 Log recording method, log inquiring method and system
CN104092670A (en) * 2014-06-25 2014-10-08 北京蓝汛通信技术有限责任公司 Method for utilizing network cache server to process files and device for processing cache files
CN105912696A (en) * 2016-04-25 2016-08-31 中国互联网络信息中心 DNS (Domain Name System) index creating method and query method based on logarithm merging

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209969A (en) * 2019-05-08 2019-09-06 口碑(上海)信息技术有限公司 The caching pre-heating mean and device of menu information
CN110704436A (en) * 2019-09-26 2020-01-17 郑州阿帕斯科技有限公司 Hbase-based index generation method and device
CN112000885A (en) * 2020-08-14 2020-11-27 广州市百果园信息技术有限公司 List service system, list generation method, server and storage medium
WO2022033291A1 (en) * 2020-08-14 2022-02-17 百果园技术(新加坡)有限公司 Ranking list service system, ranking list generation method, server, and storage medium
CN113918606A (en) * 2021-12-15 2022-01-11 湖北亿咖通科技有限公司 Log query method and device
WO2023179414A1 (en) * 2022-03-22 2023-09-28 阿里云计算有限公司 Hotspot recognition method and rate limiting method

Similar Documents

Publication Publication Date Title
CN107220287A (en) For the index managing method of log query, device, storage medium and equipment
CN104301360B (en) A kind of method of logdata record, log server and system
CN102542052B (en) Priority hash index
CN110291518A (en) Merge tree garbage index
KR102036419B1 (en) Multi-level caching method for improving graph processing performance, and multi-level caching system
CN110268394A (en) KVS tree
CN110268399A (en) Merging tree for attended operation is modified
CN107943718B (en) Method and device for cleaning cache file
CN106202367A (en) A kind of processing method and processing device of object information
CN109271355A (en) A kind of method and device of cleaning journal file caching
CN104424219B (en) A kind of management method and device of data file
CN101673192A (en) Method for time-sequence data processing, device and system therefor
CN110737682A (en) cache operation method, device, storage medium and electronic equipment
CN106682186A (en) File access control list (ACL) management method and related device and system
CN105512283A (en) Data quality management and control method and device
CN106682042B (en) A kind of relation data caching and querying method and device
CN103020175B (en) A kind of method and apparatus obtaining aggregated data
CN106776823A (en) A kind of time series data management method, equipment and device
CN108280162A (en) A kind of location privacy protection method based on caching
CN105224560A (en) Data cached lookup method and device
CN110888837A (en) Object storage small file merging method and device
CN112262379A (en) Storing data items and identifying stored data items
CN114020790A (en) Data query method and device
CN109597574A (en) Distributed data storage method, server and readable storage medium storing program for executing
CN101576854A (en) File access method, device and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20170929

RJ01 Rejection of invention patent application after publication