CN113835986A - Dynamic switching method and device for system cold and hot logs and computer equipment - Google Patents
Dynamic switching method and device for system cold and hot logs and computer equipment Download PDFInfo
- Publication number
- CN113835986A CN113835986A CN202111408191.1A CN202111408191A CN113835986A CN 113835986 A CN113835986 A CN 113835986A CN 202111408191 A CN202111408191 A CN 202111408191A CN 113835986 A CN113835986 A CN 113835986A
- Authority
- CN
- China
- Prior art keywords
- log data
- data
- log
- hot
- stage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a device for dynamically switching system cold and hot logs and computer equipment, wherein the method comprises the following steps: entering a first stage, storing first log data generated by the business system in the first stage into a server hard disk of the business system, and analyzing second log data generated by the business system in a second stage to distinguish target hot log data and target cold log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into a server hard disk, so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk. The invention solves the technical problem of low log processing efficiency.
Description
Technical Field
The invention relates to the field of data processing, in particular to a method and a device for dynamically switching system cold and hot logs and computer equipment.
Background
In a large-scale service system, a large number of server nodes are provided, and a large number of logs are generated in the operation process, and are generally stored in a linux server or a relational database in a text-file form.
The log file or the database file after the java background is landed needs to be read again in the front-end BI system, and then the log file or the database file is processed into a rear-end data interface through the background, so that a large amount of time is consumed in the middle, and the log processing efficiency is low.
Disclosure of Invention
The embodiment of the invention provides a method and a device for dynamically switching system cold and hot logs and computer equipment, and solves the technical problem of low log processing efficiency in the prior art.
In a first aspect, an embodiment of the present invention provides a method for dynamically switching a system cold and hot log, including: entering a first stage, storing first log data generated by a service system in the first stage into a server hard disk of the service system, and collecting historical operation behavior data of a user group on the service system; entering a second stage, and periodically updating the historical operation behavior data of the operation behavior of the service system based on a user group in the second stage; in the second stage, analyzing second log data generated by the service system in the ith time period by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into the server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes integers which are sequentially increased from 1.
Optionally, the periodically updating the historical operating behavior data of the operating behavior of the business system in the second stage based on the user group includes: reading the first log data from the server hard disk into the server memory; performing big data analysis on the first log data read into the server memory based on the historical operation behavior data generated in the first stage to generate a back-end data interface; and calling a front-end page buried point of a front-end BI system through the back-end data interface so as to update the historical operation behavior data.
Optionally, after the storing the target hot log data into a server memory of the service system and storing the target cold log data into the server hard disk, the method further includes: performing ith sub-optimization processing on the back-end data interface by using target hot log data and target cold log data in second log data generated in the ith time period; and calling a front-end page embedded point of the calling front-end BI system based on the ith sub-optimized back-end data interface to continuously collect the operation behavior data of the user group to the service system in the (i + 1) th time period.
Optionally, the method further comprises: and when the data volume of the historical operation behavior data meets a preset number threshold or the duration of the first stage reaches a preset time threshold, triggering to end the first stage and entering the second stage.
Optionally, the analyzing, by using the operation behavior data updated in the ith time period, second log data generated by the service system in the ith time period to distinguish target hot log data and target cold log data in the second log data includes: and performing counting weighted calculation analysis on second log data generated in the ith time period by using the updated historical operation behavior data in the ith time period so as to distinguish target hot log data and target cold log data in the second log data.
Optionally, the performing count weighted calculation analysis on the second log data by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data includes: adjusting the weight value of the corresponding log in the second log data by using the updated operation behavior data in the ith time period; according to the weight value of each log in the second log data, log sorting is carried out on the second log data; and distinguishing hot log data and cold log data in the second log data according to the log sorting result.
Optionally, the distinguishing, according to the log sorting result, hot log data and cold log data in the second log data includes: in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, reducing the proportion of the target hot log data in the second log data so as to maintain the occupancy rate of the hot log data in the server memory to the server memory lower than the preset alarm threshold.
Optionally, the method further comprises: in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, stopping storing the target hot log data into the server memory; and storing the residual logs in the target hot log data and the target cold log data into the server hard disk.
In a second aspect, an embodiment of the present invention provides a device for dynamically switching a system hot and cold log, including:
the first log processing unit is used for entering a first stage, storing first log data generated by the business system in the first stage into a server hard disk of the business system, and collecting historical operation behavior data of a user group on the business system;
the second log processing unit is used for entering a second stage and periodically updating the historical operation behavior data on the basis of the operation behavior of the service system in the second stage of the user group; in the second stage, analyzing second log data generated by the service system in the ith time period by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into the server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes integers which are sequentially increased from 1.
In a third aspect, an embodiment of the present invention provides a computer device, which includes one or more processors and one or more memories, where at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the method described in any one of the embodiments of the first aspect.
One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:
the embodiment of the invention distinguishes the hot log data and the cold log data from the generated log data by utilizing the operation behavior data of the user group to the service system, only stores the hot log data in the server memory of the service system, and stores the cold log data in the server hard disk of the service system so as to dynamically maintain the dynamic update of the hot log data in the server memory and the cold log data in the server hard disk, thereby maintaining the high-efficiency utilization of server resources, analyzing the hot log data in the server memory directly after obtaining the hot log data, saving the time consumption from the reading of the hot log data from the server hard disk to the memory, accelerating the log processing efficiency and improving the access speed of users. And because only the commonly used hot spot log data is put into the memory of the server, the pressure on the memory of the server is not large.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
FIG. 1 is a flowchart illustrating a method for dynamically switching between hot and cold logs of a system according to an embodiment of the present invention;
FIG. 2 is a detailed step diagram of the dynamic switching method of the system hot and cold log in FIG. 1;
FIG. 3 is a functional block diagram of a dynamic switching apparatus for system hot and cold logs according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device in an embodiment of the present invention.
Detailed Description
In order to better understand the technical solution, the technical solution will be described in detail with reference to the drawings and the specific embodiments.
First, it is stated that the term "and/or" appearing herein is merely one type of associative relationship that describes an associated object, meaning that three types of relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
In a first aspect, referring to fig. 1, a method for dynamically switching a system hot and cold log according to an embodiment of the present invention is applied to a service system. Specifically, the service system includes a plurality of server nodes, and the dynamic switching method for the system cold and hot logs provided by the embodiment of the present invention includes the following steps S101 to S103:
step S101: entering a first stage, storing first log data generated by the business system in the first stage into a server hard disk of the business system, and collecting historical operation behavior data of a user group on the business system.
Specifically, at each server node of the service system, operation behavior data of the user on the service system is generated. Specifically, the first stage is an initial operation stage of the service system, and since the service system does not generate enough operation behavior data yet in the initial operation stage, the operation behavior data cannot be used to analyze the first log data, and therefore, the hot log data and the cold log data in the first log data cannot be distinguished in the first stage, so that all the first log data generated in the first stage are stored in the hard disk of the server, and occupation of erroneous or inaccurate hot log data on the memory of the server can be avoided, so that the memory resource of the server is reasonably utilized.
In the first stage, historical operation behavior data of the user group on the service system is collected, so that in each period of the second stage, the cold log data and the hot log data generated in the current period of the second stage are distinguished.
Specifically, event tracking can be performed on the operation behavior of the service system by the user group through the front-end page buried point, so as to collect the historical operation behavior data of the service system by the user group in the first stage. Wherein the historical operating behavior data may include: button options clicked by the user on the front page, viewed data content, and so on.
In the first stage, the operation behavior data of the user group to the service system are continuously collected and stored in the database of the service system, so that the operation behavior data are continuously accumulated until the first stage is finished, the historical operation behavior data obtained through the continuous accumulation of the first stage are read from the database, the first log data generated in the first stage can be analyzed at the background as big data, a cold and hot log analysis structure of the first log data is obtained, and a back-end data interface is generated by java according to the cold and hot log data analysis result of the first log data.
Specifically, the front-end BI system reads first log data into a server memory from a server hard disk of a service system; analyzing the first log data read into the server memory based on the historical operation behavior data generated in the first stage, thereby generating a back-end data interface; and calling a front-end BI (behavior identification) system through the generated back-end data interface, and burying points through a front-end page of the front-end BI system to continuously collect operation behavior data of the user group on the service system so as to update the historical operation behavior data.
Therefore, the front-end BI system is called through the back-end data interface, and the operation behavior data of the user group to the service system is collected through the front-end page embedded point of the front-end BI system in the second stage, so that the historical operation behavior data of the user group to the service system is dynamically updated.
After the first phase is finished, step S102 is executed, namely, the second phase is entered, and historical operation behavior data is periodically updated on the basis of the operation behaviors of the business system in the second phase by the user group.
In a specific implementation process, when to enter the second stage, the first stage may be triggered to end and enter the second stage when the data amount of the collected historical operation behavior data meets a preset number threshold, or the duration of the first stage reaches a preset time threshold.
Step S103: in the second stage, the second log data generated by the service system in the ith time period is analyzed by using the updated operation behavior data in the ith time period to distinguish the target hot log data and the target cold log data in the ith time period; and storing the target hot log data in the ith time period into a server memory of the service system, and storing the target cold log data in the ith time period into a server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes an integer which is increased in turn from 1.
In step S102, after storing the target hot log data in the second log data generated in the ith time period into a server memory of the service system and storing the target cold log data into a server hard disk, performing ith sub-optimization processing on the backend data interface by using the target hot log data and the target cold log data in the second log data generated in the ith time period; and calling the front-end BI system based on the ith sub-optimized back-end data interface, and continuously collecting the operation behavior data of the user group to the service system by the front-end BI system in the (i + 1) th time period based on the front-end page embedded point.
After the 1 st period of the second stage is entered, distinguishing target hot log data and target cold log data from second log data generated in the 1 st period, and storing a server memory and a server hard disk correspondingly; performing 1 st sub-optimization on the rear-end data interface generated at the first stage by using target hot log data and target cold log data in second log data generated at the 1 st time period to obtain a 1 st sub-optimized rear-end data interface; after the second stage is started in the 2 nd time period, distinguishing target hot log data and target cold log data from second log data generated in the 2 nd time period, and storing a server memory and a server hard disk correspondingly; and performing 2-th sub-optimization on the 1 st sub-optimized rear-end data interface by using target hot log data and target cold log data in second log data generated in the 2 nd time period to obtain the 2 nd sub-optimized rear-end data interface, and proceeding according to the rule to dynamically perform cold and hot distinguishing on the logs in real time, and directly putting the hot log data into the server content without optimizing the rear-end data interface.
In some embodiments, the target hot log data and the target cold log data in the second log data may be distinguished by: and performing counting weighted calculation analysis on the second log data by using the updated historical operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data. The method can be realized by the following implementation processes: adjusting the weight value of the corresponding log in the second log data by using the updated operation behavior data in the ith time period; according to the weight value of each log in the second log data, log sorting is carried out on the second log data; and distinguishing hot log data and cold log data in the second log data according to the log sorting result.
For example, each time a user clicks a button option on a front-end page of a service system, views data content, and other operation behaviors, the adjustment of the weight value +1 of the corresponding log is triggered; and if the log returns within 2 seconds after clicking, the weight value of the corresponding log in the second log data is-1. For example, the operation behavior of the user joining the shopping cart triggers the adjustment of the weight value +1 of the corresponding log.
Therefore, the operation behavior of the user group on the front end page of the business system changes the weight of each piece of log data in the target log data generated in the ith time period. The time period may be divided according to 1 minute intervals, i.e.: and analyzing second log data generated in the current 1 minute based on the updated operation behavior data in the current 1 minute to distinguish target log data and target cold log data generated in the current 1 minute.
Specifically, according to the log sorting result and a preset thermal log proportion "x%", x% pieces of log data which are ranked at the top in the first log data are determined as target thermal log data and stored in a server memory; and determining other logs in the first log data as target cold log data and storing the target cold log data into the server hard disk.
Furthermore, in order to avoid the transitional consumption of the log storage in the memory to the memory resource of the server, other processing processes of the memory are influenced; in the second stage, if the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, stopping storing the target hot log data into the server memory; and storing the residual log in the target hot log data and the target cold log data in a server hard disk.
Specifically, in a server memory of a server in the service system, if the occupancy rate of the thermal log data to the server memory reaches 75%, the storage of the target thermal log data into the server memory of the service system is stopped.
Different from the above embodiment, in order to avoid the transient consumption of the memory resources of the server caused by the log storage into the memory, other processing processes of the memory are affected; in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches the preset alarm threshold, the proportion of the target hot log in the second log data is reduced to reduce the data volume of the hot log data stored in the server memory, so as to maintain the occupancy rate of the hot log data in the server memory to be lower than the preset alarm threshold.
In the specific implementation process, the hot log data in the server memory and the cold log data in the server hard disk are dynamically updated based on the pre-defined dynamic switching condition.
Specifically, the application scenarios of the service systems are different, and the customized dynamic switching conditions may be different. Specifically, for the commodity transaction system, the dynamic switching condition may be that the hot log data in the memory of the server and the cold log data in the hard disk of the server are dynamically updated based on the change of the operation behavior data of the commodity transaction system by the specific user group.
Some service systems, such as weather applications, may dynamically update hot log data in a server memory and cold log data in a server hard disk by using a seasonal change or a weather change as a dynamic switching condition, that is, distinguish cold log data from cold log data by using weather change data or seasonal change data to dynamically update hot log data in the server memory and cold log data in the server hard disk.
The embodiment of the invention distinguishes the hot log data and the cold log data from the generated log data by utilizing the operation behavior data of the user group to the service system, only stores the hot log data in the content area of the service system, and stores the cold log data in the server hard disk of the service system so as to dynamically maintain the dynamic update of the hot log data in the server memory and the cold log data in the server hard disk, thereby maintaining the high-efficiency utilization of server resources, analyzing the hot log data in the server memory directly after obtaining the hot log data, saving the time consumption for reading the hot log data from the server hard disk, accelerating the log processing efficiency and improving the access speed of users. And because only the commonly used hot spot log data is put into the memory of the server, the pressure on the memory of the server is not large.
In the following, referring to fig. 2, the technical solution provided by the embodiment of the present invention is illustrated by taking an example that a service system includes two server nodes (server node 01: server01, server node 02: server 02) to understand the present invention:
the generated first log data is executed for each server node (server 01, server 02) to step 01: storing the data into a server hard disk, and then entering step 04;
executing step 04: reading the first log data from the server hard disk to a server memory, performing log analysis on the server memory, generating a back-end data interface through java, and then entering step 05;
step 05: the back-end data interface calls a front-end BI system, the front-end BI system obtains user operation behavior data through front-end page embedded points, and then, the step 06 is carried out;
step 06: analyzing the generated second log data log based on a counting weighting algorithm by using user operation behavior data obtained by front-end buried point data, distinguishing hot day data and cold log data, executing a step 02 aiming at the cold log data, and executing a step 01 aiming at the hot log data;
step 02: storing the data into a server memory;
and 04, optimizing the back-end data interface based on the hot log data, reading cold log data from a hard disk of the server to a memory of the server to optimize the back-end data interface, giving the optimized back-end data interface, and returning to the step 05.
In a second aspect, based on the same inventive concept, an embodiment of the present invention provides a dynamic switching apparatus for a system cold and hot log, which is shown in fig. 3 and includes:
a first log processing unit 301, configured to enter a first stage, store first log data generated by a service system in the first stage into a server hard disk of the service system, and collect historical operation behavior data of a user group on the service system;
a second log processing unit 302, configured to enter a second stage, and periodically update the historical operation behavior data based on an operation behavior of the service system in the second stage based on a user group; in the second stage, analyzing second log data generated by the service system in the ith time period by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into the server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes integers which are sequentially increased from 1.
Under some embodiments, the first log processing unit 301 includes:
a reading-in subunit, configured to read the first log data from the server hard disk into the server memory;
the data analysis subunit is used for carrying out big data analysis on the first log data read into the server memory based on the historical operation behavior data generated in the first stage to generate a back-end data interface;
and the data updating subunit is used for calling a front-end page buried point of the front-end BI system through the back-end data interface so as to update the historical operation behavior data.
In some embodiments, the dynamic switching device for system hot and cold logs according to an embodiment of the present invention further includes:
the interface optimization unit is used for performing ith sub-optimization processing on the back-end data interface by using target hot log data and target cold log data in second log data generated in the ith time period;
and the front-back joint debugging unit is used for calling a front-end page embedded point of the front-end BI system based on the ith sub-optimized back-end data interface so as to continuously collect the operation behavior data of the user group on the service system in the (i + 1) th time period.
In some embodiments, the dynamic switching device for system hot and cold logs according to an embodiment of the present invention further includes:
and the triggering unit is used for triggering to finish the first stage and enter the second stage when the data volume of the historical operation behavior data meets a preset number threshold or the duration of the first stage reaches a preset time threshold.
Under some embodiments, the second log processing unit 302 includes:
and the weighted analysis subunit is used for performing counting weighted calculation analysis on the second log data generated in the ith time period by using the updated historical operation behavior data in the ith time period so as to distinguish target hot log data and target cold log data in the second log data.
In some embodiments, the weighted analysis subunit includes:
the weight calculation module is used for adjusting the weight value of the corresponding log in the second log data by using the updated operation behavior data in the ith time period;
the log sorting module is used for sorting the logs of the second log data according to the weight value of each log in the second log data;
and the log distinguishing module is used for distinguishing hot log data and cold log data in the second log data according to the log sorting result.
In some embodiments, the log differentiation module is further configured to:
in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, reducing the proportion of the target hot log data in the second log data so as to maintain the occupancy rate of the hot log data in the server memory to the server memory lower than the preset alarm threshold.
In some embodiments, the second log processing unit 302 is configured to:
a memory access stopping unit, configured to, in the second stage, stop storing the target thermal log data into the server memory when an occupancy rate of the thermal log data in the server memory to the server memory reaches a preset alarm threshold; and storing the residual logs in the target hot log data and the target cold log data into the server hard disk.
In some embodiments, the second log processing unit 302 is specifically configured to: and dynamically updating the hot log data in the server memory and the cold log data in the server hard disk based on a pre-defined dynamic switching condition.
The dynamic switching device for system cold and hot logs described in the embodiments of the present invention is a device for implementing the dynamic switching method for system cold and hot logs, and further implementation details of the dynamic switching device for system cold and hot logs may refer to the dynamic switching method for system cold and hot logs described above, and for brevity of the description, details are not repeated here.
In a third aspect, based on the same inventive concept, an embodiment of the present invention provides a computer device, which is shown in fig. 4 and includes one or more processors 402 and one or more memories 401, where at least one program code is stored in the one or more memories 401, and the at least one program code is loaded and executed by the one or more processors 402 to implement the method for dynamically switching the system hot and cold logs according to any embodiment of the first aspect.
Where in fig. 4 a bus architecture (represented by bus 400), bus 400 may include any number of interconnected buses and bridges, bus 400 linking together various circuits including one or more processors, represented by processor 402, and memory, represented by memory 401. The bus 400 may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface 405 provides an interface between the bus 400 and the receiver 403 and transmitter 404. The receiver 403 and the transmitter 404 may be the same element, i.e., a transceiver, providing a means for communicating with various other apparatus over a transmission medium. The processor 402 is responsible for managing the bus 400 and general processing, and the memory 401 may be used for storing data used by the processor 402 in performing operations.
In a fourth aspect, based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for dynamically switching the system hot and cold logs according to any one of the embodiments of the first aspect.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A method for dynamically switching a system cold and hot log is characterized by comprising the following steps:
entering a first stage, storing first log data generated by a service system in the first stage into a server hard disk of the service system, and collecting historical operation behavior data of a user group on the service system;
entering a second stage, and periodically updating the historical operation behavior data of the operation behavior of the service system based on a user group in the second stage;
in the second stage, analyzing second log data generated by the service system in the ith time period by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into the server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes integers which are sequentially increased from 1.
2. The method of claim 1, wherein the periodically updating the historical operational behavior data for the operational behavior of the business system in the second phase based on the user group comprises:
reading the first log data from the server hard disk into the server memory;
performing big data analysis on the first log data read into the server memory based on the historical operation behavior data generated in the first stage to generate a back-end data interface;
and calling a front-end page buried point of a front-end BI system through the back-end data interface so as to update the historical operation behavior data.
3. The method of claim 2, wherein after storing the target hot log data into a server memory of the business system and storing the target cold log data into the server hard disk, further comprising:
performing ith sub-optimization processing on the back-end data interface by using target hot log data and target cold log data in second log data generated in the ith time period;
and calling a front-end page embedded point of the calling front-end BI system based on the ith sub-optimized back-end data interface to continuously collect the operation behavior data of the user group to the service system in the (i + 1) th time period.
4. The method of claim 1, further comprising:
and when the data volume of the historical operation behavior data meets a preset number threshold or the duration of the first stage reaches a preset time threshold, triggering to end the first stage and entering the second stage.
5. The method of claim 1, wherein the analyzing second log data generated by the business system in the ith time period by using the operation behavior data updated in the ith time period to distinguish target hot log data and target cold log data in the second log data comprises:
and performing counting weighted calculation analysis on second log data generated in the ith time period by using the updated historical operation behavior data in the ith time period so as to distinguish target hot log data and target cold log data in the second log data.
6. The method as claimed in claim 5, wherein said performing count weighted calculation analysis on said second log data by using said updated operation behavior data of said ith time period to distinguish target hot log data and target cold log data in said second log data comprises:
adjusting the weight value of the corresponding log in the second log data by using the updated operation behavior data in the ith time period;
according to the weight value of each log in the second log data, log sorting is carried out on the second log data;
and distinguishing hot log data and cold log data in the second log data according to the log sorting result.
7. The method of claim 6, wherein the distinguishing hot log data and cold log data in the second log data according to log sorting results comprises:
in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, reducing the proportion of the target hot log data in the second log data so as to maintain the occupancy rate of the hot log data in the server memory to the server memory lower than the preset alarm threshold.
8. The method of any of claims 1-6, further comprising, after distinguishing target hot log data from target cold log data in the second log data:
in the second stage, when the occupancy rate of the hot log data in the server memory to the server memory reaches a preset alarm threshold, stopping storing the target hot log data into the server memory; and storing the residual logs in the target hot log data and the target cold log data into the server hard disk.
9. A dynamic switching device for system cold and hot logs is characterized by comprising:
the first log processing unit is used for entering a first stage, storing first log data generated by the business system in the first stage into a server hard disk of the business system, and collecting historical operation behavior data of a user group on the business system;
the second log processing unit is used for entering a second stage and periodically updating the historical operation behavior data on the basis of the operation behavior of the service system in the second stage of the user group; in the second stage, analyzing second log data generated by the service system in the ith time period by using the updated operation behavior data in the ith time period to distinguish target hot log data and target cold log data in the second log data; and storing the target hot log data into a server memory of the service system, and storing the target cold log data into the server hard disk so as to dynamically update the hot log data in the server memory and the cold log data in the server hard disk, wherein i takes integers which are sequentially increased from 1.
10. A computer device comprising one or more processors and one or more memories having at least one program code stored therein, the at least one program code being loaded and executed by the one or more processors to implement the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111408191.1A CN113835986B (en) | 2021-11-25 | 2021-11-25 | Dynamic switching method and device for system cold and hot logs and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111408191.1A CN113835986B (en) | 2021-11-25 | 2021-11-25 | Dynamic switching method and device for system cold and hot logs and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113835986A true CN113835986A (en) | 2021-12-24 |
| CN113835986B CN113835986B (en) | 2022-04-08 |
Family
ID=78971641
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111408191.1A Active CN113835986B (en) | 2021-11-25 | 2021-11-25 | Dynamic switching method and device for system cold and hot logs and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113835986B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010272167A (en) * | 2009-05-21 | 2010-12-02 | Toshiba Alpine Automotive Technology Corp | Optical disk play-back device, and optical disk play-back method |
| US8909887B1 (en) * | 2012-09-25 | 2014-12-09 | Emc Corporation | Selective defragmentation based on IO hot spots |
| CN108197289A (en) * | 2018-01-18 | 2018-06-22 | 吉浦斯信息咨询（深圳）有限公司 | A kind of data store organisation, data store query method, terminal and medium |
| CN108268485A (en) * | 2016-12-30 | 2018-07-10 | 亿阳安全技术有限公司 | A kind of daily record real-time analysis method and system |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
| CN110334036A (en) * | 2019-06-28 | 2019-10-15 | 京东数字科技控股有限公司 | A kind of method and apparatus for realizing data cached scheduling |
| CN110858210A (en) * | 2018-08-17 | 2020-03-03 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN111309732A (en) * | 2020-02-19 | 2020-06-19 | 杭州朗和科技有限公司 | Data processing method, device, medium and computing equipment |
| CN113064930A (en) * | 2020-12-29 | 2021-07-02 | 中国移动通信集团贵州有限公司 | Cold and hot data identification method and device of data warehouse and electronic equipment |
-
2021
- 2021-11-25 CN CN202111408191.1A patent/CN113835986B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010272167A (en) * | 2009-05-21 | 2010-12-02 | Toshiba Alpine Automotive Technology Corp | Optical disk play-back device, and optical disk play-back method |
| US8909887B1 (en) * | 2012-09-25 | 2014-12-09 | Emc Corporation | Selective defragmentation based on IO hot spots |
| CN108268485A (en) * | 2016-12-30 | 2018-07-10 | 亿阳安全技术有限公司 | A kind of daily record real-time analysis method and system |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
| CN108197289A (en) * | 2018-01-18 | 2018-06-22 | 吉浦斯信息咨询（深圳）有限公司 | A kind of data store organisation, data store query method, terminal and medium |
| CN110858210A (en) * | 2018-08-17 | 2020-03-03 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN110334036A (en) * | 2019-06-28 | 2019-10-15 | 京东数字科技控股有限公司 | A kind of method and apparatus for realizing data cached scheduling |
| CN111309732A (en) * | 2020-02-19 | 2020-06-19 | 杭州朗和科技有限公司 | Data processing method, device, medium and computing equipment |
| CN113064930A (en) * | 2020-12-29 | 2021-07-02 | 中国移动通信集团贵州有限公司 | Cold and hot data identification method and device of data warehouse and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| ZHENYUAN SUN等: "RALD: Reliable Transactional Software Controller for Redundancy Array of Log Structured Disks", 《 2015 IEEE 17TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING AND COMMUNICATIONS》 * |
| 梁俊杰等: "以固态硬盘为缓存的存储技术研究", 《微电子学与计算机》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113835986B (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102193929B (en) | Method and equipment for searching by using word information entropy | |
| US9832280B2 (en) | User profile configuring method and device | |
| CN111459993B (en) | Configuration updating method, device, equipment and storage medium based on behavior analysis | |
| CN106713396B (en) | Server scheduling method and system | |
| CN111881221A (en) | Method, device and equipment for customer portrait in logistics service | |
| CN110134738B (en) | Distributed storage system resource estimation method and device | |
| CN110675194A (en) | Funnel analysis method, device, equipment and readable medium | |
| CN102902775A (en) | Internet real-time computing method and internet real-time computing system | |
| CN106202092A (en) | The method and system that data process | |
| CN112328688B (en) | Data storage method, device, computer equipment and storage medium | |
| CN111382182A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112749863A (en) | Keyword price adjusting method and device and electronic equipment | |
| CN116860789A (en) | Data distribution optimization method and distributed database system | |
| CN113835986B (en) | Dynamic switching method and device for system cold and hot logs and computer equipment | |
| CN110348928A (en) | Information-pushing method, device and computer readable storage medium | |
| CN107203545B (en) | Data processing method and device | |
| CN113656702B (en) | User behavior prediction method and device | |
| CN114185938B (en) | Project traceability analysis method and system based on digital finance and big data traceability | |
| CN115713395A (en) | Flink-based user wind control management method, device and equipment | |
| US11200547B2 (en) | Payment collection control method and device, server, and computer-readable storage medium | |
| CN115269519A (en) | Log detection method and device and electronic equipment | |
| CN114035906A (en) | Virtual machine migration method and device, electronic equipment and storage medium | |
| CN111027823A (en) | Browser-based work order automatic forwarding method and device | |
| CN113094589B (en) | Intelligent service recommendation method and device | |
| CN113076348B (en) | Policy information management method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |