CN113761015A - Log processing method, device and system and storage medium - Google Patents
Log processing method, device and system and storage medium Download PDFInfo
- Publication number
- CN113761015A CN113761015A CN202011141527.8A CN202011141527A CN113761015A CN 113761015 A CN113761015 A CN 113761015A CN 202011141527 A CN202011141527 A CN 202011141527A CN 113761015 A CN113761015 A CN 113761015A
- Authority
- CN
- China
- Prior art keywords
- log
- information
- log information
- target
- service system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log processing method, a device, a system and a storage medium. The method is applied to a log processing system, the log processing system can be connected with a plurality of connected service systems, log information of each connected service system is stored in a log database of the log processing system, and the log information comprises service system source information, and the method comprises the following steps: when a positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition; and screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information. According to the technical scheme of the embodiment of the invention, the target log information related to the query condition is queried through the unique entrance provided by the log processing system, so that a user can quickly locate the connected service system with abnormality according to the service system source information in the target log information.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer application, in particular to a log processing method, device and system and a storage medium.
Background
In an application scenario where interaction of each service system is frequent, various log information is generally required to be recorded, and the log information can provide a basis for subsequent abnormal positioning.
In the process of implementing the invention, the inventor finds that the following technical problems exist in the prior art: when each service system is abnormal in the interaction process, the system responsible person of each service system needs to perform log query in each service system to locate the service system associated with the abnormality, and the abnormality locating efficiency is low.
Disclosure of Invention
The embodiment of the invention provides a log processing method, a log processing device, a log processing system and a log processing storage medium, and solves the problem of low abnormal positioning efficiency.
In a first aspect, an embodiment of the present invention provides a log processing method, where the method is applied to a log processing system, the log processing system may be connected to multiple connected service systems, a log database of the log processing system stores log information of each connected service system, and the log information includes service system source information, and the method may include:
when a positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition;
and screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information.
In a second aspect, an embodiment of the present invention further provides a log processing apparatus, where the apparatus is configured in a log processing system, the log processing system is connected to multiple connected service systems, a log database of the log processing system stores log information of each connected service system, and the log information includes service system source information, and the apparatus may include:
the log query module is used for acquiring query conditions corresponding to the positioning trigger events when the positioning trigger events of abnormal positioning are monitored, and querying in a log database according to the query conditions;
and the log display module is used for screening target log information matched with the query conditions from the log information according to the query result and displaying the target log information.
In a third aspect, an embodiment of the present invention further provides a log processing system, which may include:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method provided by any embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the log processing method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, when the log processing system acquires the query condition related to abnormal positioning, the query can be carried out in the log database according to the query condition, wherein the log database is the database in the log processing system; because the log information of each connected service system is stored in the log database, that is, the log information in the log database is from each connected service system, the target log information which is screened from each log information according to the query result and is matched with the query condition can also be from each connected service system, and in consideration of the fact that the target log information comprises the source information of the service system, after the target log information is displayed, the user can locate the connected service system with abnormality according to the displayed target log information. According to the technical scheme, the target log information related to the query condition is queried through the unique entry provided by the log processing system, so that a user can locate the connected service system with the abnormality according to the service system source information in the target log information, and therefore the effect of quickly locating the connected service system with the abnormality is achieved.
Drawings
Fig. 1 is a flowchart of a log processing method according to a first embodiment of the present invention;
fig. 2 is a schematic diagram of target log information in a log processing method according to a first embodiment of the present invention;
FIG. 3 is a diagram illustrating query results in a log processing method according to a first embodiment of the present invention;
FIG. 4 is a flowchart of a log processing method according to a second embodiment of the present invention;
fig. 5 is a schematic diagram of a data flow in a log processing method according to a second embodiment of the present invention;
fig. 6 is a flowchart of a log processing method in the third embodiment of the present invention;
fig. 7 is a block diagram showing a configuration of a log processing apparatus according to a fourth embodiment of the present invention;
fig. 8 is a schematic structural diagram of a log processing system in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before the embodiment of the present invention is described, an application scenario of the embodiment of the present invention is exemplarily described: in the prior art, if a business system with an abnormality is located from the interactive business systems, a system administrator of each business system needs to query log information of each business system based on a query condition related to the abnormality, such as querying based on an order number in the log information of an order system, querying based on a delivery number in the log information of a delivery system, and the like; and then, the system responsible person can determine whether the business system in charge of the system is abnormal or not according to the query result. It should be noted that, even if the same query condition is applicable to at least two service systems, for example, the same order number is applicable to at least two service systems, it is necessary to perform query in at least two service systems according to the same query condition, and obviously, such abnormal location efficiency is low. In addition, the business system may be a system for implementing a certain business function, such as an order system, a distribution system, etc.; the system responsible person can be operation and maintenance personnel, development personnel, testing personnel and the like of the business system; the order number and the delivery number may be one of the service numbers, that is, the service number may have different meanings in different service systems.
Correspondingly, the log processing method provided by the embodiment of the invention can be applied to a log processing system, the log processing system is in communication connection with a plurality of connected service systems, the connected service systems can be service systems which are connected with the log processing system and used for realizing certain service functions, the log processing system comprises a log database, and log information of each connected service system is stored in the log database. In this way, the user only needs to query in the log processing system according to the query condition, specifically, query in each log information of the log database, so as to query the target log information matching with the query condition from the log information, where the target log information may be from each connected service system, and the user may be an operator, a developer, a tester, or the like of the log processing system. It should be noted that each piece of log information includes service system source information, and the service system source information may indicate from which connected service system the log information originates. Therefore, after the user screens the abnormal target log information from the target log information, the abnormal connected service system can be directly positioned according to the service system source information in the abnormal target log information. In other words, during abnormal positioning, the user does not need to contact the system responsible person of each connected service system one by one to inquire in each connected service system, and the user can inquire the connected service system with abnormality through the unique entry of the log processing system, so that the efficiency of abnormal positioning is improved.
Example one
Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention. The embodiment can be applied to the case that the log processing system is used as a unique entry to inquire out the target log information matched with the inquiry condition. The method can be executed by a log processing device provided by the embodiment of the invention, the device can be realized by software and/or hardware, and the device can be integrated on a log processing system.
Referring to fig. 1, the method of the embodiment of the present invention specifically includes the following steps:
s110, when a positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition.
The location triggering event may be an event triggered by a user for performing anomaly location, the anomaly may be an anomaly found by the user and/or a client, and the client may be a user of a connected business system, such as a consignor of a logistics system, a purchaser of an order system, and the like. For example, when acquiring a query condition input by a user through a front-end page, i.e., when a positioning trigger event for detecting abnormal positioning is detected, the log processing system may be a page for inputting the query condition.
Further, a query condition corresponding to the location trigger event is obtained, where the query condition may be a keyword related to the anomaly, such as a time period, a service order number, and the like, and the service order number may be an order number, a delivery order number, and the like. Therefore, as the log database of the log processing system stores the log information of each connected service system connected with the log processing system, the log database can be queried according to the query condition.
And S120, screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information.
When the query is performed in the log database according to the query condition, target log information matched with the query condition can be screened from all log information, wherein the target log information can be log information directly related to the abnormality, such log information can include abnormal information, and the abnormal information can show the specific condition of the abnormality; or, the target log information may be log information indirectly associated with the anomaly, for example, if a service chain includes a plurality of service links, the target log information may be log information on the service chain in which the anomaly occurs, but not log information on the link in which the anomaly occurs on the service chain, and such log information does not include the anomaly information; and so on. Furthermore, the target log information is displayed, and each piece of target log information comprises the service system source information, which means that a user can determine which connected service system the entry label log information originates from according to the service system source information in the displayed target log information, so that the connected service system with the abnormality can be quickly located according to each piece of target log information.
On this basis, optionally, some target log information includes abnormal information, and some target log information does not include abnormal information, for example, four links of pre-sorting check, customer information check, picking site information check and bill inventory check in the following single service chain are taken as examples, assuming that an interface of the pre-sorting check is requested to be called, a return result of the interface is normal, that is, the log information corresponding to the calling process of the request does not include abnormal information; the interface for checking the client information is requested to be called, and the return result of the interface is abnormal, namely the log information corresponding to the calling process of the request contains abnormal information, such as error stack information. Then, when the target log information is displayed, the target log information including the abnormal information and the target log information not including the abnormal information may be distinguished based on different display effects, for example, the target log information including the abnormal information is marked with a red background color and the target log information not including the abnormal information is marked with a green background color, which is helpful for a user to quickly locate the target log information having the abnormal information from a large amount of target log information, and further quickly locate the connected service system having the abnormality according to the service system source information in the target log information having the abnormal information. Optionally, each piece of target log information may further include a service system contact, where the service system contact may be a system principal of a connected service system corresponding to the service system source information. Therefore, after the user locates the connected service system with the abnormality, the user can inform the service system contact of the connected service system to perform corresponding adjustment optimization and the like on the connected service system. Illustratively, the display effect of the target log information is as shown in fig. 2, and the target log information containing the abnormality information is marked with a gray background color.
According to the technical scheme of the embodiment of the invention, when the log processing system acquires the query condition related to abnormal positioning, the query can be carried out in the log database according to the query condition, wherein the log database is the database in the log processing system; because the log information of each connected service system is stored in the log database, that is, the log information in the log database is from each connected service system, the target log information which is screened from each log information according to the query result and is matched with the query condition can also be from each connected service system, and in consideration of the fact that the target log information comprises the source information of the service system, after the target log information is displayed, the user can locate the connected service system with abnormality according to the displayed target log information. According to the technical scheme, the target log information related to the query condition is queried through the unique entry provided by the log processing system, so that a user can locate the connected service system with the abnormality according to the service system source information in the target log information, and therefore the effect of quickly locating the connected service system with the abnormality is achieved.
On this basis, an optional technical solution is that the number of the target log information is at least two, and the displaying of the target log information may specifically include: acquiring a request single number associated with each target log information, wherein the request single number is a single number generated when an interface in a connected service system is requested to be called; and grouping and displaying the target log information according to the request list number. An interface is an interface that can implement a certain function, and may also be understood as a service, a method, a function, and the like. When the interface is called, a corresponding request single number is generated, the request single number is mainly used for connecting links called by the interface request in series, and the format of the request single number can be time character string + multi-bit random number. Therefore, when at least two pieces of target log information are displayed, the request form numbers associated with the target log information can be acquired, and the target log information associated with the same request form number can be displayed as a group.
It should be noted that, an interface may generate a request single number when being requested to be called, but the interface may include multiple sub-interfaces, and the request call of the interface means that multiple sub-interfaces therein are sequentially called, and the calling process of these sub-interfaces does not regenerate a new request single number, which is the above meaning that a request single number may connect links called by the interface request this time in series. However, the calling process of these subinterfaces still generates log information, which means that the number of target log information associated with the same request ticket number may be one, two or more.
In order to better understand the specific implementation process of the above steps, the following describes an exemplary log processing method according to this embodiment with reference to a specific example. Illustratively, as shown in fig. 3, a front-end page of an inquiring machine in the log processing system provides an input box of a service order number or a request order number, which can be selected through a pull-down menu, taking the service order number of a certain order service as an example, after the service order number input in the input box by a user is obtained, inquiring is performed in an Hbase cluster according to the service order number, wherein the Hbase cluster is a cluster formed by a plurality of hbases, and log information of each connected service system is stored in the cluster. Furthermore, target log information related to the ordering service can be determined according to the query result, and the target log information can be originated from different connected service systems. It should be noted that, a plurality of request single numbers may be involved under one service single number, and the following single service and the modification service respectively correspond to two request single numbers under one service single number, so that each target log information under the same request single number can be displayed under the same tab, so as to display the complete link called by the request. Of course, one, two, or more interfaces may be called by a request in a link of a service chain, that is, log information in a link may be associated with one, two, or more request numbers, which is not specifically limited herein.
Example two
Fig. 4 is a flowchart of a log processing method provided in the second embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the log processing system may include an information sending client, and the connected service system may store the log information in the log database by the following steps: when a call trigger event called by an interface is monitored, acquiring log information corresponding to the call trigger event, and storing the log information in a context; when the interface calling is finished, the log information in the context is sent to the information sending client, so that the log information is stored in the log database through the information sending client, and the log information in the context is deleted; the log information may include parameter information, or parameter information and exception information, where the parameter information includes a parameter entering parameter and a parameter exiting parameter. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
Referring to fig. 4, the method of this embodiment may specifically include the following steps:
s210, when a call trigger event called by an interface is monitored, obtaining log information corresponding to the call trigger event, and storing the log information in a context, wherein the log information comprises parameter information or the parameter information and abnormal information, and the parameter information comprises a parameter entering parameter and a parameter exiting parameter.
The call trigger event may be an event triggered when a certain interface in a certain connected service system is called, for example, after the connected service system related to the order placing service monitors that an order placing button is clicked, it may call an interface for pre-sorting verification, and at this time, the connected service system may monitor the call trigger event called by the interface. Further, when the interface is called, the connected service system may obtain log information generated in the calling process of the interface, and store the log information in the context. It should be noted that the interface call may involve an entry parameter and an exit parameter, and these parameter information are part of log information. In addition, when the interface call is abnormal, abnormal information may also occur, and the abnormal information may also be a part of the log information.
S220, when the interface calling is finished, the log information in the context is sent to the information sending client, so that the log information is stored in the log database through the information sending client, and the log information in the context is deleted.
When the interface calling is finished, all log information related to the interface calling is stored in the context, at the moment, the log information stored in the context can be sent to the information sending client, so that the log information is stored in the log database through the information sending client, and the log information in the context can be deleted after the log information is sent out.
In practical applications, for the log writing logic executed by the connected service system, for example, the log processing system may encapsulate the processing logic related to the log writing into a jar packet, so that the connected service system executes a corresponding log writing function by introducing the jar packet, and the log writing function may be implemented by way of annotation, API call, and the like, and the specific implementation logics of the two are substantially consistent. Taking an annotation manner as an example, when monitoring a call trigger event, the connected service system may obtain corresponding entry parameters and exit parameters based on a tangent plane technology, and of course, may also obtain exception information when there is a call exception, and store these information in a context by a thread local technology, where they are all part of log information. It should be noted that a certain interface may implement a corresponding function based on sequential calling of multiple internal sub-interfaces, and information such as an entry parameter, an exit parameter, and exception information of a calling process of the sub-interfaces may also be stored in a context, and they may be stored in a stack manner. At this time, after the interface call is ended, the information in the stack is sequentially returned layer by layer, and the returned information can be sent to a consuming terminal through the MQ client, and then consumed into the Hbase cluster through the consuming terminal. Of course, the information in the context may be deleted after the information is sent out.
And S230, when the positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition.
S240, screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information.
On this basis, as shown in fig. 5, for example, the log information of each connected service system (e.g., star system, man-mail order system, and order center system) may be sent to the Hbase cluster through the MQ cluster, and the query engine may query the Hbase cluster for the target log information matching the query condition.
According to the technical scheme of the embodiment of the invention, when the connected service system monitors the call trigger event called by the interface, the log information corresponding to the call trigger event can be stored in the context; when the interface call is finished, the log information stored in the context can be sent to the information sending client, so that the log information is stored in the log database through the information sending client, and the log information in the context is deleted after the log information is sent, thereby realizing the effect of writing the log information in the connected service system into the log processing system.
An optional technical solution, in practical application, when the log writing function is implemented based on an annotation manner, situations may be encountered in which some parameters are swallowed due to an exception of a connected service system, for example, the displayed exception prompt is not displayed, the output parameter of the output parameter is not output, and the like, and these parameters may be referred to as exception parameters, and at this time, the annotation in the annotation manner cannot obtain the exception parameter. Therefore, the log processing method may further include: when an abnormal triggering event with abnormal parameters is monitored, acquiring abnormal parameters and a parameter setting interface corresponding to the abnormal triggering event, and calling the parameter setting interface to set the abnormal parameters; and determining abnormal information corresponding to the abnormal parameters, and storing the abnormal parameters and the abnormal information in the context. That is, the log information in the context is operated through the parameter setting interface arranged inside the interface, so as to retrieve or reset the swallowed exception parameters, and further store the exception information corresponding to the exception parameters and the exception parameters in the context, optionally, the parameter setting interface may be an API interface.
EXAMPLE III
Fig. 6 is a flowchart of a log processing method provided in the third embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the log processing method may further include: when a connection triggering event of a service system connection is monitored, acquiring a to-be-connected log type of a to-be-connected service system corresponding to the connection triggering event, wherein the to-be-connected log type can comprise a service type, an operation type and service system source information; and if the type of the log to be connected is not consistent with the type of the connected log of the connected service system, allowing the service system to be connected to the log processing system. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
Referring to fig. 6, the method of this embodiment may specifically include the following steps:
s310, when a connection triggering event of the service system connection is monitored, acquiring a to-be-connected log type of the to-be-connected service system corresponding to the connection triggering event, wherein the to-be-connected log type comprises a service type, an operation type and service system source information.
The connection triggering event monitored by the log processing system may be an event triggered after a user inputs a to-be-connected log type for connection through a front-end page, where the to-be-connected log type includes a type of log information of a to-be-connected service system set by the user according to an actual situation, and the to-be-connected service system is a service system that has not been connected to the log processing system. Therefore, when a connection triggering event of the service system connection is monitored, a to-be-connected log type of the to-be-connected service system corresponding to the connection triggering event may be obtained, where the to-be-connected log type may include a service type, an operation type, and service system source information, and may also include a service system contact person, and the like, and is not specifically limited herein. In other words, before the service system to be connected is connected to the log processing system, the user may apply for configuring the log type to be connected required by the user for the service system to be connected to the log processing system. If the service system to be connected is an order system, the source information of the service system can be the order system, the service type can be the order placing service, and the operation type can be a link in the order placing service, such as pre-sorting verification, customer information verification, picking and dispatching site information verification, order inventory verification and the like.
And S320, if the type of the log to be connected is not consistent with the type of the connected log of the connected service system, allowing the service system to be connected to the log processing system.
The connected log type may be a type of log information connected to a connected service system in the log processing system, and similar to the to-be-connected log type, the connected log type may include a service type, an operation type, and service system source information, and may also include a service system contact, and the like, which is not specifically limited herein. The connected log type of each connected service system has uniqueness, so that the log type to be connected can be compared with each connected log type, if the log type to be connected does not conflict with each connected log type, the log processing system can allow the service system to be connected to the log processing system, and at the moment, the log type to be connected of the service system to be connected can be returned to the service system to be connected. Of course, when there is a conflict, corresponding prompt information may be returned to prompt the user to modify the to-be-connected log type of the to-be-connected service system, so that the to-be-connected service system is successfully connected to the log processing system.
S330, when the positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition.
And S340, screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information.
The connected log types have uniqueness, and can also be embodied in log information of the connected service system, so that the target log information matched according to the query condition can contain service system source information, service types and operation types, which is helpful for a user to know which operation type under which service type under which connected service system the user belongs to from the target log information, thereby improving the efficiency of subsequent abnormal positioning. Particularly, the connected log type is set, so that the effect of effectively unifying the log information under the multi-dimensional distributed scene according to the source information, the service type and the operation type of the service system is realized.
According to the technical scheme of the embodiment of the invention, the log processing system compares the to-be-connected log type of the to-be-connected service system corresponding to the connection trigger event with the connected log type of the connected service system, and allows the to-be-connected service system to be connected to the log processing system under the condition that the to-be-connected log type and the connected log type of the connected service system are not in conflict, so that the effect of effective connection between the to-be-connected service system and the log processing system is realized.
According to an optional technical scheme, when a configuration trigger event of the life cycle configuration is monitored, each connected log type corresponding to the configuration trigger event is used as a life cycle; correspondingly, displaying the target log information may specifically include: determining a target life cycle associated with the target log information from each life cycle; and if the missing type which belongs to the target life cycle and is not contained in each connected log type of the target log information exists, displaying the target log information and the missing type.
The configuration triggering event may be an event triggered after the user inputs a connected log type for configuration on the front-end page, and the number of the connected log types input by the user on the front-end page may be one, two, or more, that is, at least one connected log type may be used as a life cycle. If it is assumed that the connected log types input by the user include order system, order placing service and pre-sorting verification, order system, order placing service and customer information verification, order system, order placing service and purchase site information, and order system, order placing service and order inventory verification, this means that four links in the order placing service can be regarded as a life cycle.
On the basis, the target life cycle associated with the target log information can be determined from the configured life cycles, specifically, because the connected log type of each connected service system connected to the log processing system has uniqueness, the connected log type can be included in the log information of the connected service system, that is, the target log information includes the connected log type, and on the basis, because the life cycle is configured according to the connected log type, the target life cycle associated with the target log information can be determined from each life cycle according to the connected log type included in the target log information and the connected log type corresponding to each life cycle. For example, the connected log types of the log information of each entry include log type 1, log type 3 and log type 4, and a certain life cycle is configured according to log type 1, log type 2, log type 3 and log type 4, which means that the life cycle is a target life cycle associated with each target log information.
Further, comparing each connected log type in the target life cycle with each connected log type contained in each target log information, if a certain connected log type belongs to the target life cycle and is not contained in each connected log type of the target log information, such connected log type may be called as a missing type, and displaying the target log information and the missing type. Illustratively, log type 2 in the above example is the miss type. In order to better understand the application effect of the life cycle, taking 4 links in the following single business as an example, as shown in fig. 3, if log type 1 is order system, order service and pre-sort check, log type 2 is order system, order service and customer information check, log type 3 is order system, order service and picking site information, and log type 4 is order system, order service and order inventory check, when displaying, the corresponding target life cycle matched according to the connected log type of the target log information can be displayed, and since the log type 2 is a missing type, the client information check in the log type 2 can be displayed in a bold mode, which is helpful for a user to quickly confirm whether missing links exist in the target life cycle, namely whether a link is lacked in the ordering business or not, thereby further improving the efficiency of abnormal positioning.
An optional technical solution, because there may be multiple callers on a certain interface, and the caller may be a certain interface in a certain connected service system, for example, the caller of the interface a is the interface B in the service system 1 and the interface C in the service system 2, in the prior art, only the call process of the interface a is monitored for an anomaly, such distinction degree is low because it cannot distinguish the call process of the interface a for a specific connected service system (e.g., the service system 1, the service system 2) and a specific interface (e.g., the interface B, the interface C), which is not favorable for subsequent anomaly analysis by the user.
In order to solve the above problem, the log processing method may further include: acquiring a preset quantity threshold value matched with a target log type in all connected log types; and if the number of the log information which is prestored in the log database and belongs to the target log type within the first preset time range is greater than a preset number threshold, sending alarm information. The target log type can be a preset log type with an abnormal monitoring requirement, and a corresponding preset quantity threshold value is obtained, wherein the preset quantity threshold value is a preset numerical value; when the number of the log information belonging to the target log type and pre-stored in the log database within the first preset time range is greater than a preset number threshold, that is, when the calling frequency of the interface corresponding to the target log type within the first preset time range is greater than the preset number threshold, a calling abnormal condition may exist, and alarm information may be sent. That is to say, the above-mentioned scheme can be understood as an early warning mechanism for multiple operations of a normal business process, and when the normal business process is operated multiple times, a potential threat may be formed to a connected business system. For example, taking the interface call returned by the order status as an example, when the number of times of return in one minute is more than 2, alarm information is sent out.
In addition, in terms of anomaly monitoring, the log processing system may determine, based on statistical analysis, the normal and abnormal numbers for each connected log type within a second preset time range, which may be daily, hourly, per minute, and so forth. Therefore, if the abnormal quantity corresponding to a certain connected log type is increased suddenly, alarm information can be sent out. In addition, the log processing system may also perform exception presentation according to different dimensions, such as service system source information, service types, operation types, and the like, and may also provide an exception curve analysis function in different time dimensions, and the like, which is not specifically limited herein.
Example four
Fig. 7 is a block diagram of a log processing apparatus according to a fourth embodiment of the present invention, which is configured to execute the log processing method according to any of the foregoing embodiments. The device and the log processing method of the above embodiments belong to the same inventive concept, and details which are not described in detail in the embodiments of the log processing device may refer to the embodiments of the log processing method. Referring to fig. 7, the apparatus is configured in a log processing system, the log processing system is connected to a plurality of connected service systems, log information of each connected service system is stored in a log database of the log processing system, and the log information includes service system source information, and the apparatus may specifically include: a log query module 410 and a log display module 420.
The log query module 410 is configured to, when a positioning trigger event of abnormal positioning is monitored, obtain a query condition corresponding to the positioning trigger event, and query in a log database according to the query condition;
and the log display module 420 is configured to screen target log information matched with the query condition from the log information according to the query result, and display the target log information.
Optionally, on this basis, the apparatus may further include:
the log type acquisition module is used for acquiring a to-be-connected log type of a to-be-connected service system corresponding to a connection trigger event when the connection trigger event of the service system connection is monitored, wherein the to-be-connected log type comprises a service type, an operation type and service system source information;
and the service system connecting module is used for allowing the service system to be connected to the log processing system if the type of the log to be connected is inconsistent with the type of the connected log of the connected service system.
Optionally, on this basis, the apparatus may further include:
the life cycle configuration module is used for taking each connected log type corresponding to the configuration trigger event as a life cycle when the configuration trigger event of the life cycle configuration is monitored;
correspondingly, the log display module 420 may specifically include:
the target life cycle screening unit is used for determining a target life cycle associated with the target log information from each life cycle;
and the missing type display unit is used for displaying the target log information and the missing type if the missing type which belongs to the target life cycle and is not contained in each connected log type of the target log information exists.
Optionally, on this basis, the apparatus may further include:
the threshold value acquisition module is used for acquiring a preset number threshold value matched with a target log type in all connected log types; and the alarm module is used for sending alarm information if the quantity of the log information which is prestored in the log database and belongs to the target log type within the first preset time range is greater than a preset quantity threshold value.
Optionally, the number of the target log information is at least two, and on this basis, the log display module 420 may specifically include:
the system comprises a request single number acquisition unit, a request single number acquisition unit and a service processing unit, wherein the request single number acquisition unit is used for acquiring a request single number associated with each target log information, and the request single number is a single number generated when an interface in a connected service system is requested to be called;
and the log display unit is used for displaying each target log information in a grouping way according to the request single number.
Optionally, the log processing system includes an information sending client, and the log information is stored in the log database through the following modules:
the first storage module is used for acquiring log information corresponding to a call trigger event when the call trigger event called by an interface is monitored, and storing the log information in a context, wherein the log information comprises parameter information or parameter information and abnormal information, and the parameter information comprises a parameter entering parameter and a parameter exiting parameter;
and the second storage module is used for sending the log information in the context to the information sending client when the interface calling is finished, so that the log information is stored in the log database through the information sending client, and the log information in the context is deleted.
Optionally, on this basis, the apparatus may further include:
the abnormal setting module is used for acquiring abnormal parameters and a parameter setting interface corresponding to the abnormal triggering event when the abnormal triggering event with abnormal parameters is monitored, and calling the parameter setting interface to set the abnormal parameters;
and the third storage module is used for determining the abnormal information corresponding to the abnormal parameters and storing the abnormal parameters and the abnormal information in the context.
In the log processing device provided by the fourth embodiment of the present invention, when the log query module obtains the query condition related to the abnormal positioning, the query can be performed in the log database according to the query condition, where the log database is a database in the log processing system; because the log information of each connected service system is stored in the log database, that is, the log information in the log database is from each connected service system, the target log information which is screened from each log information by the log display module according to the query result and is matched with the query condition can also be from each connected service system, and in consideration of the fact that the target log information comprises the source information of the service system, after the target log information is displayed, the user can locate the connected service system with abnormality according to the displayed target log information. The device queries the target log information related to the query condition through the unique entry provided by the log processing system, so that a user can locate the connected service system with the abnormality according to the service system source information in the target log information, and the effect of quickly locating the connected service system with the abnormality is realized.
The log processing device provided by the embodiment of the invention can execute the log processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, in the embodiment of the log processing apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE five
Fig. 8 is a schematic structural diagram of a log processing system according to a fifth embodiment of the present invention, and as shown in fig. 8, the log processing system includes a memory 510, a processor 520, an input device 530, and an output device 540. The number of processors 520 in the log processing system may be one or more, and one processor 520 is taken as an example in fig. 8; the memory 510, processor 520, input device 530, and output device 540 in the log processing system may be connected by a bus or other means, such as by bus 550 in fig. 8.
The memory 510 is used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the log processing method in the embodiment of the present invention (for example, the log query module 410 and the log display module 420 in the log processing apparatus). The processor 520 executes various functional applications and data processing of the log processing system by executing software programs, instructions and modules stored in the memory 510, that is, implements the log processing method described above.
The memory 510 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the log processing system, and the like. Further, the memory 510 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 510 may further include memory located remotely from processor 520, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.
EXAMPLE six
An embodiment of the present invention provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a log processing method when executed by a computer processor, where the method is applied to a log processing system, the log processing system is connectable to a plurality of connected service systems, a log database of the log processing system stores log information of each connected service system, and the log information includes service system source information, and the method may include:
when a positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in a log database according to the query condition;
and screening target log information matched with the query conditions from the log information according to the query result, and displaying the target log information.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the log processing method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. With this understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A log processing method is applied to a log processing system, the log processing system is connected with a plurality of connected service systems, log information of each connected service system is stored in a log database of the log processing system, the log information comprises service system source information, and the method comprises the following steps:
when a positioning trigger event of abnormal positioning is monitored, acquiring a query condition corresponding to the positioning trigger event, and querying in the log database according to the query condition;
and screening target log information matched with the query condition from the log information according to a query result, and displaying the target log information in a credit line mode.
2. The method of claim 1, further comprising:
when a connection triggering event of a service system connection is monitored, acquiring a to-be-connected log type of a to-be-connected service system corresponding to the connection triggering event, wherein the to-be-connected log type comprises a service type, an operation type and source information of the service system;
and if the type of the log to be connected is not consistent with the type of the log connected of the connected service system, allowing the service system to be connected to the log processing system.
3. The method of claim 2, further comprising:
when a configuration trigger event of the life cycle configuration is monitored, taking each connected log type corresponding to the configuration trigger event as a life cycle;
correspondingly, the displaying the target log information includes:
determining a target life cycle associated with the target log information from each life cycle;
and if the missing type which belongs to the target life cycle and is not contained in each connected log type of the target log information exists, displaying the target log information and the missing type.
4. The method of claim 2, further comprising:
acquiring a preset quantity threshold value matched with a target log type in all the connected log types;
and if the number of the log information which is prestored in the log database and belongs to the target log type in a first preset time range is greater than the preset number threshold, sending alarm information.
5. The method of claim 1, wherein the number of the target log information is at least two, and wherein displaying the target log information comprises:
acquiring a request single number associated with each target log information, wherein the request single number is a single number generated when an interface in the connected service system is requested to be called;
and grouping and displaying the target log information according to the request single number.
6. The method of claim 1, wherein the log processing system comprises an information sending client, and wherein the log information is stored in the log database by:
when a call trigger event called by an interface is monitored, acquiring the log information corresponding to the call trigger event, and storing the log information in a context;
when the interface calling is finished, sending the log information in the context to the information sending client, storing the log information into the log database through the information sending client, and deleting the log information in the context;
the log information comprises parameter information, or the parameter information and abnormal information, wherein the parameter information comprises a parameter entering parameter and a parameter exiting parameter.
7. The method of claim 6, further comprising:
when an abnormal triggering event with abnormal parameters is monitored, acquiring abnormal parameters and a parameter setting interface corresponding to the abnormal triggering event, and calling the parameter setting interface to set the abnormal parameters;
and determining the abnormal information corresponding to the abnormal parameters, and storing the abnormal parameters and the abnormal information in the context.
8. A log processing apparatus configured in a log processing system, the log processing system being connected to a plurality of connected service systems, log information of each of the connected service systems being stored in a log database of the log processing system, the log information including service system source information, the apparatus comprising:
the log query module is used for acquiring a query condition corresponding to an abnormal positioning trigger event when the abnormal positioning trigger event is monitored, and querying in the log database according to the query condition;
and the log display module is used for screening target log information matched with the query condition from the log information according to the query result and displaying the target log information.
9. A log processing system, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log processing method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011141527.8A CN113761015B (en) | 2020-10-22 | 2020-10-22 | Log processing method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011141527.8A CN113761015B (en) | 2020-10-22 | 2020-10-22 | Log processing method, device, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113761015A true CN113761015A (en) | 2021-12-07 |
| CN113761015B CN113761015B (en) | 2023-09-05 |
Family
ID=78785906
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011141527.8A Active CN113761015B (en) | 2020-10-22 | 2020-10-22 | Log processing method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113761015B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101163265A (en) * | 2007-11-20 | 2008-04-16 | 中兴通讯股份有限公司 | Distributed database based on multimedia message log inquiring method and system |
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| CN107256219A (en) * | 2017-04-24 | 2017-10-17 | 卡斯柯信号有限公司 | Big data convergence analysis method applied to automatic train control system massive logs |
| CN108259241A (en) * | 2018-01-11 | 2018-07-06 | 上海有云信息技术有限公司 | A kind of abnormal localization method and device of cloud platform monitoring system |
| US20180278499A1 (en) * | 2017-03-27 | 2018-09-27 | Ca, Inc. | Rendering application log data in conjunction with system monitoring |
| CN108880847A (en) * | 2017-05-16 | 2018-11-23 | 北京微影时代科技有限公司 | A kind of method and device of positioning failure |
| CN110162420A (en) * | 2019-04-26 | 2019-08-23 | 平安科技（深圳）有限公司 | Data assisted location method, device, computer equipment and storage medium |
| CN110245035A (en) * | 2019-05-20 | 2019-09-17 | 平安普惠企业管理有限公司 | A kind of link trace method and device |
| US20190354524A1 (en) * | 2018-05-16 | 2019-11-21 | Nec Laboratories America, Inc. | Computer log retrieval based on multivariate log time series |
| CN110519465A (en) * | 2019-07-22 | 2019-11-29 | 平安科技（深圳）有限公司 | Telephony platform log inquiring method, terminal device, storage medium and device |
| CN111522922A (en) * | 2020-03-26 | 2020-08-11 | 浙江口碑网络技术有限公司 | Log information query method and device, storage medium and computer equipment |
-
2020
- 2020-10-22 CN CN202011141527.8A patent/CN113761015B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101163265A (en) * | 2007-11-20 | 2008-04-16 | 中兴通讯股份有限公司 | Distributed database based on multimedia message log inquiring method and system |
| CN106844171A (en) * | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | Mass operation and maintenance implementation method |
| US20180278499A1 (en) * | 2017-03-27 | 2018-09-27 | Ca, Inc. | Rendering application log data in conjunction with system monitoring |
| CN107256219A (en) * | 2017-04-24 | 2017-10-17 | 卡斯柯信号有限公司 | Big data convergence analysis method applied to automatic train control system massive logs |
| CN108880847A (en) * | 2017-05-16 | 2018-11-23 | 北京微影时代科技有限公司 | A kind of method and device of positioning failure |
| CN108259241A (en) * | 2018-01-11 | 2018-07-06 | 上海有云信息技术有限公司 | A kind of abnormal localization method and device of cloud platform monitoring system |
| US20190354524A1 (en) * | 2018-05-16 | 2019-11-21 | Nec Laboratories America, Inc. | Computer log retrieval based on multivariate log time series |
| CN110162420A (en) * | 2019-04-26 | 2019-08-23 | 平安科技（深圳）有限公司 | Data assisted location method, device, computer equipment and storage medium |
| CN110245035A (en) * | 2019-05-20 | 2019-09-17 | 平安普惠企业管理有限公司 | A kind of link trace method and device |
| CN110519465A (en) * | 2019-07-22 | 2019-11-29 | 平安科技（深圳）有限公司 | Telephony platform log inquiring method, terminal device, storage medium and device |
| CN111522922A (en) * | 2020-03-26 | 2020-08-11 | 浙江口碑网络技术有限公司 | Log information query method and device, storage medium and computer equipment |
Non-Patent Citations (4)
| Title |
|---|
| YAKOVLEV等: "Network anomaly detection based on flow intensity measuring in the decay model for ddos protection", T-COMM * |
| 张振;刘俊艳;: "基于微服务架构的日志监控系统的设计与实现", 软件, no. 11 * |
| 徐杨;袁峰;林琪;汤德佑;李东;: "基于混合人工免疫算法的流程挖掘事件日志融合方法", 软件学报, no. 02 * |
| 黄媛媛;王彬;龙卉;刘学生;: "基于微服务架构的日志系统", 电子技术与软件工程, no. 02 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113761015B (en) | 2023-09-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111680068B (en) | Verification method, device, equipment and storage medium | |
| CN112199276B (en) | Method, device, server and storage medium for detecting change of micro-service architecture | |
| CN110730246A (en) | Distributed link tracking method under micro-service architecture | |
| CN112631913B (en) | Method, device, equipment and storage medium for monitoring operation faults of application program | |
| CN112965874B (en) | Configurable monitoring alarm method and system | |
| US11310140B2 (en) | Mitigating failure in request handling | |
| CN110362455B (en) | Data processing method and data processing device | |
| CN110752969B (en) | Performance detection method, device, equipment and medium | |
| CN112737800B (en) | Service node fault positioning method, call chain generating method and server | |
| US11169896B2 (en) | Information processing system | |
| CN113760641A (en) | Service monitoring method, device, computer system and computer readable storage medium | |
| CN110597861A (en) | Real-time alarm method, device and equipment and computer readable storage medium | |
| CN112395172A (en) | Visual display method based on application software automation monitoring data | |
| CN107885634B (en) | Method and device for processing abnormal information in monitoring | |
| CN110968479A (en) | Business-level full-link monitoring method for application program and server | |
| CN114064402A (en) | Server system monitoring method | |
| CN104461847B (en) | Data processor detection method and device | |
| CN106789335B (en) | Method and system for processing information | |
| US10572458B2 (en) | Method and apparatus of collecting and reporting database application incompatibilities | |
| CN113761015A (en) | Log processing method, device and system and storage medium | |
| CN116645082A (en) | System inspection method, device, equipment and storage medium | |
| KR101973728B1 (en) | Integration security anomaly symptom monitoring system | |
| CN114531338A (en) | Monitoring alarm and tracing method and system based on call chain data | |
| CN111835566A (en) | System fault management method, device and system | |
| CN112698980A (en) | Exception handling method, electronic device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |