CN114443410A - Service log processing method and system and Internet of things system - Google Patents
Service log processing method and system and Internet of things system Download PDFInfo
- Publication number
- CN114443410A CN114443410A CN202011218065.5A CN202011218065A CN114443410A CN 114443410 A CN114443410 A CN 114443410A CN 202011218065 A CN202011218065 A CN 202011218065A CN 114443410 A CN114443410 A CN 114443410A
- Authority
- CN
- China
- Prior art keywords
- service
- log
- metadata
- target metadata
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3068—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data format conversion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a service log processing method and system and an Internet of things system, and relates to the technical field of Internet of things communication. The service log processing method comprises the following steps: collecting service logs in a plurality of micro-service modules, wherein the service logs comprise service identifiers and service data; acquiring target metadata in the service data; and storing the target metadata and the corresponding target service log in an index partition. The scheme of the invention can control the data volume of the index partition, reduce the occupation of memory and disk resources and improve the query efficiency of the service log.
Description
Technical Field
The invention belongs to the technical field of communication of the Internet of things, and particularly relates to a service log processing method and system and an Internet of things system.
Background
Currently, most internet of things systems have log management systems. Due to the micro-service of the current internet of things, a simple request often needs to be made to a plurality of micro-service modules at the back end. In order to fully acquire the operation information of the user in the server, the existing log management system needs to collect the logs in a whole amount, which brings huge pressure to message middleware and processing tools. And then sending the collected full logs to an index partition engine for storage, wherein the index partition engine realizes quick query through a file system cache placed in a memory. The full log storage, which stores a large amount of useless logs into the index partition engine, consumes a large amount of memory and disk space, so that the query efficiency of the service log is seriously reduced. Meanwhile, logs generated by different microservice modules are stored in the index partition engine according to different index partitions, and a public query index partition is generated for facilitating query, so that additional transfer query is added during query to generate a new index partition, and more memory space is consumed.
Disclosure of Invention
The embodiment of the invention aims to provide a service log processing method, a service log processing system and an Internet of things system, so that the problems of low query efficiency and excessive memory space occupation caused by full log collection in the prior art are solved.
In order to achieve the above object, an embodiment of the present invention provides a service log processing method, including:
collecting service logs in a plurality of micro-service modules, wherein the service logs comprise service identifiers and service data;
acquiring target metadata in the service data;
and storing the target metadata and the corresponding target service log in an index partition.
Optionally, the collecting the service logs in the multiple micro service modules includes:
and collecting service logs in the micro service modules according to a preset filtering principle.
Optionally, the service log further includes:
at least one of time information and redundant information;
wherein the time information comprises at least one field, each field representing a unit of time; the redundant information includes: at least one of a year, a log level, a thread identity, and a control class name field.
Optionally, the obtaining target metadata in the service data includes:
segmenting the service data according to key information in the service data to obtain a plurality of single metadata in the service data;
and respectively converting the plurality of single metadata into target metadata with the same format.
Optionally, the target metadata includes at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification, log status code, and log body field.
Optionally, the storing the target metadata and the corresponding target service log in an index partition includes:
adding routing information corresponding to a plurality of target metadata;
and storing the target metadata and the corresponding target service log in an index partition according to the routing information.
Optionally, the routing information is a product identity field, and the target metadata and the corresponding target service log having the same product identity field are stored in the same index partition.
Optionally, after collecting the service logs in the multiple micro service modules, the method includes:
and classifying and temporarily storing the service log according to the set name.
Optionally, the service log processing method further includes:
receiving a query request, the query request including a key field;
and determining target metadata comprising the key fields and corresponding target service logs according to the key fields, and displaying the target metadata and the corresponding target service logs.
Optionally, the method for processing the service log further includes:
deleting the expired service log after the first time;
after a second time, multiple small segments in the same index partition are merged into one large segment.
An embodiment of the present invention further provides a service log processing system, including:
the log acquisition module is used for acquiring service logs in the micro service modules, wherein the service logs comprise service identifiers and service data;
the processing module is used for acquiring target metadata in the service data;
and the search engine module is used for acquiring the target metadata and the corresponding target service log and storing the target metadata and the corresponding target service log in the index partition.
Optionally, the log collection module is specifically configured to:
and collecting service logs in the micro service modules according to a preset filtering principle.
Optionally, the service log further includes:
at least one of time information and redundant information;
wherein the time information comprises at least one field, each field representing a unit of time; the redundant information includes: at least one of a year, a log level, a thread identity, and a control class name field.
Optionally, the processing module is specifically configured to:
acquiring the service data, and segmenting the service data according to key information in the service data to acquire a plurality of single metadata in the service data;
and respectively converting the single metadata into target metadata with the same format, and sending the target metadata to a search engine module.
Optionally, the target metadata includes at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification, log status code, and log body field.
Optionally, the processing module is further specifically configured to:
adding routing information corresponding to a plurality of target metadata;
and sending the target metadata and the corresponding target service log to the search engine module according to the routing information.
Optionally, the routing information is a product identity field, and the target metadata and the corresponding target service log having the same product identity field are sent to the same index partition of the search engine module.
Optionally, the service log processing system further includes:
and the message middleware is used for classifying and temporarily storing the service logs sent by the log acquisition modules according to the set names and sending the service log messages to the processing module according to the set names.
Optionally, the service log processing system further includes:
and the query module is used for receiving a query request, wherein the query request comprises a key field, and determining target metadata comprising the key field and a corresponding target service log according to the key field.
Optionally, the query request is triggered by user input in a query interface or by an external system connected with an external interface.
Optionally, the service log processing system further includes:
and the timing module is used for deleting the expired service log after the first time and merging a plurality of small sections in the same index partition into a large section after the second time.
An embodiment of the present invention further provides an internet of things system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the service log processing method as described in any one of the above.
An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the service log processing method according to any one of the above.
The technical scheme of the invention at least has the following beneficial effects:
in the above scheme, the service log processing method includes: collecting service logs in a plurality of micro-service modules, wherein the service logs comprise service identifiers and service data; acquiring target metadata in the service data; and storing the target metadata and the corresponding target service log in an index partition. The scheme of the invention collects the service log by identifying the service identifier and stores the service log according to the target metadata, thereby controlling the data volume of the index partition, reducing the occupation of memory and disk resources and improving the query efficiency of the service log.
Drawings
Fig. 1 is a schematic flowchart illustrating a service log processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a service log processing system according to an embodiment of the present invention;
fig. 3 is a specific schematic diagram of a service log processing system according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments.
The embodiment of the invention provides a service log processing system, a service log processing method and an Internet of things system, aiming at the problems of low query efficiency and excessive memory space occupation in the prior art of full log collection.
As shown in fig. 1, an embodiment of the present invention provides a method for processing a service log, including:
step S11: collecting service logs in a plurality of micro-service modules, wherein the service logs comprise service identifiers and service data;
here, the micro service module may be an access server, an authentication module, a device online module, a statistics module, an application programming interface module, or the like, which the device accesses, depending on the function.
In this step, the log file in the microservice module includes a common log and a service log, and the service log includes metadata concerned by the user. In order to conveniently and uniformly process and store the service logs, the collected service logs are in a standard format according to a pre-established standardized format method, namely the service logs comprise service identifiers and service data, and the service identifiers can indicate that the logs are the service logs. Therefore, only the service logs containing the specific service identifier in the micro-service module can be collected based on the service identifier, and the rest common logs are directly discarded.
Step S12: acquiring target metadata in the service data;
in this step, the service data is processed to obtain the target metadata, so as to store the target metadata in the subsequent process, thereby improving the query efficiency of the service log. And the service log lacking the target metadata is directly discarded, so that the normalization of the data is ensured.
Step S13: and storing the target metadata and the corresponding target service log in an index partition.
In this step, service logs in different protocols and different micro-service modules are stored in the same index partition due to the unification of target metadata, so that a scheme of storing single index partitions is realized.
It should be noted that, in the service log processing method according to the embodiment of the present invention, the service log is standardized, the service log is collected by identifying the service identifier, and is stored according to the target metadata, so as to control the data volume of the index partition, reduce the memory and disk resources occupied, and improve the query efficiency of the service log.
It should be appreciated that the service log processing method of the embodiment of the invention can be applied to an internet of things system.
Specifically, step S11: the collecting the service logs in the plurality of micro-service modules comprises the following steps:
and collecting service logs in the micro service modules according to a preset filtering principle.
It should be noted that, in the embodiment of the present invention, a "-training" character string may be used as a service identifier, and the preset filtering principle is that a service log containing the "-training" in the micro service module can be collected, and other common logs are discarded.
Further, the service log further includes:
at least one of time information and redundant information;
wherein the time information comprises at least one field, each field representing a unit of time; the redundant information includes: at least one of a year, a log level, a thread identity, and a control class name field.
It should be noted that the time information in the embodiment of the present invention includes at least one of the following time unit fields: year, month, day, hour, minute, second, and millisecond fields.
It should be further noted that the format of the service log collected in the embodiment of the present invention conforms to a standardized format established in advance, that is, different protocols and different microservice modules print the service log according to the standardized format, and the service log in the standardized format is helpful for performing unified processing and storage. The standardized format adopted by the service log of the embodiment of the present invention is explained as follows:
yyyy-MM-dd HH MM: ss.SSS XXX-tracing { json string };
wherein "yyyy-MM-dd" means "year-month-day"; "HH: mm: ss.SSS" means "hour: and (2) minutes: hour, millisecond "; "XXX" represents "redundant information"; the "-training" string represents a service identity, which is an identity of whether a service log was collected or not, requiring that the string cannot exist in the json string and other parts of the service log, and that the string cannot exist in a normal log.
The json string is service data, and is defined as the following table:
specifically, step S12: the obtaining of the target metadata in the service data includes:
segmenting the service data according to key information in the service data to obtain a plurality of single metadata in the service data;
and respectively converting the plurality of single metadata into target metadata with the same format.
It should be noted that the service data is content in the json character string, that is, metadata. And according to the key information shown in the table, dividing the content in the json character string to obtain a plurality of single metadata. The single segmented metadata is of a character string type and needs to be converted into an integer type, so that target metadata in the service data is obtained. Here, the character string data about time also needs to be converted into data of an integer type after converting the time in different time zones into the same time zone.
The target metadata includes at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification (name), log status code, and log body field.
Specifically, step S13: the storing the target metadata and the corresponding target service log in an index partition includes:
adding routing information corresponding to a plurality of target metadata;
and storing the target metadata and the corresponding target service log in an index partition according to the routing information.
It should be noted that, the service log in the above standardized format is added with routing information, and the service log is stored and queried in a routing manner, so that the query efficiency is significantly improved.
Further, the routing information is a product identity field, and the target metadata and the corresponding target service log having the same product identity field are stored in the same index partition.
It should be noted that, when the service logs of multiple microservice modules with different protocols are stored in the same index partition according to the routing information, the service logs of all the protocols and the microservice modules are located by querying one index partition when the log query is implemented, so that the service logs related to multiple index partitions are prevented from being searched, the query complexity is simplified, and the query efficiency is improved.
Specifically, step S11: after the collecting the service logs in the multiple microservice modules, the method includes:
and classifying and temporarily storing the service log according to the set name.
When the data amount of the collected service logs is too large, in order to improve the concurrency of data processing, the collected service logs are classified and stored according to the collection names. Here, the service log, such as a clothing order type, includes a set name divided according to the kind of the clothing order, and the set name may be a clothing order, a trousers order, or the like. And classifying and temporarily storing the service logs according to different set names.
In an optional embodiment of the present invention, the service log processing method further includes:
receiving a query request, the query request including a key field;
and determining target metadata comprising the key fields and corresponding target service logs according to the key fields, and displaying the target metadata and the corresponding target service logs.
It should be noted that the query request may be triggered by a user input in the query interface, or by an external system through an external interface. During query, the key field is used, for example, the product identity field is used as a route, and once query can be carried out, so that the key field can be positioned in the index partition of the service log of the product, and the query efficiency is improved.
In an optional embodiment of the present invention, the service log processing method further includes:
deleting the expired service log after the first time;
after a second time, multiple small segments in the same index partition are merged into one large segment.
It should be noted that the data amount of the index partition is reduced by deleting redundant or expired contents in the index partition. And merging the target metadata of a plurality of small sections in the same index partition into one large section of target metadata so as to control the data volume of the index partition and improve the query efficiency.
As shown in fig. 2 to fig. 3, an embodiment of the present invention further provides a service log processing system, including:
the log collection module 21 is configured to collect service logs in the multiple micro service modules 31, where the service logs include service identifiers and service data;
it should be noted that, since one micro service module 31 on the platform of the internet of things corresponds to one log collection module 21, the number of the log collection modules 21 in the embodiment of the present invention may be one or more. The log collection module 21 occupies less memory and processor resources, and can be deployed in a virtual machine or a container together with a service program, and hardly occupies system resources.
The processing module 22 is configured to obtain target metadata in the service data;
it should be noted that the processing module 22 according to the embodiment of the present invention may be an ETL (Extract, Transform, Load, Extract, convert, and Load) module, and the ETL module is deployed in a cluster manner, and when the Load is too large, the concurrent processing capability of the system may be improved by adding a node.
And the search engine module 23 is configured to obtain the target metadata and the corresponding target service log, and store the target metadata and the corresponding target service log in the index partition.
It should be noted that, the Search engine module 23 in the embodiment of the present invention employs an Elastic Search (Lucene-based Search server) query engine, and the Elastic Search employs a full-text index partition mode, so as to improve query efficiency, and employs a distributed mode for deployment, so as to further improve query performance of the system, and can achieve near real-time Search, have a second-level response speed, and be stable and reliable.
The service log processing system can be applied to an Internet of things system and other devices needing service log processing, service logs are standardized, the service logs are collected by identifying service identifiers and stored according to target metadata, the data volume of index partitions is controlled, occupied memory and disk resources are reduced, and the query efficiency of the service logs is improved.
The log collection module 21 is specifically configured to:
and collecting the service logs in the micro service modules 31 according to a preset filtering principle.
It should be noted that the log collection module 21 may configure a log directory and a filtering principle that need to be collected, where the filtering principle is to use a "-training" character string as a service identifier, the log collection module 21 only collects service logs that include "-training" in the microservice module, and the rest of common logs are discarded.
Here, the service log further includes:
at least one of a time field and a redundancy field;
wherein the time field comprises at least one field, each field representing a unit of time; the redundancy field includes: at least one of a year, a log level, a thread identity, and a control class name field.
It should be noted that the format of the service log acquired by the log acquisition module 21 in the embodiment of the present invention conforms to a predefined standardized format, that is, different protocols and different micro service modules 31 all print the service log according to the standardized format, and the standardized format is already described in the service log processing method described above and is not repeated.
Wherein the processing module 22 is specifically configured to:
acquiring the service data, and segmenting the service data according to key information in the service data to acquire a plurality of single metadata in the service data;
and respectively converting the single metadata into target metadata with the same format, and sending the target metadata to a search engine module.
It should be noted that the processing module 22 obtains the service data, i.e., the metadata, collected in the log collection module 21, and converts the metadata of the character string type into the metadata of the integer type, so as to obtain the target metadata. The processing module 22 may also discard the service log with the lost metadata, implement secondary filtering, and ensure the normalization of the data.
The target metadata includes at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification, log status code, and log body field.
Wherein, the processing module 22 is further specifically configured to:
adding routing information corresponding to a plurality of target metadata;
and sending the target metadata and the corresponding target service log to the search engine module 23 according to the routing information.
Here, the routing information is a product identity field, and the target metadata and the corresponding target service log having the same product identity field are sent to the same index partition of the search engine module 23.
It should be noted that, the processing module 22 sends the target metadata of the same product and the corresponding target service log to the same index partition in the search engine module 23 in the service logs of multiple micro service modules 31 with different protocols and using the product identity as the routing information, and when the log is queried, locates the service logs of all the protocols and the micro service modules 31 by querying one index partition, thereby avoiding searching for related service logs across multiple index partitions, simplifying query complexity, and improving query efficiency.
In an optional embodiment of the present invention, the service log processing system further includes:
the message middleware 32 is configured to classify and temporarily store the service logs sent by the log collection modules 21 according to the set names, and send the service log messages to the processing module 22 according to the set names.
It should be noted that, when the data volume of the service log processing system is excessive, in order to improve the concurrency of data processing, the log acquisition module 21 sends the acquired service logs to the message middleware 32, the message middleware 32 classifies and stores the acquired service logs according to the set names, extracts corresponding service logs according to the set names and sends the service logs to the processing module 22, and the message middleware 32 has a peak-eliminating decoupling function.
In an optional embodiment of the present invention, the service log processing system further includes:
the query module 33 is configured to receive a query request, where the query request includes a key field, and determine, according to the key field, target metadata including the key field and a corresponding target service log.
Further, the query request is triggered by user input in a query interface or by an external system connected with an external interface.
It should be noted that the query module 33 may include a front-end query module and a presentation module, the front-end query module is used for inputting a key field, and the key field may be one or more target metadata, and supports multiple dimensional queries. After the user inputs or selects the key field, the back-end query service receives the query request including the key field, searches the corresponding target service log in the search engine module 23 according to the target metadata, and displays the target service log on the display module. The query module may further include an external interface, which may be an application programming interface of an external service. The external system is connected with the service log processing system through an external interface, after the external system is authorized by the service log processing system, the external interface sends an inquiry request of the external system, including the key field, to a back-end inquiry service, and the key field, namely the target service log corresponding to the target metadata, is returned. The back-end query service can also be used as the routing information according to the product identity field, and the index partition of the service log of the product can be positioned by one-time query, so that the search across multiple index partitions is avoided, and the query efficiency is obviously improved.
In an optional embodiment of the present invention, the service log processing system further includes:
and the timing module is used for deleting the expired service log after the first time and merging a plurality of small sections in the same index partition into a large section after the second time.
It should be noted that, by using the timing module, for example, redundant or expired contents in the index partition can be deleted every morning, so as to reduce the data amount in the index partition; meanwhile, requests can be sent periodically, and the target metadata of a plurality of small segments in the same index partition is merged into the target metadata of one large segment, so that the data volume in the index partition can be controlled, and the query efficiency is improved.
An embodiment of the present invention further provides an internet of things system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the service log processing method as described above is implemented.
It should be noted that, the internet of things system adopts the above-mentioned service log processing method, and has the following technical effects:
filtering the collected service logs, reducing the data volume to be stored under the condition of limited memory resources, and improving the query efficiency; the service logs of different protocols and different micro-service modules uniformly adopt a pre-established standardized format, so that target metadata in all the service logs are the same and are stored in the same index partition in a routing mode, query is facilitated, and query efficiency is further improved.
An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the service log processing method as described above.
It should be noted that, the computer-readable storage medium implements the service log processing method described above, and has the technical effects described above, and details are not repeated here.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (23)
1. A method for processing service logs is characterized by comprising the following steps:
collecting service logs in a plurality of micro-service modules, wherein the service logs comprise service identifiers and service data;
acquiring target metadata in the service data;
and storing the target metadata and the corresponding target service log in an index partition.
2. The method of claim 1, wherein the collecting the service logs in the plurality of microservice modules comprises:
and collecting service logs in the micro service modules according to a preset filtering principle.
3. The traffic log processing method according to claim 1, wherein the traffic log further comprises:
at least one of time information and redundant information;
wherein the time information comprises at least one field, each field representing a unit of time; the redundant information includes: at least one of year, log level, thread identity, and control class name fields.
4. The method of claim 1, wherein the obtaining the target metadata in the service data comprises:
segmenting the service data according to key information in the service data to obtain a plurality of single metadata in the service data;
and respectively converting the plurality of single metadata into target metadata with the same format.
5. The traffic log processing method of claim 1, wherein the target metadata comprises at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification, log status code, and log body field.
6. The method of claim 1, wherein storing the target metadata and the corresponding target service log in an index partition comprises:
adding routing information corresponding to a plurality of target metadata;
and storing the target metadata and the corresponding target service log in an index partition according to the routing information.
7. The service log processing method according to claim 6, wherein the routing information is a product identity field, and the target metadata having the same product identity field and the corresponding target service log are stored in the same index partition.
8. The method of claim 1, wherein after collecting the service logs in the plurality of microservice modules, the method comprises:
and classifying and temporarily storing the service log according to the set name.
9. The service log processing method of claim 1, further comprising:
receiving a query request, the query request including a key field;
and determining target metadata comprising the key fields and corresponding target service logs according to the key fields, and displaying the target metadata and the corresponding target service logs.
10. The service log processing method of claim 1, further comprising:
deleting the expired service log after the first time;
after a second time, the small segments in the same index partition are merged into one large segment.
11. A service log processing system, comprising:
the log acquisition module is used for acquiring service logs in the micro service modules, wherein the service logs comprise service identifiers and service data;
the processing module is used for acquiring target metadata in the service data;
and the search engine module is used for acquiring the target metadata and the corresponding target service log and storing the target metadata and the corresponding target service log in the index partition.
12. The service log processing system of claim 11, wherein the log collection module is specifically configured to:
and collecting service logs in the micro service modules according to a preset filtering principle.
13. The traffic log processing system of claim 11, wherein the traffic log further comprises:
at least one of time information and redundant information;
wherein the time information comprises at least one field, each field representing a unit of time; the redundant information includes: at least one of a year, a log level, a thread identity, and a control class name field.
14. The service log processing system of claim 11, wherein the processing module is specifically configured to:
acquiring the service data, and segmenting the service data according to key information in the service data to acquire a plurality of single metadata in the service data;
and respectively converting the single metadata into target metadata with the same format, and sending the target metadata to a search engine module.
15. The transaction log processing system of claim 11, wherein the target metadata comprises at least one of the following fields:
link identity, log category, request start time, request end time, message identity, product identity, device identification, log status code, and log body field.
16. The service log processing system of claim 14, wherein the processing module is further specifically configured to:
adding routing information corresponding to a plurality of target metadata;
and sending the target metadata and the corresponding target service log to the search engine module according to the routing information.
17. The service log processing system of claim 16, wherein the routing information is a product identity field, and the target metadata and the corresponding target service log having the same product identity field are sent to a same index partition of the search engine module.
18. The transaction log processing system of claim 11, further comprising:
and the message middleware is used for classifying and temporarily storing the service logs sent by the log acquisition modules according to the set names and sending the service log messages to the processing module according to the set names.
19. The service log processing system of claim 11, further comprising:
and the query module is used for receiving a query request, wherein the query request comprises a key field, and determining target metadata comprising the key field and a corresponding target service log according to the key field.
20. The service log processing system of claim 19 wherein the query request is triggered by user input in a query interface or by an external system connected through an external interface.
21. The service log processing system of claim 11, further comprising:
and the timing module is used for deleting the expired service log after the first time and merging a plurality of small sections in the same index partition into a large section after the second time.
22. An internet of things system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the traffic log processing method according to any one of claims 1 to 10 when executing the computer program.
23. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements a service log processing method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011218065.5A CN114443410A (en) | 2020-11-04 | 2020-11-04 | Service log processing method and system and Internet of things system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011218065.5A CN114443410A (en) | 2020-11-04 | 2020-11-04 | Service log processing method and system and Internet of things system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114443410A true CN114443410A (en) | 2022-05-06 |
Family
ID=81361605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011218065.5A Pending CN114443410A (en) | 2020-11-04 | 2020-11-04 | Service log processing method and system and Internet of things system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443410A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115378802A (en) * | 2022-08-24 | 2022-11-22 | 深圳市晨北科技有限公司 | Log collection method, device and equipment and computer readable storage medium |
-
2020
- 2020-11-04 CN CN202011218065.5A patent/CN114443410A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115378802A (en) * | 2022-08-24 | 2022-11-22 | 深圳市晨北科技有限公司 | Log collection method, device and equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103020204B (en) | A kind of method and its system carrying out multi-dimensional interval query to distributed sequence list | |
| CN102236581B (en) | Mapping reduction method and system thereof for data center | |
| CN103235820B (en) | Date storage method and device in a kind of group system | |
| CN109947796B (en) | Caching method for query intermediate result set of distributed database system | |
| CN103034735A (en) | Big data distributed file export method | |
| CN113312376B (en) | Method and terminal for real-time processing and analysis of Nginx logs | |
| CN114116762A (en) | Offline data fuzzy search method, device, equipment and medium | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN114443410A (en) | Service log processing method and system and Internet of things system | |
| WO2017000592A1 (en) | Data processing method, apparatus and system | |
| CN111343269B (en) | Data downloading method, device, computer equipment and storage medium | |
| CN111897792A (en) | Distributed file access method, system, medium and device | |
| CN113761079A (en) | Data access method, system and storage medium | |
| CN109800285B (en) | Flexible medical record data extraction method and system and database server | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| US11915044B2 (en) | Distributed task assignment in a cluster computing system | |
| CN111045987B (en) | Ceph-based distributed file system metadata access acceleration method and system | |
| CN110895538A (en) | Data retrieval method, device, storage medium and processor | |
| CN113792111A (en) | Unstructured data management system and method based on distributed storage | |
| KR20120085375A (en) | Analysis system for log data | |
| CN115048437A (en) | Data service method and system based on nuclear power Paas platform | |
| CN111581220A (en) | Storage and retrieval method, device, equipment and storage medium for time series data | |
| CN111061719A (en) | Data collection method, device, equipment and storage medium | |
| CN112148925A (en) | User identification correlation query method, device, equipment and readable storage medium | |
| CN112084249B (en) | Access record extraction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |