CN110019045B - Log floor method and device - Google Patents
Log floor method and device Download PDFInfo
- Publication number
- CN110019045B CN110019045B CN201711489249.3A CN201711489249A CN110019045B CN 110019045 B CN110019045 B CN 110019045B CN 201711489249 A CN201711489249 A CN 201711489249A CN 110019045 B CN110019045 B CN 110019045B
- Authority
- CN
- China
- Prior art keywords
- log
- landing
- component
- category
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a log landing method and a log landing device, wherein the method comprises the following steps: receiving a log access request of a service, wherein the log access request carries a log category required to be accessed and at least one target landing component; analyzing the log access request, and determining that the log of the service access belongs to a consumed log according to the log category; and adding configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, and writing the log into the target landing component to finish log landing. Based on the log landing method provided by the invention, when the consumed log is landed to a new landing component, the landing of the log can be realized only by adding the configuration information of the target landing component which needs to be landed in the configuration file of the log, repeated consumption of the same log can be avoided, the log can be landed to the component quickly and effectively for the service, and the service access efficiency is further improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log landing method and apparatus.
Background
With the continuous development of computer technology, the storage amount and the calculation amount of various data are also increasing. In the face of increasingly huge data volumes, when a service needs to perform data access, how to effectively and quickly write the log data needed by each service is a problem to be solved.
In the conventional technology, when a database cluster receives a data access request, a data log needs to be landed in each target system, if the requirements of different target systems on the log are the same, the database cluster needs to realize relevant codes for each target system, and the data log is repeatedly developed and repeatedly consumed. By adopting the mode, resources are wasted, the data access efficiency is reduced, and the running speed of the system is slowed down.
Disclosure of Invention
The present invention provides a log landing method and apparatus to overcome or at least partially solve the above problems.
According to one aspect of the present invention, there is provided a log-landing method including:
receiving a log access request of a service, wherein the log access request carries a log category required to be accessed and at least one target landing component;
analyzing the log access request, and determining that the log of the service access belongs to a consumed log according to the log category;
and adding configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, and writing the log into the target landing component to finish log landing.
Optionally, the parsing the log access request, determining, according to the log category, that the log of the service access belongs to a consumed log includes:
analyzing the log access request, obtaining a log category carried in the log access request, and judging whether the log category exists in a preset daily category list or not;
if yes, determining that the log of the service access belongs to the consumed log.
Optionally, the analyzing the log access request, obtaining the log category, and after determining whether the log category exists in the preset daily category list, further includes:
if not, determining that the log accessed by the service belongs to an unconsumed log;
and adding the log category to the preset log category list, and consuming the log from the appointed cluster.
Optionally, the adding the log category to the preset log category list and consuming the log from the designated cluster includes:
adding field information corresponding to the log category in the preset log category list;
and collecting log information corresponding to the log category from the appointed cluster.
Optionally, adding configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, writing the log into the target landing component, and completing log landing, including:
adding field information of the target landing component in a landing component command specified in a configuration file of the log;
determining basic configuration information required by the target landing component to realize log landing according to the type of the target landing component;
and adding basic configuration information of the target landing component in the configuration file to provide an access interface of the target landing component, and writing the log into the target landing component to finish log landing.
Optionally, after adding the configuration information of the target landing component in the configuration file according to the type of the target landing component, the method further includes:
restarting the application corresponding to the log category.
Optionally, the log access request further carries a landing path for writing the log into the target landing component;
the step of adding the configuration information of the target landing component in the configuration file of the log according to the type of the target landing component, writing the log into the target landing component, and after finishing log landing, further comprises:
and returning a message of successful log landing to the service, and reading the log by the service based on the landing path.
Alternatively, the method is applied to the Kafka system.
According to another aspect of the present invention, there is also provided a log landing apparatus, including:
the request receiving module is configured to receive a log access request of a service, wherein the log access request carries a log type required to be accessed and at least one target landing component;
the analysis module is configured to analyze the log access request and determine that the log accessed by the service belongs to the consumed log according to the log category;
and the landing module is configured to add the configuration information of the target landing component into the configuration file of the log according to the type of the target landing component, write the log into the target landing component and finish log landing.
Optionally, the parsing module includes:
the judging unit is configured to analyze the log access request, acquire the log category carried in the log access request and judge whether the log category exists in a preset daily category list or not;
and the first determining unit is configured to determine that the log accessed by the service belongs to the consumed log when the log class exists in a preset daily class list.
Optionally, the parsing module further includes:
a second determining unit configured to determine that the log of the service access belongs to an unconsumed log when the log category does not exist in a preset day category list; and adding the log category to the preset log category list, and consuming the log from the appointed cluster.
Optionally, the second determining unit is further configured to:
adding field information corresponding to the log category in the preset log category list; and collecting log information corresponding to the log category from the appointed cluster.
Optionally, the floor module is further configured to:
adding field information of the target landing component in a landing component command specified in a configuration file of the log;
determining basic configuration information required by the target landing component to realize log landing according to the type of the target landing component;
and adding basic configuration information of the target landing component in the configuration file to provide an access interface of the target landing component, and writing the log into the target landing component to finish log landing.
Optionally, the apparatus further includes: and the restarting module is configured to restart the application corresponding to the log category.
Optionally, the log access request further carries a landing path for writing the log into the target landing component;
the apparatus further comprises: and the information sending module is configured to return a message of successful log landing to the service, and the service reads the log based on the landing path.
According to another aspect of the present invention, there is also provided an electronic apparatus including:
a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the log-hooking method according to any of the preceding claims.
According to another aspect of the present invention, there is also provided a computer-readable storage medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the log-landing method according to any one of the above.
The invention provides a more efficient log landing method and device, wherein after receiving a log access request of a service, the log landing method acquires a log type required to be accessed by the service and a target landing component of the log by analyzing the log access request, and when determining that the log corresponding to the log type belongs to a consumed log, the log landing method can directly add configuration information of the target landing component in a configuration file of the log so as to write the log into the target landing component and further finish log landing. Based on the log landing method provided by the invention, when the consumed log is landed to a new landing component, the landing of the log can be realized only by adding the configuration information of the target landing component which needs to be landed in the configuration file of the log, repeated consumption of the same log can be avoided, the log can be landed to the component quickly and effectively for the service, and the service access efficiency is further improved.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a flow diagram of a log-landing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a Hamal uniform floor frame structure according to an embodiment of the invention;
FIG. 3 is a data flow diagram of a Hamal-based unified floor frame in accordance with an embodiment of the present invention;
FIG. 4 is a schematic diagram of a log landing device according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a log landing device according to a preferred embodiment of the present invention;
FIG. 6 is a block diagram schematic of a computing device for performing a log-landing method according to an embodiment of the present invention; and
fig. 7 is a schematic diagram of a memory unit for holding or carrying program code for implementing a log-landing method according to the present invention, according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a flow chart of a log-landing method according to an embodiment of the present invention, as shown in fig. 1, the log-landing method according to an embodiment of the present invention includes:
step S102, receiving a log access request of a service, wherein the log access request carries a log category required to be accessed and at least one target landing component;
step S104, analyzing the log access request, and determining that the log accessed by the service belongs to the consumed log according to the log category;
and step S106, adding configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, and writing the log into the target landing component to finish log landing.
The embodiment of the invention provides a more efficient log landing method, which is characterized in that after a log access request of a service is received, a log category which is required to be accessed by the service and a target landing component of the log are obtained by analyzing the log access request, and when the log corresponding to the log category is determined to belong to a consumed log, the configuration information of the target landing component can be directly added into a configuration file of the log, so that the log is written into the target landing component to finish log landing. According to the log landing method provided by the embodiment of the invention, when the consumed log is landed to the new landing component, the landing of the log can be realized only by adding the configuration information of the target landing component which needs to be landed in the configuration file of the log, repeated consumption of the same log can be avoided, the log can be landed to the component quickly and effectively for the service, and the service access efficiency is further improved.
Consumption logs, i.e., pulling data from a data storage system. Embodiments of the present invention are preferably applied to the Kafka system. Kafka is a distributed publish/subscribe based messaging system. In the embodiment of the invention, kafka is stored as a bridge, and data in each database enters into Kafka, and Kafka only provides service for each service by reading the data. When each service needs to read data, the data can be pulled into Kafka. For example, when different businesses need to drop the same log into multiple landing components such as HDFS (Hadoop distributed file system), redis, HBase, etc., the same log may be consumed many times, resulting in many repeated consumption, and further wasting resources.
In the log landing method provided by the embodiment of the invention, when a request that a service needs to consume a certain log is received, the log required by the service is landed in a new landing component. At this time, it may be first determined whether the log has been consumed, and if so, the configuration information of the new floor component may be directly added to the configuration file of the log. For example, with a log, the original service only needs to fall to HDFS, and now the service wants to fall to the same log at the dis, only the dis-related configuration needs to be added to the configuration file of the log.
Optionally, the step S104 may further determine that the log belongs to the consumed log by: analyzing the log access request, obtaining a log category carried in the log access request, and judging whether the log category exists in a preset daily category list or not; if yes, determining that the log accessed by the service belongs to the consumed log. If not, determining that the log accessed by the service belongs to an unconsumed log; and adding the log category to a preset log category list, and consuming the log from the appointed cluster.
For the Kafka cluster, each message issued to the Kafka cluster has a category called topic, and each consumer must specify its topic. In an embodiment of the present invention, when a data access request for a service is received, the request may specify a topic to be consumed from Kafka. The cluster may pre-store a topics or topic list (category list), after receiving a request of a new service, parse the request to obtain a topic to be consumed in the request, and determine whether the topic exists in a preset topic list, if so, directly determine that the topic has been consumed. If the topic is not in the preset topic list, the topic is added to the preset topic list.
Preferably, adding the log category to the preset log category list and consuming the log from the specified cluster may include: adding field information corresponding to the log category in a preset log category list; log information corresponding to the log category is collected from the specified cluster. For example, the topic field carried in the request may be added to an existing topic list. And then consuming according to a protocol for analyzing the topic, namely realizing the collection of the log. That is, the topic field in the topic list can be arbitrarily increased or decreased according to the service requirement.
In step S106, the configuration information of the target landing component may be added to the configuration file of the log according to the type of the target landing component, and the log may be written into the target landing component, so as to complete the landing of the log. The configuration information of the floor component is added in the configuration file of the log, which is actually the realization of the interface. The access interface to the consumed log in the embodiment of the invention can be a loadable interface. Since multiple landing components, such as Redis, HBase, HDFS, are included, when a business needs to land the log to one or more of the components, such as Redis, only one Redis interface needs to be implemented to meet the business's needs. In this embodiment, the relevant configuration information of the dis is added in the configuration file of the log, that is, the log realizes that the dis can access the interface, which is equivalent to the fact that the log needed by the service is written into the dis, so as to complete the landing of the log.
In practical applications, the configuration information of each landing component is different. For example, the service needs to land the log to the HDFS, and the configuration information to be added may include: the root directory of the floor, the compression mode and the floor format are also adopted. That is, each landing component has its own configuration information, and then when log landing is required, it is only necessary to provide relevant information.
In addition, if the log is applied to the two components of the HDFS and the Redis in the access request submitted by the service, the log is written into the two components, and when the log is written into different landing components, the format of the log can be converted according to the data storage formats of the landing components so as to meet the data format requirements of the landing components, and the access service is provided for the service quickly and efficiently.
Optionally, after adding the configuration information of the floor component to the configuration file of the log, an Application corresponding to the log category, that is, application, may be restarted. Application is equivalent to service level, each log belongs to a service, and when a service needs to add a floor component, the Application corresponding to the log is restarted, so that the newly added configuration information is validated.
Further, the log access request also carries a landing path for writing the log into the target landing component, after the configuration file of the log is modified, a message of successful landing of the log can be returned to the service, and the service reads the log based on the landing path. The message that the log returned to the service is successfully landed may be a mail for the service or may be other message, which indicates that the specified log is landed to the specified landing component according to the request of the service.
As mentioned above, the log of the service request is written to each landing component to implement the landing of the log. Preferably, each landing component in this embodiment may be implemented and managed by a Flink. In addition to the above floor components, the Flink may implement a variety of source, sink components, which are the target floor components mentioned above, such as HDFS, redis, HBase, hive, ELK, and the like.
Based on the log landing method provided by the embodiment, the embodiment of the invention can provide a landing architecture Hamal commonly realized by Kafka and Flink, and unified monitoring and management of data are performed based on the framework. As shown in FIG. 2, the unified floor architecture Hamal based on Kafka and Flink provided by the embodiment of the invention. The framework processes the data stream by Kafka, through which the consumer's progress of consumption is preserved.
The offset is a record of which point to start to acquire the log information, although it is removed from Kafka, but not necessarily in real time, and it is also possible to locate data two days ago or several days ago by the offset, and land to sink components according to the service needs after the located log two days ago or several days ago. When the log of the located time period has landed, then the later data may land in real time.
In the unified floor architecture provided in this embodiment, the offset mainly includes two threads, checkPointer and burrow (Lag Monitor).
The CheckPointer effect, once the user changes data, the buffer becomes dirty. The user's commit does not mean that the change has been written to a data file, i.e., a WAL log (write-ahead log) file, but rather that a checkpoint process is required to do so. When a checkpointing operation occurs, all dirty pages are written into the table and index file; at this point a special checkpoint record is written to the WAL log file. When a crash occurs, the crash recovery program looks at the latest checkpoint record, decides where to begin the REDO (REDO log I file) operation from by the special checkpoint record in the WAL log file. Thus, after a checkpoint has been performed, we do not need the log section before this point.
Burrow is a suitable way to monitor the health of Kafka consumers, which provides a comprehensive overview of consumer status to the user and provides a delayed verification service for consumers without the need to specify thresholds. The Burrow monitors the deviation value of the data submitted by all consumers according to the topic partition, and calculates the state of the corresponding consumers according to the requirements.
In the Hamal unified floor frame provided by the embodiment of the invention, kafka can be combined with Hamal Framework (Hamal architecture) and Flink on Yarn Runtime (Flink on Yarn operation), and a plurality of sink components realized based on Flink can be included: HDFS (BucketingSink), HBase (HBase Sink), graylog (Graylog Sink), redis, cassandra, etc. When receiving a log access request of the service, the request carries the log type of the accessed log and the landing component of the log, and then relevant configuration is carried out. The data flow based on the framework described above may be as shown in fig. 3.
And the client of the message is read from the client of the message by the Kafka Consumer, the Consumer pulls (pull) log data from the reader, then the data is subjected to deserialization, format conversion is carried out on the processed data according to different component types, and then the processed data is written into each component to finish log landing.
The above embodiment will be described in detail by way of a preferred embodiment.
1. Receiving a log access request from a service, wherein the request carries landing of a specific log to hdfs, kafka and corresponding landing paths;
2. modifying the configuration file of the log, adding hdfs and kafka fields in a floor component command of the log, and adding basic configuration information of the hdfs and the kafka;
3. restarting Application corresponding to the log; the log can be written into hdfs and kafka respectively according to the landing path carried in the log access request while restarting
4. And feeding back information of successful landing to the service, wherein the service can directly read the log according to the landing path in the log access request.
The process of the above preferred embodiment can be implemented by the following code:
the// designation of from which Kafka cluster to consume
kafka.clusterId＝shbt2
kafka.retries＝5
kafka.auto.offset.reset＝earliest
The following is the topics which needs to be consumed from kafka, and topics can be increased or decreased according to the business
hamal.source.kafka.topics＝d.x.6.m,d.x.6,d.u.6.m,d.u.6,d.s.6.m,d.s.6,d.c.6.m,d.c.6
Protocol defining how to parse the topic
hamal.source.kafka.message.deserializer.class＝com.mvad.hamal.lib.source.io.RawlogDeserializer
The data processing process is to do special treatment such as filtering, sampling and the like
hamal.processor.class＝com.mvad.hamal.lib.processor.DSPProcessor
The designation falls to hdfs, kafka (kafka itself is also a component that can fall to ground)
hamal. Sink. Target=hdfs, kafka// can also be added with a redis
Configuration of the/(kafka sink Assembly
hamal.sink.kafka.clusterId＝shbt2
hamal.sink.kafka.message.serializer.class＝com.mvad.hamal.lib.source.io.DSPSampleKafkaMessageSerializer
Configuration information of/(hdfs sink component
hamal.sink.hdfs.basedir＝/mvad/rawlog/dsp
hamal.sink.hdfs.format＝LzoThriftBlockWithMeta
hamal.sink.hdfs.bucketing.filename.prefix＝dsp
hamal.sink.hdfs.bucketing.filename.suffix＝.dsp.lzo
As can be seen from the above embodiments, when several components are configured in the configuration file of the log, the configuration information of each component needs to be correspondingly added, and the service can be flexibly configured by adding or subtracting the topic consumed by the user or adding or subtracting the components.
Based on the same inventive concept, the embodiment of the invention also provides a log landing device. As shown in fig. 4, the log landing device according to an embodiment of the present invention may include:
the request receiving module 410 is configured to receive a log access request of a service, where the log access request carries a log category of a required access and at least one target landing component;
the parsing module 420 is configured to parse the log access request, and determine that the log accessed by the service belongs to the consumed log according to the log category;
the landing module 430 is configured to add the configuration information of the target landing component to the configuration file of the log according to the type of the target landing component, and write the log into the target landing component to complete the landing of the log.
In a preferred embodiment of the present invention, as shown in fig. 5, the parsing module 420 may include:
a judging unit 421 configured to parse the log access request, obtain a log category carried in the log access request, and judge whether the log category exists in a preset daily category list;
the first determining unit 422 is configured to determine that the log of the service access belongs to a consumed log when the log category exists in a preset day category list.
In a preferred embodiment of the present invention, as shown in fig. 5, the parsing module 420 may further include:
a second determining unit 423 configured to determine that the log of the service access belongs to an unconsumed log when the log category does not exist in the preset day category list; and adding the log category into a preset log category list, and consuming the log from the appointed cluster.
In a preferred embodiment of the present invention, the second determining unit 423 may be further configured to: adding field information corresponding to the log category in a preset log category list; log information corresponding to the log category is collected from the specified cluster.
In a preferred embodiment of the present invention, the landing module 430 may also be configured to:
adding field information of a target landing component in a landing component command specified in a configuration file of a log; determining basic configuration information required by the target landing component to realize log landing according to the type of the target landing component; and adding basic configuration information of the target landing component in the configuration file to provide an access interface of the target landing component, writing the log into the target landing component, and finishing log landing.
In a preferred embodiment of the present invention, as shown in fig. 5, the apparatus may further include:
the restart module 440 is configured to restart the application corresponding to the log category.
In a preferred embodiment of the present invention, the log access request further carries a landing path for writing the log into the target landing component;
as shown in fig. 5, the apparatus may further include:
the information sending module 450 is configured to return a message that the log is successful in landing to the service, and the service reads the log based on the landing path.
The embodiment of the invention provides a log landing method and a log landing device, which are characterized in that after a log access request of a service is received, a log category which is required to be accessed by the service and a target landing component of the log are obtained by analyzing the log access request, and when the fact that the log corresponding to the log category belongs to a consumed log is determined, the configuration information of the target landing component can be directly added into a configuration file of the log, so that the log is written into the target landing component to finish log landing. According to the log landing method provided by the embodiment of the invention, when the consumed log is landed to the new landing component, the landing of the log can be realized only by adding the configuration information of the target landing component which needs to be landed in the configuration file of the log, repeated consumption of the same log can be avoided, the log can be landed to the component quickly and effectively for the service, and the service access efficiency is further improved. In addition, each landing component in the embodiment of the invention can be configured in a centralized way, when a new service has different landing requests for the same log, the access service can be provided for the service efficiently and quickly, and configuration, namely online, is achieved by adding configuration information in the configuration file of the log, so that the access efficiency is greatly improved.
Furthermore, the embodiment of the invention can consume one log to fall to a plurality of systems, avoid repeated consumption, greatly save resources, effectively promote the degree of expandability and system tolerance error, and has high availability and expandability.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in a log-to-ground device according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
The embodiment of the invention also provides electronic equipment, which comprises: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the log-landing method according to any one of the preceding claims.
An embodiment of the present invention provides a computer-readable storage medium, in which the computer-readable storage medium stores one or more programs, which when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the log-landing method according to any one of the above.
For example, FIG. 6 illustrates a computing device that may implement the log-to-ground method. The computing device conventionally includes a computer program product or computer-readable medium in the form of a processor 610 and a memory 620. The memory 620 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. The memory 620 has a memory space 630 storing program code 631 for performing any of the method steps described above. For example, the memory space 630 storing program codes may include respective program codes 631 for realizing the respective steps in the above method, respectively. The program code can be read from or written to one or more computer program products. These computer program products comprise a program code carrier such as a hard disk, a Compact Disc (CD), a memory card or a floppy disk. Such a computer program product is typically a portable or fixed storage unit as shown for example in fig. 7. The storage unit may have memory segments, memory spaces, etc. arranged similarly to the memory 620 in the computing device of fig. 6. The program code may be compressed, for example, in a suitable form. In general, the storage unit comprises computer readable code 631', i.e. code readable by a processor such as 610, for performing the method steps of the invention, which when run by a computing device causes the computing device to perform the steps in the method described above.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.
By now it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been shown and described herein in detail, many other variations or modifications of the invention consistent with the principles of the invention may be directly ascertained or inferred from the present disclosure without departing from the spirit and scope of the invention. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.
Claims (15)
1. A log landing method, comprising:
receiving a log access request of a service, wherein the log access request carries a log category required to be accessed and at least one target landing component;
analyzing the log access request, and determining that the log of the service access belongs to a consumed log according to the log category;
adding configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, and writing the log into the target landing component to finish log landing;
after the log access request is parsed, the method further comprises:
acquiring a log category carried in the log access request, and judging whether the log category exists in a preset log category list or not;
if not, determining that the log accessed by the service belongs to an unconsumed log;
and adding the log category to the preset log category list, and consuming the log from the appointed cluster.
2. The method of claim 1, wherein the obtaining the log category carried in the log access request, and after determining whether the log category exists in a preset log category list, further comprises:
if yes, determining that the log of the service access belongs to the consumed log.
3. The method of claim 1, wherein the adding the log category to the preset log category list and consuming logs from a specified cluster comprises:
adding field information corresponding to the log category in the preset log category list;
and collecting log information corresponding to the log category from the appointed cluster.
4. The method of claim 1, wherein the adding the configuration information of the target landing component in the configuration file of the log according to the type of the target landing component, writing the log into the target landing component, and completing log landing includes:
adding field information of the target landing component in a landing component command specified in a configuration file of the log;
determining basic configuration information required by the target landing component to realize log landing according to the type of the target landing component;
and adding basic configuration information of the target landing component in the configuration file to provide an access interface of the target landing component, and writing the log into the target landing component to finish log landing.
5. The method of claim 1, wherein the adding configuration information of the target floor component in the configuration file according to the type of the target floor component further comprises:
restarting the application corresponding to the log category.
6. The method of any of claims 1-5, wherein the log access request further carries a floor path for writing the log to the target floor component;
the step of adding the configuration information of the target landing component in the configuration file of the log according to the type of the target landing component, writing the log into the target landing component, and after finishing log landing, further comprises:
and returning a message of successful log landing to the service, and reading the log by the service based on the landing path.
7. The method of any one of claims 1-5, wherein the method is applied to a Kafka system.
8. A log landing device, comprising:
the request receiving module is configured to receive a log access request of a service, wherein the log access request carries a log type required to be accessed and at least one target landing component;
the analysis module is configured to analyze the log access request and determine that the log accessed by the service belongs to the consumed log according to the log category;
the landing module is configured to add configuration information of the target landing component in a configuration file of the log according to the type of the target landing component, write the log into the target landing component and finish log landing;
wherein, the parsing module includes:
the judging unit is configured to analyze the log access request, acquire the log category carried in the log access request and judge whether the log category exists in a preset log category list;
a second determining unit configured to determine that the log of the service access belongs to an unconsumed log when the log class does not exist in a preset log class list; and adding the log category to the preset log category list, and consuming the log from the appointed cluster.
9. The apparatus of claim 8, wherein the parsing module comprises:
and the first determining unit is configured to determine that the log accessed by the service belongs to the consumed log when the log class exists in a preset log class list.
10. The apparatus of claim 8, wherein the second determination unit is further configured to:
adding field information corresponding to the log category in the preset log category list; and collecting log information corresponding to the log category from the appointed cluster.
11. The apparatus of claim 8, wherein the landing module is further configured to:
adding field information of the target landing component in a landing component command specified in a configuration file of the log;
determining basic configuration information required by the target landing component to realize log landing according to the type of the target landing component;
and adding basic configuration information of the target landing component in the configuration file to provide an access interface of the target landing component, and writing the log into the target landing component to finish log landing.
12. The apparatus of claim 8, further comprising: and the restarting module is configured to restart the application corresponding to the log category.
13. The apparatus of any of claims 8-12, wherein the log access request further carries a floor path for writing the log to the target floor component;
the apparatus further comprises: and the information sending module is configured to return a message of successful log landing to the service, and the service reads the log based on the landing path.
14. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the log-landing method of any of claims 1-7.
15. A computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs as the log landing method of any one of 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711489249.3A CN110019045B (en) | 2017-12-29 | 2017-12-29 | Log floor method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711489249.3A CN110019045B (en) | 2017-12-29 | 2017-12-29 | Log floor method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019045A CN110019045A (en) | 2019-07-16 |
| CN110019045B true CN110019045B (en) | 2024-04-16 |
Family
ID=67187196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711489249.3A Active CN110019045B (en) | 2017-12-29 | 2017-12-29 | Log floor method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019045B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928906A (en) * | 2019-11-08 | 2020-03-27 | 杭州安恒信息技术股份有限公司 | Method for writing carbon data only once based on flink |
| CN111240858B (en) * | 2020-01-07 | 2024-04-26 | 中国建设银行股份有限公司 | Event scheduling method and component |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104243532A (en) * | 2013-06-21 | 2014-12-24 | 鸿富锦精密工业（深圳）有限公司 | Data access method and system |
| CN104281515A (en) * | 2013-07-09 | 2015-01-14 | 腾讯科技（深圳）有限公司 | Adjustment method and device for log data |
| CN105260293A (en) * | 2015-11-06 | 2016-01-20 | Tcl集团股份有限公司 | Output method, output device and terminal equipment of log information |
| CN105830107A (en) * | 2013-12-19 | 2016-08-03 | 维萨国际服务协会 | Cloud-based transactions methods and systems |
| CN106021315A (en) * | 2016-05-05 | 2016-10-12 | 智者四海（北京）技术有限公司 | Log management method and system for application program |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106874160A (en) * | 2017-01-23 | 2017-06-20 | 上海斐讯数据通信技术有限公司 | Log server and its management method |
| CN107025163A (en) * | 2017-03-31 | 2017-08-08 | 北京商办通互联网科技有限公司 | A kind of passive type production, the demand realization method and system of consumer computers daily record |
-
2017
- 2017-12-29 CN CN201711489249.3A patent/CN110019045B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104243532A (en) * | 2013-06-21 | 2014-12-24 | 鸿富锦精密工业（深圳）有限公司 | Data access method and system |
| CN104281515A (en) * | 2013-07-09 | 2015-01-14 | 腾讯科技（深圳）有限公司 | Adjustment method and device for log data |
| CN105830107A (en) * | 2013-12-19 | 2016-08-03 | 维萨国际服务协会 | Cloud-based transactions methods and systems |
| CN105260293A (en) * | 2015-11-06 | 2016-01-20 | Tcl集团股份有限公司 | Output method, output device and terminal equipment of log information |
| CN106021315A (en) * | 2016-05-05 | 2016-10-12 | 智者四海（北京）技术有限公司 | Log management method and system for application program |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106874160A (en) * | 2017-01-23 | 2017-06-20 | 上海斐讯数据通信技术有限公司 | Log server and its management method |
| CN107025163A (en) * | 2017-03-31 | 2017-08-08 | 北京商办通互联网科技有限公司 | A kind of passive type production, the demand realization method and system of consumer computers daily record |
Non-Patent Citations (2)
| Title |
|---|
| RTA:Real time actionable events detection as a service;YunJIeQiu 等;《2016 IEEE international conference on web services》;第1-4页 * |
| 大数据环境下的分布式数据流处理关键技术探析;陈付梅 等;《计算机应用》;第37卷(第3期);第620-627页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019045A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104537076B (en) | A kind of file read/write method and device | |
| US9590880B2 (en) | Dynamic collection analysis and reporting of telemetry data | |
| US7624118B2 (en) | Data processing over very large databases | |
| US8140591B2 (en) | Enabling workflow awareness within a business process management (BPM) system | |
| CN112506743B (en) | Log monitoring method, device and server | |
| CN111209258A (en) | Tax end system log real-time analysis method, equipment, medium and system | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN111177237B (en) | Data processing system, method and device | |
| CN102880503A (en) | Data analysis system and data analysis method | |
| US9971563B2 (en) | Systems and methods for low interference logging and diagnostics | |
| CN106713396B (en) | Server scheduling method and system | |
| CN111241122B (en) | Task monitoring method, device, electronic equipment and readable storage medium | |
| CN109190025B (en) | Information monitoring method, device, system and computer readable storage medium | |
| CN110019045B (en) | Log floor method and device | |
| CN114090378A (en) | Custom monitoring and alarming method based on Kapacitor | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN113297245A (en) | Method and device for acquiring execution information | |
| CN110033242B (en) | Working time determining method, device, equipment and medium | |
| CN111796993A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US10496467B1 (en) | Monitoring software computations of arbitrary length and duration | |
| CN114218303A (en) | Transaction data processing system, processing method, medium and equipment | |
| CN117435367B (en) | User behavior processing method, device, equipment, storage medium and program product | |
| CN112732165B (en) | Offset management method, offset management device and storage medium | |
| CN118170728A (en) | File merging method and device, electronic equipment and storage medium | |
| CN107943942B (en) | Event-driven multi-turn file acquisition system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |