CN111913922B - Binary structured log generation method, device, equipment and storage medium - Google Patents
Binary structured log generation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111913922B CN111913922B CN201910388560.1A CN201910388560A CN111913922B CN 111913922 B CN111913922 B CN 111913922B CN 201910388560 A CN201910388560 A CN 201910388560A CN 111913922 B CN111913922 B CN 111913922B
- Authority
- CN
- China
- Prior art keywords
- log
- matched
- standard
- application request
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012545 processing Methods 0.000 claims abstract description 49
- 230000006399 behavior Effects 0.000 claims description 54
- 238000004590 computer program Methods 0.000 claims description 4
- 238000012958 reprocessing Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 17
- 238000004458 analytical method Methods 0.000 abstract description 14
- 230000005540 biological transmission Effects 0.000 abstract description 13
- 230000006870 function Effects 0.000 description 27
- 238000005457 optimization Methods 0.000 description 23
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003993 interaction Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a method, a device, equipment and a storage medium for generating a binary structured log, wherein the method comprises the following steps: acquiring an application request matched with user behavior, wherein the application request comprises: behavior data in the form of key-value pairs; according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; and processing the structured log into a binary structured log, and writing the binary structured log into a file. According to the technical scheme, the behavior data in the application request is processed by using the standard structure body model, so that the structured log is generated, no intermediate file is generated, storage resources are saved, a complex log analysis process is avoided, meanwhile, the binary structured log is obtained through the serialization operation, the log volume is greatly reduced, the log is convenient to transmit, and the resource consumption in the data transmission process is reduced.
Description
Technical Field
The embodiment of the invention relates to the field of Internet, in particular to a method, a device, equipment and a storage medium for generating a binary structured log.
Background
With the continuous development of internet technology, the application population and the service types of the internet are continuously growing, and the running of various services enables the internet to generate a large number of logs every day, and the logs are important components of internet big data analysis, so that the method has extremely high mining and analysis values.
Processing these huge amounts of logs first requires log parsing, i.e., extracting meaningful fields to convert unstructured data into structured data. Typically, all logs are printed into text files, which are then parsed by writing a log parser.
However, the logs with different formats need to write different types of log analysis programs, and the analysis programs with the downstream are also required to change by changing the log formats, so that the analysis cost is high, the analysis process is complex, and errors are easy to occur; in addition, since the log data amount is excessively large, a large amount of resources are required to be consumed in the storing and transmitting process, and thus the storing and transmitting costs are extremely high.
Disclosure of Invention
The embodiment of the invention provides a method, a device, computer equipment and a storage medium for generating a binary structured log. The method and the system realize the direct generation of the structured log according to the user application request, avoid the complex log analysis process and save the resource consumption of log data in the storage and transmission processes.
In a first aspect, an embodiment of the present invention provides a method for generating a binary structured log, including:
acquiring an application request matched with user behavior, wherein the application request comprises: behavior data in the form of key-value pairs;
according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
and processing the structured log into a binary structured log, writing the binary structured log into a file, and generating a binary structured file.
In a second aspect, an embodiment of the present invention provides a device for generating a binary structured log, including:
an application request acquisition module, configured to acquire an application request matched with a user behavior, where the application request includes: behavior data in the form of key-value pairs;
the structured log acquisition module is used for processing the behavior data in the form of the key value pair in the application request according to a standard structure model matched with the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
and the binary structured log acquisition module is used for processing the structured log into a binary structured log, writing the binary structured log into a file and generating a binary structured file.
In a third aspect, an embodiment of the present invention provides a computer apparatus, including: one or more processors;
a storage means for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for generating a binary structured log according to any embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, where a computer program is stored, where the program when executed by a processor implements a method for generating a binary structured log according to any embodiment of the present invention.
According to the technical scheme provided by the embodiment of the invention, the behavior data in the application request is processed by using the standard structure model, so that the structured log is directly generated, no intermediate file is generated, the storage resource is saved, the complex log analysis process is avoided, and the analysis cost is saved. Meanwhile, the binary structured log is obtained through the serialization operation, so that the computer is convenient to read, the log volume is reduced, log transmission is convenient, the data transmission efficiency is improved, and the resource consumption in the data transmission process is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for generating a binary structured log according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a method for generating a binary structured log according to a second embodiment of the present invention;
FIG. 3 is a block diagram of a binary structured log generating device according to a third embodiment of the present invention;
fig. 4 is a block diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
It should be further noted that, for convenience of description, only some, but not all of the matters related to the present invention are shown in the accompanying drawings. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently, or at the same time. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example 1
Fig. 1 is a flowchart of a method for generating a binary structured log according to an embodiment of the present invention, where the embodiment is applicable to a case where the structured log is directly generated according to a user application request and no intermediate file is generated, and the method may be performed by a device for generating a binary structured log according to the embodiment of the present invention, where the device may be implemented in a software and/or hardware manner, and may be generally integrated on a server. As shown in fig. 1, the method specifically includes the following operations:
s110, acquiring an application request matched with user behaviors, wherein the application request comprises: behavior data in the form of key-value pairs.
In general, when using an application program (APP), a client user transmits various application requests, such as a route search request, a navigation request, or a purchase request, to a server. The application request is typically sent to the server in the form of an HTTP request that includes behavior data in the form of various key-value pairs, so the HTTP request is typically printed directly to a file by the server and the file is stored as log data.
The user behavior includes at least one behavior data relating to one or more of a plurality of elements of time, place, character, interaction, and interaction content; for example, the user has searched for a good, and then the time of the search activity includes the time at which the search occurred and/or the page dwell time; the location includes the geographic location where the user is located, the network address, and/or the click location in the page; the persona includes personal information of the user, such as account information logged in or registered at the time of search, and also includes terminal equipment, platform, channel, and/or network information used, such as a terminal equipment name and/or IP address used at the time of search; interaction, namely the type of the operation executed at this time is search; interactive content, namely search content and searched related information.
The application request matches the user behavior, e.g., the user issues a search request, then the search behavior of the user is represented; the user issues a purchase request representing the user's purchase behavior. The application request may be triggered by clicking a function key, for example, clicking a "search" key or a "purchase" key to trigger a search request or a purchase request; the browsing request of the user can also be triggered by clicking on the text or the picture, for example, clicking on the text or the picture to view the detail; the payment request can also be triggered by a biometric feature of the terminal device, for example by fingerprint recognition or facial recognition of the mobile terminal; in the embodiment of the present invention, optionally, a triggering manner of the application request is not specifically limited. The application request may be sent by an application client installed in the intelligent terminal, or may be sent by a page accessed by a third-party browser.
The application request includes: behavior data in the form of key-value pairs. The key value pair is a data storage mode and comprises a key name and a key value, wherein the key name represents the name of the behavior data, and the key value represents the value given by the behavior data, namely the given specific content; for example, a key value pair is "amount=50 yuan", then "amount" is the key name of the behavior data, and "50 yuan" represents the value of the behavior data, i.e., the specific content assigned.
S120, according to a standard structure model matched with the application request, processing behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name, and an area to be filled that matches the standard field name.
In the prior art, a server takes an application request as log data, prints and generates a text log by using a text log generation tool, such as log4j, glog and the like, analyzes the text log by writing a log analysis program, and performs structural processing on the text log. Compared with the prior art, the method and the device have the advantages that the standard structure model is utilized to directly generate the structured log for the application request of the user, the step of printing the application request to generate a file is not needed, any intermediate file is not generated, the application request is directly stored as the structured log, storage resources are saved, and meanwhile, the complex log analysis process is avoided.
The standard structure model is established according to various information included in the application request, and different application requests correspond to different standard structure models. The standard structure model may be manually established by a worker, for example, for a purchase request of a user, in which a plurality of information such as a user's name, contact, commodity name, time of order, and address are included, so that the standard structure model including the above information may be previously established for processing the purchase request; the standard structure model may also be automatically created according to an application request, for example, when the obtained application request does not find a corresponding standard structure model, the standard structure model is created according to the application request, and the key name of the key value pair included in the application request is used as the standard field name of the standard structure model, and the area to be filled is set for each standard field name. In the embodiment of the present invention, optionally, the building manner of the standard structural body model is not specifically limited.
The standard structure model includes at least one standard field name, and an area to be filled that matches the standard field name. Optionally, in the embodiment of the present invention, a standard field name is sequentially acquired as a target field name in the standard structural body model; if the behavior data has the target key name matched with the target field name, filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model; and returning to execute the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed. And if the behavior data does not have the target field value matched with the target field name, discarding filling processing of the region to be filled corresponding to the target field name in the standard structure model.
For example, standard field names for a standard structural model include: "amount", "time", "commodity name" and "account type", and a region to be filled that matches each standard field name; the behavior data in the form of key value pairs in the application request includes: "amount=10 elements", "time=5 points", "order number=208A", "commodity name=clothing", and "account type=member"; according to the filling rule, the standard structure model sequentially acquires the sum, the time, the commodity name and the account type as target field names, fills the corresponding key value into the area to be filled according to the behavior data in the form of the key value pair in the application request, and finally comprises the following steps: "amount=10 elements", "time=5 points", "commodity name=clothing", and "account type=member".
Optionally, in the embodiment of the present invention, a set function may be used to fill a key value corresponding to the target key name in a region to be filled corresponding to the target field name in the standard structure model. The set function is an associative container for storing data of the same data type and for retrieving data from the same data set. In particular, the value of each member of the set function is unique and cannot be modified.
S130, processing the structured log into a binary structured log, and writing the binary structured log into a file to generate a binary structured file.
In an embodiment of the present invention, optionally, the structured log is processed into a binary structured log by using a serialization processing tool, where the serialization processing tool includes Protobuf. Serialization (Serialization) is a process of converting state information of a target object into a storable and/or transmittable form to facilitate storage and/or transmission of data, and in an embodiment of the present invention, the form is in a binary format. Protobuf is a protocol for serializing structured data, has the characteristics of flexibility and high efficiency, has smaller code stream after Protobuf serialization, has higher speed, and is suitable for executing massive serialization operation.
Because the binary system is widely applied to computers, the structured log is processed into the binary structured log, so that the computer is convenient to read, the volume of the log is reduced, the resource consumption in the transmission process is reduced, and the efficiency of the subsequent large data processing is improved.
Further, after the binary structured file is generated, the binary structured file may be stored locally, or may be sent to a dedicated log storage platform for log storage.
In the embodiment of the invention, if the binary structured file is stored locally, the viewing authority can be correspondingly opened to the user with the requirement of log viewing, and before the user views a certain log, the binary structured log needs to be converted into a data format which is convenient for the user to read.
Optionally, after the structured log is processed into a binary structured log and written into a file, if a viewing request for the binary structured log is detected, an anti-serialization processing tool is adopted to reprocess the binary structured log into the structured log and display the structured log. Deserialization (Deserialization) is a process of converting a storable or transmissible form such as a binary format into an object, and in the embodiment of the present invention, the process of converting into a structured log is used to display the actual form of the object.
According to the technical scheme provided by the embodiment of the invention, the behavior data in the application request is processed by using the standard structure model, so that the structured log is directly generated, no intermediate file is generated, the storage resource is saved, the complex log analysis process is avoided, and the analysis cost is saved. Meanwhile, the binary structured log is obtained through the serialization operation, so that the computer is convenient to read, the log volume is reduced, log transmission is convenient, the data transmission efficiency is improved, and the resource consumption in the data transmission process is reduced.
Example two
Fig. 2 is a flowchart of a method for generating a binary structured log according to a second embodiment of the present invention, where the embodiment is based on the foregoing embodiment, and the application request further includes: request type. Correspondingly, the method of the embodiment specifically comprises the following operations:
s210, acquiring an application request matched with user behaviors, wherein the application request comprises: behavior data in the form of key-value pairs and request types.
The application requests are divided into different request types according to different functions, and a plurality of application requests can correspond to one request type. For example, clicking on different text or pictures to view the details, the application requests are different, but the request types are the same, and the request types are all browsing requests.
S220, acquiring a log type matched with the request type, and acquiring a standard structure model matched with the application request from a plurality of alternative standard structure models according to the log type.
One request type corresponds to one log type, for example, the above-mentioned browsing request corresponds to a browsing log. And one log type corresponds to a plurality of alternative standard structure models, and the standard structure model matched with the application request is acquired from the alternative standard structure models. For example, the standard structure model corresponding to the browsing log includes a standard structure model for processing a picture browsing request, and further includes a standard structure model for processing a text browsing request, and the corresponding standard structure model is acquired according to a specific application request.
S230, according to a standard structure model matched with the application request, processing behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name, and an area to be filled that matches the standard field name.
S240, processing the structured log into a binary structured log, and writing the binary structured log into a file to generate a binary structured file.
Optionally, the method for generating the binary structured log in the embodiment of the present invention further includes: receiving at least one standard structure model uploaded by at least one log data generation platform, and generating a model view matched with the standard structure model; the standard structure model is associated with a log type; and issuing the model view to at least one log data using platform matched with the log data generating platform so that the log data using platform can view the binary structured file corresponding to the standard structural body model by using the model view.
Obtaining a standard structure model from the log data generation platform, wherein the standard structure model is associated with the log type, so that a model view generated by the standard structure model can reflect the structure of the corresponding log type; the model view may be in the format of a table, column, view, or the like. The platform for generating the log data and the platform for using the log data are often different platforms, and when the log data using platform receives the model views issued by the matched log data generating platform, the model views can be used for viewing the binary structured files generated by the log data generating platform.
Specifically, when the staff performs function optimization on the code layer, the log format changes, the behavior data in the form of key value pairs in the application request changes, and corresponding changes are generated in the standard structure model matched with the log format and the generated structured log and binary structured log, for example, after the navigation request is sent out, the original function only provides a route generated according to the starting place and the arrival place, but after the function optimization, the display of the congestion condition is increased, at this time, the behavior data in the form of key value pairs related to the congestion condition is increased in the application request, statistics on the congestion condition is also added in the standard structure model corresponding to the application request, and further, the generated structured log and binary structured log comprise records related to the congestion condition.
When the log data use platform views the binary structured file before function optimization by using the model view after function optimization, the log data can still be normally acquired, the log format is changed but backward compatibility is ensured, namely, the model view of the version after function optimization can open the binary structured file of the version before function optimization; however, since the binary structured file is generated before the function optimization, and does not include the relevant record after the function optimization, the record is not displayed when the model view after the function optimization opens the binary structured file, for example, the record related to the congestion condition is not included in the binary structured file generated before the function optimization, and therefore, the record related to the congestion condition is not displayed when the model view after the function optimization opens the binary structured file before the function optimization.
When the log data use platform views the binary structured file after function optimization by using the model view before function optimization, the log data can be normally obtained as well, the log format is changed but forward compatibility is ensured, namely, the model view of the version before function optimization can open the binary structured file of the version after function optimization; however, since the model view is generated before the functional optimization, and does not include the relevant record after the functional optimization, opening the binary structured file by the model view before the functional optimization does not display the relevant record, for example, the record related to the congestion condition is not included in the model view generated before the functional optimization, and thus opening the binary structured file after the functional optimization by using the model view before the functional optimization does not display the record related to the congestion condition.
According to the technical scheme of the embodiment of the invention, the matched log types are acquired according to the request types of the application requests, and then the target standard structure body model is acquired from the plurality of standard structure body models matched with the log types, so that each application request only needs to search for the required standard structure body model in one log type matched with the request type, and does not need to search for the target standard structure body model in all the standard structure body models, the acquisition efficiency is improved, and the conversion speed of each application request into the structured log is further improved. Meanwhile, the log format is managed by using the model view, so that forward compatibility and backward compatibility of the log format are realized, normal use of logs before and after function optimization is ensured, and data loss is avoided.
Example III
Fig. 3 is a block diagram of a binary structured log generating device according to a third embodiment of the present invention, where the device includes: an application request acquisition module 310, a structured log acquisition module 320, and a binary structured log acquisition module 330.
An application request obtaining module 310, configured to obtain an application request matched with a user behavior, where the application request includes: behavior data in the form of key-value pairs;
the structured log obtaining module 320 is configured to process the behavior data in the form of the key value pair in the application request according to a standard structure model matched with the application request, so as to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
the binary structured log obtaining module 330 is configured to process the structured log into a binary structured log, and write the binary structured log into a file to generate a binary structured file.
According to the technical scheme provided by the embodiment of the invention, the behavior data in the application request is processed by using the standard structure model, so that the structured log is directly generated, no intermediate file is generated, the storage resource is saved, the complex log analysis process is avoided, and the analysis cost is saved. Meanwhile, the binary structured log is obtained through the serialization operation, so that the computer is convenient to read, the log volume is reduced, log transmission is convenient, the data transmission efficiency is improved, and the resource consumption in the data transmission process is reduced.
Optionally, based on the above technical solution, the structured log obtaining module 320 may specifically include:
the target field name acquisition unit is used for sequentially acquiring a standard field name as a target field name in the standard structure body model;
a key value filling unit, configured to fill, if a target key name matched with the target field name exists in the behavior data, a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model;
and the return unit is used for returning and executing the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed.
Optionally, based on the above technical solution, the structured log obtaining module 320 may further include:
and the filling discarding unit is used for discarding the filling processing of the region to be filled corresponding to the target field name in the standard structural body model if the target field value matched with the target field name does not exist in the behavior data.
Optionally, based on the above technical solution, the key value filling unit is specifically configured to:
and filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model by adopting a set function.
Optionally, based on the above technical solution, the binary structured log obtaining module 330 is specifically configured to:
and processing the structured log into a binary structured log by using a serialization processing tool.
Optionally, on the basis of the above technical solution, the generating device of the binary structured log may further include:
and the anti-serialization processing module is used for reprocessing the binary structured log into the structured log and displaying the structured log by adopting an anti-serialization processing tool when the checking request for the binary structured log is detected.
Optionally, based on the above technical solution, the application request may further include: request type.
Optionally, on the basis of the above technical solution, the generating device of the binary structured log may further include:
and the standard structure model acquisition module is used for acquiring the log type matched with the request type and acquiring the standard structure model matched with the application request from a plurality of alternative standard structure models according to the log type.
Optionally, on the basis of the above technical solution, the generating device of the binary structured log may further include:
the model view generation module is used for receiving at least one type of standard structure body model uploaded by the at least one log data generation platform and generating a model view matched with the standard structure body model; the standard structure model is associated with a log type;
and the log structure type acquisition module is used for issuing the model view to at least one log data use platform matched with the log data generation platform so that the log data use platform can view the binary structured file corresponding to the standard structure body model by utilizing the model view.
The device can execute the binary structured log generation method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment may be found in the method provided by any embodiment of the present invention.
Example IV
Fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention. Fig. 4 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in fig. 4 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in FIG. 4, the computer device 12 is in the form of a general purpose computing device. Components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, a bus 18 that connects the various system components, including the system memory 28 and the processing units 16.
Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computer device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. The computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, commonly referred to as a "hard disk drive"). Although not shown in fig. 4, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored in, for example, memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.
The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 12 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Moreover, computer device 12 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with computer device 12, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing the binary structured log generation method provided by the embodiment of the present invention: acquiring an application request matched with user behavior, wherein the application request comprises: behavior data in the form of key-value pairs; according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name; and processing the structured log into a binary structured log, writing the binary structured log into a file, and generating a binary structured file.
Example five
A fifth embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements a method for generating a binary structured log as provided in all the inventive embodiments of the present application:
acquiring an application request matched with user behavior, wherein the application request comprises: behavior data in the form of key-value pairs;
according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
and processing the structured log into a binary structured log, writing the binary structured log into a file, and generating a binary structured file.
Any combination of one or more computer readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (9)
1. A method for generating a binary structured log, comprising:
acquiring an application request matched with user behaviors; wherein the application request includes: behavior data in the form of key-value pairs;
according to a standard structure model matched with the application request, processing the behavior data in the form of the key value pair in the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
processing the structured log into a binary structured log, writing the binary structured log into a file, and generating a binary structured file;
receiving at least one standard structure model uploaded by at least one log data generation platform, and generating a model view matched with the standard structure model; wherein the standard structure model is associated with a log type;
issuing the model view to at least one log data use platform matched with the log data generation platform so that the log data use platform can view a binary structured file corresponding to the standard structure body model by utilizing the model view;
the processing the behavior data in the form of the key value pair in the application request according to the standard structure model matched with the application request to obtain a structured log matched with the application request comprises the following steps:
sequentially acquiring a standard field name from the standard structure body model as a target field name;
if the behavior data has the target key name matched with the target field name, filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model;
and returning to execute the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed.
2. The method according to claim 1, further comprising, after sequentially acquiring one standard field name as a target field name in the standard structure model:
and if the behavior data does not have the target field value matched with the target field name, discarding filling processing of the region to be filled corresponding to the target field name in the standard structural body model.
3. The method according to claim 1, wherein the filling the key value corresponding to the target key name into the region to be filled corresponding to the target field name in the standard structure model includes:
and filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure model by adopting a set function.
4. The method of claim 1, wherein the processing the structured log into a binary structured log comprises:
and processing the structured log into the binary structured log by using a serialization processing tool.
5. The method of claim 1, wherein after said processing the structured log into a binary structured log and writing to a file, generating a binary structured file, comprising:
and when detecting a viewing request for the binary structured log, reprocessing the binary structured log into the structured log by adopting a deserializing processing tool and displaying the structured log.
6. The method of claim 1, wherein the application request further comprises: a request type;
before the behavior data in the form of the key value pair in the application request is processed according to the standard structure model matched with the application request to obtain the structured log matched with the application request, the method further comprises the following steps:
and acquiring a log type matched with the request type, and acquiring a standard structural body model matched with the application request from a plurality of alternative standard structural body models according to the log type.
7. A binary structured log generation apparatus, comprising:
the application request acquisition module is used for acquiring application requests matched with user behaviors; wherein the application request includes: behavior data in the form of key-value pairs;
the structured log acquisition module is used for processing the behavior data in the form of the key value pair in the application request according to a standard structure model matched with the application request to obtain a structured log matched with the application request; wherein the standard structure model comprises: at least one standard field name and a region to be filled matched with the standard field name;
the binary structured log acquisition module is used for processing the structured log into a binary structured log, writing the binary structured log into a file and generating a binary structured file;
the model view generation module is used for receiving at least one type of standard structure body model uploaded by the at least one log data generation platform and generating a model view matched with the standard structure body model; wherein the standard structure model is associated with a log type;
the log structure type acquisition module is used for issuing the model view to at least one log data use platform matched with the log data generation platform so that the log data use platform can view binary structured files corresponding to the standard structure body model by utilizing the model view;
wherein, the structured log obtaining module includes: the system comprises a target field name acquisition unit, a key value filling unit and a return unit;
the target field name acquisition unit is used for sequentially acquiring a standard field name in the standard structure body model as a target field name;
the key value filling unit is used for filling a key value corresponding to the target key name into a region to be filled corresponding to the target field name in the standard structure body model if the target key name matched with the target field name exists in the behavior data;
and the return unit is used for returning and executing the operation of sequentially acquiring one standard field name from the standard structure body model as a target field name until the processing of all the standard field names in the standard structure body model is completed.
8. A computer device, the computer device comprising:
one or more processors;
a storage means for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of generating a binary structured log as recited in any of claims 1-6.
9. A computer storage medium having stored thereon a computer program, which when executed by a processor implements a method of generating a binary structured log according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910388560.1A CN111913922B (en) | 2019-05-10 | 2019-05-10 | Binary structured log generation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910388560.1A CN111913922B (en) | 2019-05-10 | 2019-05-10 | Binary structured log generation method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111913922A CN111913922A (en) | 2020-11-10 |
| CN111913922B true CN111913922B (en) | 2024-02-20 |
Family
ID=73242675
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910388560.1A Active CN111913922B (en) | 2019-05-10 | 2019-05-10 | Binary structured log generation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111913922B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559458B (en) * | 2020-12-11 | 2021-11-30 | 深圳金信诺高新技术股份有限公司 | Method and system for reducing log size |
| CN113844505A (en) * | 2021-08-31 | 2021-12-28 | 通号城市轨道交通技术有限公司 | Train log processing method and device |
| CN113742408B (en) * | 2021-09-03 | 2024-02-02 | 唯品会(广州)软件有限公司 | Protobuf protocol dynamic analysis-based data interaction method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106371953A (en) * | 2015-07-22 | 2017-02-01 | 奥普塔姆软件股份有限公司 | Compact binary event log generation |
| CN108038018A (en) * | 2017-12-22 | 2018-05-15 | 杭州闪捷信息科技有限公司 | Expansible daily record data storage method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9703675B2 (en) * | 2014-08-11 | 2017-07-11 | Microsoft Technology Licensing, Llc | Structured logging and instrumentation framework |
-
2019
- 2019-05-10 CN CN201910388560.1A patent/CN111913922B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106371953A (en) * | 2015-07-22 | 2017-02-01 | 奥普塔姆软件股份有限公司 | Compact binary event log generation |
| CN108038018A (en) * | 2017-12-22 | 2018-05-15 | 杭州闪捷信息科技有限公司 | Expansible daily record data storage method and device |
Non-Patent Citations (2)
| Title |
|---|
| Least squares adaptive algorithms suitable for multiplierless LMMSE detection in 3/sup rd/ generation mobile systems;C. Litchfield等;《IEEE Xplore》;全文 * |
| 基于Web Log的数据预处理研究;向坚持, 陈晓红, 刘相滨, 徐选华;湖南师范大学自然科学学报(04);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111913922A (en) | 2020-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111913922B (en) | Binary structured log generation method, device, equipment and storage medium | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| CN109471851B (en) | Data processing method, device, server and storage medium | |
| CN111815421B (en) | Tax policy processing method and device, terminal equipment and storage medium | |
| CN110287146B (en) | Method, device and computer storage medium for downloading application | |
| CN112231407B (en) | DDL synchronization method, device, equipment and medium of PostgreSQL database | |
| CN111291882A (en) | Model conversion method, device, equipment and computer storage medium | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN110191021A (en) | A kind of protocol testing method, device, electronic equipment and storage medium | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| CN112417016B (en) | Data exchange method, system, equipment and storage medium | |
| CN110955640A (en) | Cross-system data file processing method, device, server and storage medium | |
| CN113158101A (en) | Visual page rendering method, device, equipment and storage medium | |
| WO2019071907A1 (en) | Method for identifying help information based on operation page, and application server | |
| CN112612546A (en) | Page loading method and device, electronic equipment and storage medium | |
| CN113761412A (en) | Application page display method and device, electronic equipment, medium and application system | |
| CN112507259A (en) | Webpage loading method and device, electronic equipment and storage medium | |
| CN113836014A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN113609195A (en) | Report generation method, report generation device, electronic equipment and storage medium | |
| CN113792138B (en) | Report generation method and device, electronic equipment and storage medium | |
| CN110795651A (en) | Method and device for saving front-end data and electronic equipment | |
| CN113760894A (en) | Data calling method and device, electronic equipment and storage medium | |
| CN113419711A (en) | Page guiding method and device, electronic equipment and storage medium | |
| CN112581568A (en) | Dynamic poster generation method, device, server and storage medium | |
| CN113448649B (en) | Redis-based home page data loading server and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |