CN112069048A - Log processing method, device and storage medium - Google Patents
Log processing method, device and storage medium Download PDFInfo
- Publication number
- CN112069048A CN112069048A CN202010943762.0A CN202010943762A CN112069048A CN 112069048 A CN112069048 A CN 112069048A CN 202010943762 A CN202010943762 A CN 202010943762A CN 112069048 A CN112069048 A CN 112069048A
- Authority
- CN
- China
- Prior art keywords
- information
- log
- single label
- user identity
- label information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 54
- 238000000034 method Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000009194 climbing Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 241000411851 herbal medicine Species 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The embodiment of the invention provides a log processing method, a log processing device and a storage medium, which relate to the technical field of data processing, and the log processing method provided by the embodiment of the invention is characterized in that original logs are obtained and are split and recombined according to preset separators to generate a plurality of single label information; the single label information is inquired according to preset dimension information to obtain an inquiry result, the inquiry result comprises the number of the single label information corresponding to the dimension information, the original log is split and recombined to form a plurality of single label information, a plurality of one-to-many data types are converted into a plurality of one-to-one data types, the data types are simplified, when the user log is specifically analyzed, inquiry and sorting can be directly carried out according to the set dimension, and the efficiency of data processing and analysis is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a log processing method, device and storage medium.
Background
With the rapid development of information technology and communication technology, it is necessary to widely apply and popularize various application software and collect user data logs and analyze user behaviors.
Aiming at different application software and systems, the solution at the present stage is only to collect basic information of a user, such as generating logs by simple use records, access amount of the application software, access times, interests and hobbies, and performing simple analysis on user logs of a mobile phone, but the analysis processing method facing mass user log data in the prior art is complicated, a plurality of data are stored in a cross-aliasing manner, and the processing efficiency is low when a large amount of data exist.
Disclosure of Invention
The invention aims to provide a log processing method, a log processing device and a log processing storage medium, so that statistical analysis of data can be quickly realized on user data of a mobile phone, and the processing efficiency is improved.
The technical scheme adopted by the invention is as follows
In a first aspect, an embodiment of the present invention provides a log processing method, where the method includes:
acquiring an original log, wherein the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels;
splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information comprises a corresponding relation between the user identity information and a type label;
and querying the single label information according to preset dimension information to obtain a query result, wherein the query result comprises the number of the single label information corresponding to the dimension information.
In an optional embodiment, the step of splitting and reassembling the original log according to a preset delimiter to generate a plurality of single label information includes:
preliminarily dividing the original log according to the user identity information to obtain an intermediate log, wherein the intermediate log comprises a corresponding relation between the user identity information and a plurality of types of tags;
dividing the intermediate log according to preset separators to obtain user identity information and a plurality of type labels;
and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
In an optional embodiment, after the step of querying the plurality of single tag information according to the preset dimension information to obtain the query result, the method further includes:
and deleting the intermediate log and the single label information.
In an optional embodiment, the querying the single tag information according to the preset dimension information to obtain a query result includes:
reading a first preset number of data blocks, wherein each data block comprises a second preset number of single label information;
inquiring according to the single label information in each data block according to a preset dimension respectively to generate a preliminary result, wherein the preliminary result comprises the number of the single label information corresponding to the preset dimension inquired in the data block;
and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain the query result.
In an optional embodiment, before querying a plurality of single tag information according to preset dimension information, the log processing method further includes:
and acquiring a query instruction, wherein the query instruction comprises preset dimension information, and the dimension information is at least one of the plurality of label information.
In a second aspect, an embodiment of the present invention provides a log processing apparatus, where the log processing apparatus is configured to implement the log processing method according to any one of the foregoing embodiments, and the log processing apparatus includes:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring an original log, and the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels;
the splitting module is used for splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information comprises a corresponding relation between the user identity information and a type label;
and the processing module is used for inquiring the single label information according to preset dimension information to obtain an inquiry result, wherein the inquiry result comprises the number of the single label information corresponding to the dimension information.
In an optional embodiment, the splitting module is configured to perform preliminary segmentation on the original log according to the user identity information to obtain an intermediate log, where the intermediate log includes a correspondence between one piece of user identity information and multiple types of tags; dividing the intermediate log according to preset separators to obtain user identity information and a plurality of type labels; and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
In an optional embodiment, the splitting module is further configured to delete the intermediate log and the single label information after the step of querying the plurality of single label information according to preset dimension information to obtain a query result.
In an optional embodiment, the processing module is configured to query the single tag information in each data block according to a preset dimension, and generate a preliminary result, where the preliminary result includes the number of the single tag information corresponding to the preset dimension queried in the data block; and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain the query result.
In a third aspect, an embodiment of the present invention provides a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a log processing method as described in any one of the foregoing embodiments.
Compared with the prior art, the application provides a log processing method, a log processing device and a storage medium, wherein the log processing method comprises the following steps: acquiring an original log, wherein the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels; splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information comprises a corresponding relation between the user identity information and a type label; and querying the single label information according to preset dimension information to obtain a query result, wherein the query result comprises the number of the single label information corresponding to the dimension information. The original log is cut to generate a plurality of single label information, so that the analysis processing result of the log can be quickly generated by inquiring from any dimension, and the analysis processing efficiency of the log is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic view of an electronic device provided in this embodiment;
fig. 2 is a schematic flowchart of a log processing method provided in this embodiment;
fig. 3 is a schematic flowchart of another log processing method provided in this embodiment;
fig. 4 is a schematic flowchart of another log processing method provided in this embodiment;
fig. 5 is a schematic flowchart of another log processing method provided in this embodiment;
FIG. 6 is a schematic diagram of preliminary result fusion provided in the present embodiment;
fig. 7 is a schematic flowchart of another log processing method provided in this embodiment;
fig. 8 is a schematic functional block diagram of a log processing apparatus according to this embodiment.
Description of reference numerals: 210-a processor; 211-a memory; 212-a bus; 213-a communication interface; 300-log processing means; 310-an acquisition module; 320-splitting module; 330-processing module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. indicate an orientation or a positional relationship based on that shown in the drawings or that the product of the present invention is used as it is, this is only for convenience of description and simplification of the description, and it does not indicate or imply that the device or the element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present invention.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
With the rapid development of information technology and communication technology, it is necessary to widely apply and popularize various application software and collect user data logs and analyze user behaviors. Aiming at different application software and systems, the solution at the present stage is only to collect basic information of a user, such as generating logs by simple use records, access amount of the application software, access times, interests and hobbies, and performing simple analysis on user logs of a mobile phone, but the analysis processing method facing mass user log data in the prior art is complicated, a plurality of data are stored in a cross-aliasing manner, and the processing efficiency is low when a large amount of data exist.
In order to solve the above problem, an embodiment of the present application provides an electronic device, please refer to fig. 1, where fig. 1 illustrates a schematic structural diagram of the electronic device provided in the embodiment. The electronic device includes a processor 210, a memory 211, and a bus 212. The processor 210 and the memory 211 are connected by a bus 212, and the processor 210 is configured to execute an executable module, such as a computer program, stored in the memory 211.
The processor 210 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the log processing method provided by this embodiment may be implemented by an integrated logic circuit of hardware in the processor 210 or instructions in the form of software. The Processor 210 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
The Memory 211 may comprise a Random Access Memory (RAM) and may further comprise a non-volatile Memory (non-volatile Memory), such as at least one disk Memory.
The bus 212 may be an ISA (Industry Standard architecture) bus, a PCI (peripheral Component interconnect) bus, an EISA (extended Industry Standard architecture) bus, or the like. Only one bi-directional arrow is shown in fig. 1, but this does not indicate only one bus 212 or one type of bus 212.
The memory 211 is used for storing programs, such as programs corresponding to the log processing device. The log processing means includes at least one software functional module which may be stored in the memory 211 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the electronic device. The processor 210, upon receiving the execution instruction, executes the program to implement the steps of the log processing method.
Possibly, the electronic device provided in the embodiment of the present application further includes a communication interface 213. The communication interface 213 is connected to the processor 210 via a bus. The communication interface 213 may be used for connecting external devices, such as at least one camera, audio capturing device, etc.
It should be understood that the structure shown in fig. 1 is merely a structural schematic diagram of a portion of an electronic device, which may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.
Referring to fig. 2, fig. 2 shows a flowchart of the log processing method provided in this embodiment, where the log processing method includes steps 110 to 130.
Step 110: and acquiring an original log, wherein the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels.
The original log is collected user data, and comprises user identity information of a user and a corresponding relation of a plurality of type labels, wherein the type labels comprise information of interests, hobbies, access sources, access time, local cities and the like of the user. The user data is stored in a predetermined format, as shown in table 1, table 1 provides one possible storage form of the original log.
In one possible implementation, the raw log is stored at a server, which may be a distributed server.
TABLE 1
Step 120: splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information includes a correspondence between user identity information and a type label.
The original log stores a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels, and in the actual processing process, the label types corresponding to one user identity information are too many, so that the analysis and processing efficiency is reduced when the data is too much. In this embodiment, the original log is split and reassembled according to the preset delimiter, and a plurality of single label information is generated, where the single label information refers to a corresponding relationship between one user identity information and one type label. The original log is split and recombined, namely a plurality of one-to-many data types are converted into a plurality of one-to-one data types, and the data types are simplified, so that the data processing and statistical analysis at the later stage are facilitated.
Step 130: and querying the plurality of single label information according to preset dimension information to obtain a query result, wherein the query result comprises the number of the single label information corresponding to the dimension information.
Splitting and recombining the original log to generate a plurality of single label information, and then querying the plurality of single label information according to preset dimension information to obtain a query result. The dimension information is one or more types of labels, for example, if the hobby is taken as a dimension of football, all the single label information is inquired, the number of all the users hobby for football can be determined, and if the hobby is taken as the football, and the city is taken as a dimension of Beijing, all the single label information is inquired, the number of all the users hobby for football of Beijing can be determined.
In the log processing method provided by this embodiment, an original log is obtained, and the original log is split and reassembled according to a preset delimiter, so as to generate a plurality of pieces of single label information; the single label information is inquired according to preset dimension information to obtain an inquiry result, the inquiry result comprises the number of the single label information corresponding to the dimension information, the original log is split and recombined to form a plurality of single label information, a plurality of one-to-many data types are converted into a plurality of one-to-one data types, the data types are simplified, when the user log is specifically analyzed, inquiry and sorting can be directly carried out according to the set dimension, and the efficiency of data processing and analysis is improved.
On the basis of fig. 2, referring to fig. 3, for how to segment and reassemble the original log, this embodiment provides another possible implementation manner, fig. 3 shows a flowchart of another log processing method provided by this embodiment, and step 120 includes the following sub-steps:
step 120-1: and carrying out primary segmentation on the original log according to the user identity information to obtain an intermediate log, wherein the intermediate log comprises a corresponding relation between the user identity information and a plurality of types of tags.
As shown in table 1, the original log information stores the correspondence between a plurality of user identity information and a plurality of types of tags, with the user identity information as an index. That is, the original log stores a plurality of one-to-many data types, and in one possible implementation, the original log is divided into intermediate logs by using the user identity information as a separator. As shown in tables 2 and 3, the intermediate logs obtained by the division can be stored in the forms shown in tables 2 and 3.
TABLE 2
| User ID | User interest | Name of city | Accessing a source | Time of access | Item ID | Key word | Others |
| User08 | Football, mountain-climbing | Beijing | Taobao (treasure made of Chinese herbal medicine) | 2020-07-21 | 10012 | Philips | … |
TABLE 3
In a possible implementation manner, if an original log includes N pieces of user identity information and a correspondence between the N pieces of user identity information and a plurality of types of tags, the original log is divided to form N intermediate logs. Each intermediate log comprises a corresponding relation between user identity information and a plurality of types of labels.
Step 120-2: and segmenting the intermediate log according to a preset separator to obtain user identity information and a plurality of type labels.
And segmenting the middle log again according to a preset separator to obtain user identity information and a plurality of type labels.
In a possible implementation manner, a punctuation mark or a check mark is used as a separator to segment an intermediate log again to obtain user identity information and a plurality of type labels.
Taking the intermediate log shown in table 2 as an example, dividing the intermediate log into punctuation marks is divided into User identity information of User08, and includes: hobby football; mountain climbing is favored; the city is Beijing; the visit source is Taobao; the visit time is 2020, 7, 21 days; the item ID is 10012; the keyword is a philips-like type label.
Step 120-3: and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
The user identity information and the type labels are respectively subjected to attention combination to generate a plurality of single label information, and for example, in order to obtain the hobby situation of the user, the user identity information and the type labels are recombined to generate the single label information.
For example, for the intermediate log shown in table 2, User identity information of User08 and football preference are generated; and the User identity information is User08, and the hobby is mountain climbing single label information. For a plurality of intermediate logs, a plurality of pieces of single tag information may be generated, and in a possible implementation manner, the generated plurality of pieces of single tag information may be stored by using the user identity information as an index, as shown in table 4, where table 4 shows a possible storage form of the single tag information.
TABLE 4
| User ID | User interest | Time of access |
| User01 | Football game | 2020-07-21 |
| User01 | Climbing mountain | 2020-07-21 |
| User01 | Game machine | 2020-07-21 |
| User01 | Travel toy | 2020-07-21 |
| User02 | Football game | 2020-07-21 |
| User02 | Travel toy | 2020-07-21 |
| User02 | Photography | 2020-07-21 |
| User03 | Photography | 2020-07-21 |
| User03 | Food | 2020-07-21 |
| User03 | Dancing | 2020-07-21 |
| User04 | Travel toy | 2020-07-20 |
| User04 | Dancing | 2020-07-20 |
| User03 | Photography | 2020-07-26 |
| User03 | Food | 2020-07-26 |
| User03 | Dancing | 2020-07-26 |
It is understood that since the same user may have multiple hobbies, there may be multiple hobbies tags corresponding to the same user identity information.
After the single tag information is generated, the user log may be analyzed and counted according to the single tag information, in a possible implementation manner, referring to fig. 4, before the single tag information is generated, the log processing method further includes:
step 111: and acquiring a query instruction, wherein the query instruction comprises preset dimension information, and the dimension information is at least one of the plurality of label information.
And acquiring a query instruction, wherein the query instruction comprises dimension information which needs to be queried by a user, and the dimension information is at least one of a plurality of types of tags. For example, the user's hobbies need to be counted if the number of users whose hobbies are football needs to be counted; favorites are dimension information included in the query.
And after the query instruction is obtained, processing the original log according to the dimension information to obtain single label information corresponding to hobbies.
In a possible implementation manner, after the single tag information is generated, the user log may be analyzed and counted according to the single tag information, referring to fig. 5, step 130 includes the following steps:
step 130-1: and reading a first preset number of data blocks, wherein each data block comprises a second preset number of single label information.
In one possible implementation, in combination with the computing power of the processor, a second preset number of pieces of single tag information are used as one data block, and all pieces of single tag information are divided into a first preset number of data blocks.
For example, if there are 12 pieces of single tag information, three pieces of single tag information are used as one data block to form one data block, and when statistical processing is performed on data, the four data blocks are read first.
In one possible implementation, because the log amount is large, the hapoop mapreduce computation framework is used for performing the grouping computation, and the four data blocks can be read by the same processor, but in other possible implementations, the four data blocks can be read by distributed processors respectively to improve the processing efficiency.
Step 130-2: and querying according to the single label information in each data block according to the preset dimension to generate a preliminary result, wherein the preliminary result comprises the quantity of the single label information corresponding to the preset dimension queried in the data block.
And respectively querying the single label information in each data block according to a preset dimension to generate a preliminary result. The preliminary result comprises the quantity of single label information corresponding to the preset dimensionality inquired in the data block. For example, the first data block includes information as shown in table 5:
TABLE 5
| User01 | Football game |
| User01 | Climbing mountain |
| User01 | Game machine |
Then, performing query statistics on the first data block with the label of this type as a dimension can obtain the following preliminary results, as shown in table 6:
TABLE 6
| Football game | 1 person |
| Climbing mountain | 1 person |
| Game machine | 1 person |
Step 130-3: and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain a query result.
And after the plurality of data blocks are respectively processed, carrying out the thunder addition fusion on a plurality of preliminary results obtained by processing the plurality of data blocks to obtain a query result.
For example, query processing is performed on the second data block to obtain a preliminary result corresponding to the second data block, query processing is performed on the third data block to obtain a preliminary result corresponding to the third data block, and query processing is performed on the fourth data block to obtain a preliminary result corresponding to the fourth data block.
And fusing the preliminary result corresponding to the first data block, the preliminary result corresponding to the second data block, the preliminary result corresponding to the third data block and the preliminary result corresponding to the fourth data block to obtain a final query result. As shown in fig. 6.
Referring to fig. 7, after step 130, the method further comprises:
step 140: and deleting the intermediate log and the single label information.
After the data is processed, the generated intermediate log and the single label information are deleted, so that excessive space occupation is avoided.
To execute the corresponding steps in the above embodiments and various possible implementations, an implementation of the log processing apparatus is given below, please refer to fig. 8, and fig. 8 is a log processing apparatus 300 according to a preferred embodiment of the present invention. It should be noted that the basic principle and the technical effect of the log processing apparatus 300 provided in the present embodiment are substantially the same as the air conditioner control method provided in the foregoing embodiment, and for the sake of brief description, for parts not mentioned in the present embodiment, reference may be made to the corresponding contents in the foregoing embodiment. The log processing apparatus 300 provided in this embodiment includes an obtaining module 310, a splitting module 320, and a processing module 330.
The obtaining module 310 is configured to obtain an original log, where the original log includes a plurality of user identity information and a corresponding relationship between the user identity information and a plurality of types of tags.
It is to be understood that, in one possible implementation manner, the obtaining module 310 may be configured to perform the step 110 in the above-mentioned figures to achieve the corresponding technical effect.
The splitting module 320 is configured to split and recombine the original log according to a preset separator to generate a plurality of single label information; the single label information includes a correspondence between user identity information and a type label.
It will be appreciated that in one possible implementation, the splitting module 320 may be configured to perform the step 120 in the above-mentioned figures to achieve the corresponding technical effect.
The processing module 330 is configured to query the multiple pieces of single label information according to the preset dimension information to obtain a query result, where the query result includes the number of the single label information corresponding to the dimension information.
It will be appreciated that in one possible implementation, the processing module 330 may be configured to perform the step 130 in the above figures to achieve the corresponding technical effect.
In an optional embodiment, the splitting module 320 is configured to perform initial segmentation on an original log according to user identity information to obtain an intermediate log, where the intermediate log includes a correspondence between the user identity information and multiple types of tags; dividing the intermediate log according to a preset separator to obtain user identity information and a plurality of type labels; and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
It is to be understood that, in one possible implementation, the splitting module 320 may be configured to perform steps 120-1 to 120-3 in the above-mentioned figures to achieve the corresponding technical effect.
In an optional embodiment, the splitting module is further configured to delete the intermediate log and the single label information after the step of querying the plurality of single label information according to the preset dimension information to obtain the query result.
It will be appreciated that in one possible implementation, the splitting module 320 may be configured to perform step 140 in the above-described figures to achieve corresponding technical effects.
In an optional embodiment, the processing module is configured to query the single tag information in each data block according to a preset dimension, and generate a preliminary result, where the preliminary result includes the number of the single tag information corresponding to the preset dimension queried in the data block; and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain a query result.
It is understood that in one possible implementation, the processing module 330 may be configured to perform steps 130-1 to 130-3 in the above-mentioned figures to achieve the corresponding technical effect.
An embodiment of the present invention provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the log processing method as described in any one of the foregoing embodiments is implemented.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Although the present invention is disclosed above, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method of log processing, the method comprising:
acquiring an original log, wherein the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels;
splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information comprises a corresponding relation between the user identity information and a type label;
and querying the single label information according to preset dimension information to obtain a query result, wherein the query result comprises the number of the single label information corresponding to the dimension information.
2. The log processing method according to claim 1, wherein the step of splitting and reassembling the original log according to a preset delimiter to generate a plurality of single label information comprises:
preliminarily dividing the original log according to the user identity information to obtain an intermediate log, wherein the intermediate log comprises a corresponding relation between the user identity information and a plurality of types of tags;
dividing the intermediate log according to preset separators to obtain user identity information and a plurality of type labels;
and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
3. The log processing method according to claim 2, wherein after the step of querying the plurality of single label information according to the preset dimension information to obtain the query result, the method further comprises:
and deleting the intermediate log and the single label information.
4. The log processing method according to claim 1, wherein the querying the single tag information according to the preset dimension information to obtain the query result comprises:
reading a first preset number of data blocks, wherein each data block comprises a second preset number of single label information;
inquiring according to the single label information in each data block according to a preset dimension respectively to generate a preliminary result, wherein the preliminary result comprises the number of the single label information corresponding to the preset dimension inquired in the data block;
and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain the query result.
5. The log processing method according to claim 1, wherein before querying the plurality of single label information according to the preset dimension information, the log processing method further comprises:
acquiring a query instruction, wherein the query instruction comprises preset dimension information, and the dimension information is at least one of a plurality of label information.
6. A log processing apparatus for implementing the log processing method according to any one of claims 1 to 5, the log processing apparatus comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring an original log, and the original log comprises a plurality of user identity information and corresponding relations between the user identity information and a plurality of types of labels;
the splitting module is used for splitting and recombining the original log according to preset separators to generate a plurality of single label information; the single label information comprises a corresponding relation between the user identity information and a type label;
and the processing module is used for inquiring the single label information according to preset dimension information to obtain an inquiry result, wherein the inquiry result comprises the number of the single label information corresponding to the dimension information.
7. The log processing apparatus according to claim 6, wherein the splitting module is configured to perform preliminary segmentation on the original log according to the user identity information to obtain an intermediate log, where the intermediate log includes a correspondence between the user identity information and a plurality of type tags; dividing the intermediate log according to preset separators to obtain user identity information and a plurality of type labels; and combining the user identity information with the plurality of types of labels one by one to generate a plurality of single label information.
8. The log processing apparatus according to claim 7, wherein the splitting module is further configured to delete the intermediate log and the single label information after the step of querying the plurality of single label information according to the preset dimension information to obtain the query result.
9. The log processing apparatus according to claim 6, wherein the processing module is configured to perform querying according to a preset dimension according to single tag information in each data block, and generate a preliminary result, where the preliminary result includes the number of the single tag information corresponding to the preset dimension queried in the data block; and carrying out classification statistics on a plurality of preliminary results corresponding to the plurality of data blocks to obtain the query result.
10. A storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements a log processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943762.0A CN112069048A (en) | 2020-09-09 | 2020-09-09 | Log processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943762.0A CN112069048A (en) | 2020-09-09 | 2020-09-09 | Log processing method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112069048A true CN112069048A (en) | 2020-12-11 |
Family
ID=73663511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010943762.0A Pending CN112069048A (en) | 2020-09-09 | 2020-09-09 | Log processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112069048A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632020A (en) * | 2020-12-25 | 2021-04-09 | 中国电子科技集团公司第三十研究所 | Log information type extraction method and mining method based on spark big data platform |
| CN112800666A (en) * | 2021-01-18 | 2021-05-14 | 上海派拉软件股份有限公司 | Log behavior analysis training method and identity security risk prediction method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| CN109739821A (en) * | 2018-12-18 | 2019-05-10 | 中国科学院计算机网络信息中心 | Daily record data bedding storage method, apparatus and storage medium |
| CN109831316A (en) * | 2018-12-17 | 2019-05-31 | 国网浙江省电力有限公司 | Massive logs real-time analyzer, real-time analysis method and readable storage medium storing program for executing |
| CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
-
2020
- 2020-09-09 CN CN202010943762.0A patent/CN112069048A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| CN109831316A (en) * | 2018-12-17 | 2019-05-31 | 国网浙江省电力有限公司 | Massive logs real-time analyzer, real-time analysis method and readable storage medium storing program for executing |
| CN109739821A (en) * | 2018-12-18 | 2019-05-10 | 中国科学院计算机网络信息中心 | Daily record data bedding storage method, apparatus and storage medium |
| CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632020A (en) * | 2020-12-25 | 2021-04-09 | 中国电子科技集团公司第三十研究所 | Log information type extraction method and mining method based on spark big data platform |
| CN112632020B (en) * | 2020-12-25 | 2022-03-18 | 中国电子科技集团公司第三十研究所 | Log information type extraction method and mining method based on spark big data platform |
| CN112800666A (en) * | 2021-01-18 | 2021-05-14 | 上海派拉软件股份有限公司 | Log behavior analysis training method and identity security risk prediction method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108268586B (en) | Data processing method, device, medium and computing equipment across multiple data tables | |
| CN110943961A (en) | Data processing method, device and storage medium | |
| CN111400392B (en) | Multi-source heterogeneous data processing method and device | |
| CN103902535A (en) | Method, device and system for obtaining associational word | |
| CN112069048A (en) | Log processing method, device and storage medium | |
| CN109829073B (en) | Image searching method and device | |
| CN108334609A (en) | Method, apparatus, equipment and the storage medium of the access of JSON formatted datas are realized in Oracle | |
| US20160019266A1 (en) | Query generating method and query generating device | |
| US20200204688A1 (en) | Picture book sharing method and apparatus and system using the same | |
| CN110889036A (en) | Multi-dimensional information processing method and device and terminal equipment | |
| CN112632213A (en) | Address information standardization method and device, electronic equipment and storage medium | |
| CN112015806A (en) | Method and device for storing data by block chain | |
| CN114691356A (en) | Data parallel processing method and device, computer equipment and readable storage medium | |
| CN111107443A (en) | DASH fragment file merging method, terminal device and storage medium | |
| CN110674383A (en) | Public opinion query method, device and equipment | |
| US20150032749A1 (en) | Method of creating classification pattern, apparatus, and recording medium | |
| CN114297236A (en) | Data blood relationship analysis method, terminal equipment and storage medium | |
| CN114611039A (en) | Asynchronous loading rule analysis method and device, storage medium and electronic equipment | |
| US9426173B2 (en) | System and method for elimination of spam in a data stream according to information density | |
| CN109543024B (en) | Text processing method and device | |
| CN111695031A (en) | Label-based searching method, device, server and storage medium | |
| CN111611056A (en) | Data processing method and device, computer equipment and storage medium | |
| CN112752165A (en) | Subtitle processing method, subtitle processing device, server and computer-readable storage medium | |
| CN111008220A (en) | Dynamic identification method and device of data source, storage medium and electronic device | |
| CN111324778A (en) | Data and service processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |