CN116991818A

CN116991818A - Log data processing method, device, medium and electronic equipment

Info

Publication number: CN116991818A
Application number: CN202310645429.5A
Authority: CN
Inventors: 任文龙
Original assignee: Beijing Volcano Engine Technology Co Ltd
Current assignee: Beijing Volcano Engine Technology Co Ltd
Priority date: 2023-06-01
Filing date: 2023-06-01
Publication date: 2023-11-03

Abstract

The disclosure relates to a log data processing method, a log data processing device, a medium and an electronic device, wherein the log data processing method comprises the following steps: receiving a query statement corresponding to log data to be analyzed in a data analysis engine; according to the query statement, querying a target data table in a data warehouse, and determining the log data in a log data source and a log file to which the log data belongs, wherein the log data source is an external data source corresponding to the data analysis engine, and the target data table contains field information of the log file; acquiring the log data from a log file of the log data source; carrying out virtualization partitioning on the log data to obtain log data blocks corresponding to the log data; the log data is loaded to the data analysis engine based on the log data block. Therefore, the loading of the log files in the external data source can be directly realized based on the query statement, and the log analysis efficiency is improved.

Description

Log data processing method, device, medium and electronic equipment

Technical Field

The disclosure relates to the technical field of computers, and in particular relates to a log data processing method, a log data processing device, a log data processing medium and electronic equipment.

Background

In the related art, the distributed data storage is implemented in various ways to store the print log of the running service, the query range of the online query is limited, the computing nodes are limited, and more mature services have the requirement of log offline analysis. In the related art, a business party is required to pull all log files from a log platform to a local place, and further performs corresponding offline analysis by combining the local log files and a log analysis tool. However, partial data may exist in the log file, which is inconvenient for the business party to pull, thereby reducing the accuracy and efficiency of offline analysis of the log.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In a first aspect, the present disclosure provides a log data processing method, the method including:

receiving a query statement corresponding to log data to be analyzed in a data analysis engine;

according to the query statement, querying a target data table in a data warehouse, and determining the log data in a log data source and a log file to which the log data belongs, wherein the log data source is an external data source corresponding to the data analysis engine, and the target data table contains field information of the log file;

Acquiring the log data from a log file of the log data source;

carrying out virtualization partitioning on the log data to obtain log data blocks corresponding to the log data;

the log data is loaded to the data analysis engine based on the log data block.

In a second aspect, the present disclosure provides a log data processing apparatus, the apparatus comprising:

the receiving module is used for receiving query sentences corresponding to log data to be analyzed in the data analysis engine;

the first determining module is used for determining the log data in a log data source and a log file to which the log data belongs according to a target data table in the query statement query data warehouse, wherein the log data source is an external data source corresponding to the data analysis engine, and the target data table contains field information of the log file;

the acquisition module is used for acquiring the log data from the log file of the log data source;

the first processing module is used for carrying out virtualized partitioning on the log data to obtain log data blocks corresponding to the log data;

and the loading module is used for loading the log data to the data analysis engine based on the log data block.

In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which when executed by a processing device performs the steps of the method of the first aspect.

In a fourth aspect, the present disclosure provides an electronic device comprising:

a storage device having a computer program stored thereon;

processing means for executing said computer program in said storage means to carry out the steps of the method of the first aspect.

In the technical scheme, query sentences corresponding to log data to be analyzed in the data analysis engine are received; and inquiring a target data table in a data warehouse according to the inquiry statement, determining the log data in a log data source and a log file to which the log data belongs, and acquiring the log data from the log file of the log data source. And then carrying out virtualized partitioning on the log data to obtain log data blocks corresponding to the log data, and loading the log data to the data analysis engine based on the log data blocks. Therefore, the loading of the log files in the external data source can be directly realized based on the query statement, so that service personnel can pull the log data to be analyzed from the external data source based on a general operation mode when the data of log analysis is loaded, the labor cost of the process of extracting the data from the source end, converting the data into the form and loading the Load to the destination end is effectively reduced, and the data processing ecology of the data analysis engine can be seamlessly docked and the data query efficiency is improved. In addition, the log data in the external data source is loaded to the data analysis engine in the form of a log Input block, so that the I/O (Input/Output) resource consumption in the data pulling process is effectively reduced, the analysis efficiency of the log data is further improved, and the method is suitable for a log analysis scene under big data.

Additional features and advantages of the present disclosure will be set forth in the detailed description which follows.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale. In the drawings:

fig. 1 is a flowchart of a log data processing method provided in accordance with one embodiment of the present disclosure.

FIG. 2 is a schematic diagram of loading log data under one embodiment in the present disclosure.

Fig. 3 is a block diagram of a log data processing apparatus provided in accordance with one embodiment of the present disclosure.

Fig. 4 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

It will be appreciated that prior to using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed and authorized of the type, usage range, usage scenario, etc. of the personal information related to the present disclosure in an appropriate manner according to the relevant legal regulations.

For example, in response to receiving an active request from a user, a prompt is sent to the user to explicitly prompt the user that the operation it is requesting to perform will require personal information to be obtained and used with the user. Thus, the user can autonomously select whether to provide personal information to software or hardware such as an electronic device, an application program, a server or a storage medium for executing the operation of the technical scheme of the present disclosure according to the prompt information.

As an alternative but non-limiting implementation, in response to receiving an active request from a user, the manner in which the prompt information is sent to the user may be, for example, a popup, in which the prompt information may be presented in a text manner. In addition, a selection control for the user to select to provide personal information to the electronic device in a 'consent' or 'disagreement' manner can be carried in the popup window.

It will be appreciated that the above-described notification and user authorization process is merely illustrative and not limiting of the implementations of the present disclosure, and that other ways of satisfying relevant legal regulations may be applied to the implementations of the present disclosure.

Meanwhile, it can be understood that the data (including but not limited to the data itself, the acquisition or the use of the data) related to the technical scheme should conform to the requirements of the corresponding laws and regulations and related regulations.

Fig. 1 is a flowchart of a log data processing method according to an embodiment of the disclosure, where the method may include:

in step 11, a query statement corresponding to log data to be analyzed in the data analysis engine is received.

The data analysis engine may be a tool for log analysis, such as Spark tool, which is a fast general purpose computing engine designed for large-scale data processing. In this embodiment, a business person who needs to perform log offline analysis may submit a query statement in the visual interface, so as to pull corresponding data from an external log data source based on the query statement for log analysis. The query statement may be, for example, SQL (Structured Query Language ) logic, whereby the ETL schema may be implemented based on spark SQL. In this embodiment it is possible to represent the process of uploading log data from an external log data source to the data analysis engine.

In step 12, according to the query statement, the target data table in the data warehouse is queried, and the log data in the log data source and the log file to which the log data belongs are determined, wherein the log data source is an external data source corresponding to the data analysis engine, and the target data table contains field information of the log file.

The data warehouse can be Hive, which is a data warehouse tool based on Hadoop, and can be used for data extraction, conversion and loading. This is a mechanism by which large-scale data stored in Hadoop can be stored, queried, and analyzed. The Hive data warehouse tool can map structured data files into a database table and provide SQL query functions.

In this embodiment, the generated log file may be mapped into a Hive table, with specific fields defined on the Hive table structure. And realizing the corresponding relation between the automatic association physical data file and the Hive table, namely the target data table by a metadata management Coordinator module in the log platform. For example, the log file can be divided into files under multiple tenants, each tenant corresponds to a Hive table, and the data links of log data of multiple service parties can be located under the tenant in a Hive table. The Hive metadata service Metastore can be used to package Hive tables, so that data of each service can be mapped to the Hive tables in a one-to-one correspondence to obtain the target data table, and the quick positioning of log data stored in which log files is facilitated.

For example, the service personnel can know the table name of the designated Hive table, and the data links of different service parties under the corresponding tenant of the Hive table can be identified by the field, so that the data links of different service parties, namely the log files belonging to the service parties, can be obtained based on the attribute values of the different service parties under the field. Meanwhile, the required log files under the service side can be further determined by combining the query conditions in the query statement, and the storage information of the log files in the log data source can be determined based on the target data table, namely, the physical files in which the required log files are specifically stored are positioned, so that the corresponding log data can be conveniently pulled.

In step 13, log data is obtained from a log file of a log data source.

After determining the log file, the log data corresponding to the query statement may be determined from the log file, and as an example, all field information of the log determined in the log file may be pulled to be the log data.

In step 14, the log data is virtualized and partitioned, and a log data block corresponding to the log data is obtained. By way of example, each log data block resulting from virtualization may be represented comprehensively by a list of data blocks.

In step 15, log data is loaded to the data analysis engine based on the log data block.

As an example, access to an external data source may be achieved through a Spark Datasouce V mechanism. The applicant found through research that most of the related art is to interact with Spark in the manner of data lines (Record) and map directly to Spark internal RDD (Resilient Distributed Datasets, elastic distributed data set). By the mode, the system I/O in the data pulling process is more, and when the data volume of the external data source is huge, the I/O resource occupation is larger.

Based on the above, in the embodiment of the present disclosure, the log data may be virtualized and blocked, so that the log data in the external data source may be loaded to the data analysis engine in the form of the log data block, so as to effectively reduce the I/O resource consumption in the data pulling process.

In the technical scheme, query sentences corresponding to log data to be analyzed in the data analysis engine are received; and inquiring a target data table in a data warehouse according to the inquiry statement, determining the log data in a log data source and a log file to which the log data belongs, and acquiring the log data from the log file of the log data source. And then carrying out virtualized partitioning on the log data to obtain log data blocks corresponding to the log data, and loading the log data to the data analysis engine based on the log data blocks. Therefore, the loading of the log files in the external data source can be directly realized based on the query statement, so that service personnel can pull log data to be analyzed from the external data source based on a general operation mode when carrying out data loading of log analysis, the labor cost of offline analysis of the service ETL is effectively reduced, the data processing ecology of the data analysis engine can be seamlessly connected, and the data query efficiency is improved. In addition, the log data in the external data source is loaded to the data analysis engine in the form of a log input block, so that the I/O resource consumption in the data pulling process is effectively reduced, the analysis efficiency of the log data is further improved, and the method is suitable for a log analysis scene under big data.

If the log file has index information, the log file can be directly located to a specific log file based on the index information, if the log_id of the log file can be used for constructing an index, the corresponding log file can be directly determined based on the corresponding log_id during query. If the log files do not have index information, each log file needs to be traversed in turn to be compared with the log_ID of the query for judgment. Based on this, the present disclosure provides the following examples.

In a possible embodiment, the performing virtualization partitioning on the log data to obtain a log data block corresponding to the log data may include:

if the log file has index information, determining a plurality of time segments, and mapping the log data in the log file to log data blocks corresponding to the time segments to which the time information of the log file belongs, wherein the time segments are in one-to-one correspondence with the log data blocks.

For example, the time segmentation may be performed according to a preset duration, for example, the preset duration may be set according to an actual application scenario, and if the pulled log data is the data of the same day, the preset duration may be set to 1 hour, that is, each hour corresponds to one time segment, for example, the log file generated from 0 point to 1 point may be mapped to the same log data block. Further, when the log data is virtualized, the log data block may be stored based on the time information of the log file to which the log data belongs, if the time information of the log file to which the log data belongs is 12 minutes at 0 point, the log data block corresponding to the time segment of 0 point-1 point is mapped; if the time information of the log file to which the log data belongs is 12 minutes at 1 point, the log data is mapped to a log data block corresponding to a time segment of 1 point to 2 points.

Therefore, through the technical scheme, the virtual partitioning of the queried data in the external data source can be realized so as to correspond to the partitioning units of the processing data set in the data analysis engine one by one, and the analysis efficiency of log data in the data analysis engine can be ensured while the data pulling efficiency is improved.

And if the log file does not have index information, sequentially traversing each log data, if the number of the log data mapped in the current log data block does not reach a number threshold, mapping the traversed log data to the current log data block, and if the number of the log data mapped in the current log data block reaches a number threshold, mapping the traversed log data to a next log data block.

The number threshold may be set based on an actual application scenario. In an embodiment, each log data block may be mapped in turn based on the number threshold, and the next log data block may be mapped after the mapping of the current log data block is completed. For example, if the current log data block is the first log data block, if the number of log data mapped therein does not reach the number threshold, mapping the traversed log data to the first log data block. And after multiple mapping, if the number of the log data mapped in the first log data block reaches a number threshold, mapping the traversed log data to the next log data block, namely, the second log data block, so that the efficiency and the stability of data pulling are facilitated.

For example, the maximum data amount corresponding to the log data block may be preset, each log data may be traversed in sequence, if the data amount of the log data mapped in the current log data block does not reach the maximum data amount, the traversed log data is mapped to the current log data block, and if the data amount of the log data mapped in the current log data block reaches the maximum data amount, the traversed log data is mapped to the next log data block. As another example, the maximum data amount and the number threshold may also be set, and when each of the log data is traversed, if either one of the two is reached, the traversed log data is mapped to the next log data block.

In one possible embodiment, each of the log data blocks includes a plurality of log data segments. For example, a data amount threshold corresponding to each log data segment may be preset, and the data amount threshold may be set based on an actual application scenario, for example, may be set to 5M, which is not limited in this disclosure. Illustratively, a schematic diagram of loading log data under one embodiment in the present disclosure is shown in FIG. 2. Wherein A1-An are used to represent n log data blocks datablocks, and A11-A1m are used to represent m log data segment fragments in log data block A1. Wherein T is used to represent a physical file set, that is, a plurality of physical file sets are corresponding to the bottom layer of the log storage platform, a single physical file set corresponds to a plurality of log data blocks, and each log data block corresponds to at most one physical file set. In this embodiment, A1-An may be included in the data block list. As based on fig. 2, in the data analysis engine Spark, the driver may be a driver, which is a program for driving the whole operation, and the execution unit may be an Executor, which is a JVM process in a working node (Worker) in the cluster, and is responsible for running a specific task in the Spark job.

Accordingly, an exemplary implementation of the loading the log data to the data analysis engine based on the log data block may include:

and for each log data block, reading each log data segment in the log data block to an operation memory of the data analysis engine in sequence, and analyzing the log data segment to obtain a log record in a target format, wherein the target format is a data format in which the data analysis engine operates.

For example, a cursor may be set in a log data block to identify a log data segment that has been read, such as for log data block A1, the initial cursor indicating the starting position of the log data block A1, after which the first log data segment A11 therein is read, at which point the cursor indicates the position of the next log data segment A12.

After the first log data segment a11 is read to the running memory of the data analysis engine, the data in the log data segment may be parsed to be a log record that can be identified by the data analysis engine. After the analysis of the data in the log data segment a11 is completed, the running memory from the log data segment a12 to the data analysis engine can be further read based on the cursor, the data of the log data segment a12 is analyzed, and so on until each log data segment in the current log data block A1 is read. The next log data block A2 can then be read. The reading manner is the same as that described above, and will not be described again here.

Therefore, through the technical scheme, each log data block can be further divided into a plurality of log data segments, and the log data segments are used as the minimum data set of the pulled data in the data analysis engine, so that the I/O resource consumption in the data pulling process can be effectively reduced, the influence of one-time pulling of excessive data on various other services in the whole cluster can be avoided, and the efficiency and stability of the log analysis process are ensured.

In a possible embodiment, each of the log data segments comprises a plurality of records, e.g. each record corresponds to a data line. As indicated above, most of the interactions within the data analysis engine are mapped directly to the Spark internal RDD [ lnternalRow ] with the data lines Record, and the Spark SQL operates entirely using the RDD [ lnternalRow ] type when executing the physical planning operation RDD. Correspondingly, in order to adapt to data processed by the data analysis engine Spark, correspondingly, the parsing the log data segment to obtain a log record in a target format may include:

analyzing each field corresponding to each record and the field value of each field aiming at each record in the log data segment;

Converting a field in the record as a key value and a field value of the field as a value into a key value pair structure;

and performing target format conversion on the structure based on the key value to obtain the log record.

For example, the log data segment may include N records, where the log data segment is loaded into a runtime memory of the data analysis engine, and the records are further parsed, e.g., each record may be traversed in turn to generate a key-value pair structure corresponding to the record. For example, if each record contains M fields, the log record corresponding to the log data segment may be represented by a Map structure, i.e., N sets of K-V data structures, each set of K-V data structures further containing M sets of K-V pairs (key value pairs).

And then traversing the K-V data structure in turn, and obtaining a log record based on a target format conversion mode in the data analysis engine, wherein the target format can be RDD [ lnternalRow ], which can be subjected to format conversion based on a mode of converting RDD [ lnternalRow ] in the field, and is not described herein.

Therefore, through the technical scheme, the log data segment can be further analyzed, and the mapping from the data block to the K-V data structure is realized, so that when RDD [ lnternalRow ] is converted from the inside of the data analysis engine Spark, a plurality of log records in the log data segment can be acquired from the analyzed memory K-V data structure through one-time I/O processing, the consistency of pulled data of an external data source and the data type of memory function operation processing of the data analysis engine can be realized, the I/O resource consumption can be effectively reduced, and the log analysis efficiency is improved.

In one possible embodiment, before the step of virtualizing the log data into blocks, to obtain the log data blocks corresponding to the log data, the method may further include:

a query field in the query statement is determined.

Wherein the target data table contains a plurality of fields, and only partial fields in the fields which are returned when the data is queried can be determined based on the query statement in the embodiment. For example, if the query statement is an SQL statement, a field corresponding to the Select in the SQL statement may be used as the query field, that is, a field that needs to be returned after the data query.

Deleting other fields except the query field in the log data obtained from the log file of the log data source, and taking the deleted data as new log data.

In this embodiment, field deletion is performed on the log data obtained from the log file of the log data source, that is, before loading the data of the external data source, unnecessary fields (i.e., fields other than the query field) in the queried data may be deleted, so that only the field to be queried is reserved in the new log data, thereby effectively reducing the amount of data to be loaded, and simultaneously ensuring the data required for meeting the result display of the data query, effectively reducing the occupation of the data loading on the bandwidth, and improving the loading and analysis efficiency of the log data.

and compressing the log data obtained from the log file of the log data source, and taking the data obtained after the compression as new log data.

As an example, the compression processing may be compression processing of directly acquired log data, so that the data amount of log data to be loaded may be reduced. As another example, it may compress data for each field in log data separately, thereby obtaining compressed data for each field, and regard the compressed data for each field as new log data. The compression process may be based on a compression method commonly used in the art, and is not limited herein.

Therefore, the log data to be loaded is compressed, so that the data quantity to be loaded is further reduced, the method is more suitable for a loading scene of the log data under a large data set, and the data loading efficiency is further improved.

As another example, after the compressed data of each field is obtained, the compressed data of other fields except the query field may be further deleted, that is, the compressed data of the query field is used as new target data to perform subsequent virtualized block processing, so that loading and analysis efficiency of log data may be further improved.

In one possible embodiment, the method may further comprise:

when a predicate pushing mode is started in the data analysis engine, determining target query conditions for predicate pushing in the query statement.

And in Hive, predicate pushdown is started by configuring a hive.optimal.ppd parameter to be true. Predicate pushdown may be to move the filter expression as close as possible to the data source so that irrelevant data can be skipped directly when actually executing. Predicate as in SQL is mainly like, betwen, is null, in, =, +|! =etc. For example, the target query condition of the query statement for predicate downconversion is determined based on the predicate downconversion judgment rule, i.e. the condition of directly performing data filtering in the external data source. If predicate writing of the reserved table cannot be performed in join, a router is needed; in the case of join association, the filtering condition can be predicate-pushed in either join or where, and the target query condition can be determined based on the actual query statement and rules of predicate-push in Hive.

Correspondingly, the determining the log data in the log data source and the log file to which the log data belongs according to the target data table in the query statement query data warehouse comprises the following steps:

And inquiring the target data table according to the target inquiry condition, and determining the log data and the log file to which the log data belongs.

In this embodiment, a query in a target data table, such as a where condition hit index, may be performed based on the target query condition for predicate downpushing, and data filtering may be performed directly at the location of the external data source without the need to execute the target query condition again in the data analysis engine.

Therefore, through the technical scheme, the target query condition pushed down by the predicate is executed in advance at the data source end, the output of the data source end is reduced, the data transmission I/O is reduced, invalid data is prevented from being loaded into the data analysis engine, the offline analysis of a data set with a super-large scale and a long period can be supported, the data quantity of data transmission can be reduced, the influence of data query and filtering of a large amount of data loaded into the data analysis engine on the log analysis efficiency can be avoided, and the execution speed of the data analysis engine for processing the data set is improved.

Based on the embodiments described above, after loading log data into the data analysis engine, a corresponding log analysis process may be performed based on functions and methods in the data analysis engine, and the result of the analysis task end log analysis may be output to a specified HDFS (Hadoop Distributed File System ) directory in the data analysis engine, and finally presented to the result page through an asynchronous Rest Api in the data analysis engine. The process may be implemented based on a native method in the data analysis engine, which is not described herein.

Based on the same inventive concept, the present disclosure further provides a log data processing apparatus, as shown in fig. 3, the apparatus 10 includes:

the receiving module 100 is configured to receive a query statement corresponding to log data to be analyzed in the data analysis engine;

a first determining module 200, configured to determine, according to the query statement, the log data in a log data source and a log file to which the log data belongs, where the log data source is an external data source corresponding to the data analysis engine, and the target data table includes field information of the log file;

an obtaining module 300, configured to obtain the log data from a log file of the log data source;

the first processing module 400 is configured to perform virtualized partitioning on the log data to obtain a log data block corresponding to the log data;

and the loading module 500 is used for loading the log data to the data analysis engine based on the log data block.

Optionally, each of the log data blocks comprises a plurality of log data segments;

the loading module comprises:

the loading sub-module is used for reading each log data segment in the log data block to the running memory of the data analysis engine in sequence aiming at each log data block, and analyzing the log data segment to obtain a log record in a target format, wherein the target format is a data format in which the data analysis engine operates.

Optionally, each log data segment includes a plurality of records therein;

the loading submodule comprises:

the analysis submodule is used for analyzing each field corresponding to each record and the field value of each field aiming at each record in the log data segment;

the first conversion sub-module is used for converting the fields in the record as key values and the field values of the fields as value values into a key value pair structure;

and the second conversion sub-module is used for carrying out target format conversion on the structure based on the key value to obtain the log record.

Optionally, the first processing module includes:

the first processing sub-module is used for determining a plurality of time segments if the log file has index information, and mapping log data in the log file to log data blocks corresponding to the time segments to which the time information of the log file belongs, wherein the time segments are in one-to-one correspondence with the log data blocks;

and the second processing sub-module is used for traversing each log data if the log file does not have index information, mapping the traversed log data to the current log data block if the number of the log data mapped in the current log data block does not reach a number threshold, and mapping the traversed log data to the next log data block if the number of the log data mapped in the current log data block reaches a number threshold.

Optionally, the apparatus further comprises:

the second determining module is used for determining a query field in the query statement before the first processing module performs virtualized partitioning on the log data to obtain a log data block corresponding to the log data;

and the second processing module is used for deleting other fields except the query field in the log data acquired from the log file of the log data source, and taking the deleted data as new log data.

Optionally, the apparatus further comprises:

and the third processing module is used for compressing the log data obtained from the log file of the log data source before the first processing module performs virtualization blocking on the log data to obtain a log data block corresponding to the log data, and taking the data obtained after the compression processing as new log data.

Optionally, the apparatus further comprises:

the third determining module is used for determining target query conditions for predicate pushing in the query statement when a predicate pushing mode is started in the data analysis engine;

the first determination module is further to:

Referring now to fig. 4, a schematic diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 4 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 4, the electronic device 600 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

In general, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 4 shows an electronic device 600 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 601.

It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a query statement corresponding to log data to be analyzed in a data analysis engine; according to the query statement, querying a target data table in a data warehouse, and determining the log data in a log data source and a log file to which the log data belongs, wherein the log data source is an external data source corresponding to the data analysis engine, and the target data table contains field information of the log file; acquiring the log data from a log file of the log data source; carrying out virtualization partitioning on the log data to obtain log data blocks corresponding to the log data; the log data is loaded to the data analysis engine based on the log data block.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented in software or hardware. The name of a module is not limited to the module itself in some cases, and for example, the receiving module may also be described as "a module that receives a query statement corresponding to log data to be analyzed in the data analysis engine".

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, example 1 provides a log data processing method, wherein the method comprises:

acquiring the log data from a log file of the log data source;

the log data is loaded to the data analysis engine based on the log data block.

Example 2 provides the method of example 1, wherein each of the log data blocks comprises a plurality of log data segments, according to one or more embodiments of the present disclosure;

the loading the log data to the data analysis engine based on the log data block includes:

Example 3 provides the method of example 2, wherein each of the log data segments comprises a plurality of records, in accordance with one or more embodiments of the present disclosure;

the analyzing the log data segment to obtain the log record under the target format includes:

According to one or more embodiments of the present disclosure, example 4 provides the method of example 1, wherein the performing virtualization partitioning on the log data to obtain a log data block corresponding to the log data includes:

if the log file has index information, determining a plurality of time segments, and mapping log data in the log file to log data blocks corresponding to the time segments to which the time information of the log file belongs, wherein the time segments are in one-to-one correspondence with the log data blocks;

and if the log file does not have index information, traversing each log data, if the number of the log data mapped in the current log data block does not reach a number threshold, mapping the traversed log data to the current log data block, and if the number of the log data mapped in the current log data block reaches a number threshold, mapping the traversed log data to a next log data block.

According to one or more embodiments of the present disclosure, example 5 provides the method of example 1, wherein, before the step of virtualizing the log data to obtain a log data block corresponding to the log data, the method further includes:

determining a query field in the query statement;

According to one or more embodiments of the present disclosure, example 6 provides the method of example 1, wherein, before the step of virtualizing the log data to obtain a log data block corresponding to the log data, the method further includes:

According to one or more embodiments of the present disclosure, example 7 provides the method of example 1, wherein the method further comprises:

when a predicate pushing mode is started in a data analysis engine, determining target query conditions for predicate pushing in the query statement;

The determining the log data in the log data source and the log file to which the log data belongs according to the target data table in the query statement query data warehouse comprises the following steps:

Example 8 provides a log data processing apparatus according to one or more embodiments of the present disclosure, wherein the apparatus comprises:

According to one or more embodiments of the present disclosure, example 9 provides a computer-readable medium having stored thereon a computer program which, when executed by a processing device, implements the steps of the method of any of examples 1-7.

In accordance with one or more embodiments of the present disclosure, example 10 provides an electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to implement the steps of the method of any one of examples 1-7.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims. The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

Claims

1. A method of log data processing, the method comprising:

acquiring the log data from a log file of the log data source;

the log data is loaded to the data analysis engine based on the log data block.

2. The method of claim 1, wherein each of the log data blocks comprises a plurality of log data segments;

3. The method of claim 2, wherein each of the log data segments comprises a plurality of records;

4. The method of claim 1, wherein the performing virtualized partitioning on the log data to obtain a log data block corresponding to the log data includes:

5. The method of claim 1, wherein prior to the step of virtualizing the log data to obtain the log data block corresponding to the log data, the method further comprises:

determining a query field in the query statement;

6. The method of claim 1, wherein prior to the step of virtualizing the log data to obtain the log data block corresponding to the log data, the method further comprises:

7. The method according to claim 1, wherein the method further comprises:

8. A log data processing apparatus, the apparatus comprising:

9. A computer readable medium on which a computer program is stored, characterized in that the program, when being executed by a processing device, carries out the steps of the method according to any one of claims 1-7.

10. An electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing said computer program in said storage means to carry out the steps of the method according to any one of claims 1-7.