CN112463527A

CN112463527A - Data processing method, device, equipment, system and storage medium

Info

Publication number: CN112463527A
Application number: CN202011273009.1A
Authority: CN
Inventors: 杨元; 覃建策; 于化棣; 谭友信; 陈邦忠
Original assignee: Perfect World Holding Group Ltd
Current assignee: Perfect World Holding Group Ltd
Priority date: 2020-11-13
Filing date: 2020-11-13
Publication date: 2021-03-09

Abstract

The embodiment of the application provides a data processing method, a device, equipment, a system and a storage medium. In the embodiment of the application, the log management device is additionally arranged between the service end and the computing end, and can acquire log data provided by the service end; determining at least one target attribute field required by a computing end; and extracting and storing at least one log record matched with the at least one target attribute field from the log data so that the computing end can access the at least one log record to perform data computation. Accordingly, in the embodiment, the service end and the computing end do not directly communicate with each other, which can effectively reduce application intrusion to the service end; moreover, the log management device can collect, arrange and store the log data provided by the service end according to the calculation requirement of the calculation end, so that the use of the calculation end is more convenient, and the data collection complexity of the calculation end is reduced.

Description

Data processing method, device, equipment, system and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, system, and storage medium.

Background

Currently, in the process of building a feature computing platform, log data collection is usually performed by means of "flash + log" or "mysql + canel" and the like. The flash is a system for collecting, aggregating and transmitting mass logs in a distributed manner, and is high in availability, high in reliability. Canel is an open source project, is realized based on java, and monitors binlog log logs of mysql to acquire data in a mode of simulating slave becoming mysql.

In the "mysql + canel" mode, there is a risk of application intrusion; in the "flash + log" mode, the log collection complexity is too high, and the efficiency is too low.

Disclosure of Invention

Aspects of the present application provide a data processing method, apparatus, device and storage medium, so as to reduce complexity of data acquisition and/or reduce application intrusion in a data acquisition process.

An embodiment of the present application provides a data processing method, including:

determining at least one first target attribute field required by a first computing end according to a first computing requirement of the first computing end;

acquiring log data provided by a service end;

extracting at least one log record matching the at least one first target attribute field from the log data;

and storing the at least one log record so that the first computing terminal can access the log record to perform data computation.

An embodiment of the present application further provides a log management device, including:

the first interaction module is used for determining at least one first target attribute field required by a first computing end according to a first computing requirement of the first computing end;

the second interaction module is used for acquiring log data provided by the service end;

the processing module is used for extracting at least one log record matched with the at least one first target attribute field from the log data; and storing the at least one log record for the first computing terminal to access the at least one log record for data computing.

The embodiment of the application also provides a computing device, which comprises a memory, a processor and a communication component;

the memory is to store one or more computer instructions;

the processor, coupled with the memory and the communication component, to execute the one or more computer instructions to:

determining at least one target attribute field required by a first computing end according to a first computing requirement of the first computing end;

acquiring log data provided by a service end through the communication assembly;

extracting at least one log record matching the at least one first target attribute field from the log data; and storing the at least one log record for the first computing terminal to access the at least one log record for data computing.

An embodiment of the present application further provides a data processing system, including: the system comprises a service end, a log management device and a first computing end, wherein the log management device is respectively in communication connection with the service end and the first computing end;

the log management device is used for determining at least one first target attribute field required by the first computing end according to the first computing requirement of the first computing end; acquiring log data provided by the service end; extracting at least one log record matching the at least one first target attribute field from the log data; storing the at least one log record;

the first computing terminal is configured to access at least one log record corresponding to the first computing terminal from the log management device to perform data computation.

Embodiments of the present application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the aforementioned data processing method.

In the embodiment of the application, the log management device is additionally arranged between the service end and the computing end, and can acquire log data provided by the service end; determining at least one target attribute field required by a computing end; and extracting and storing at least one log record matched with the at least one target attribute field from the log data so that the computing end can access the at least one log record to perform data computation. Accordingly, in the embodiment, the service end and the computing end do not directly communicate with each other, which can effectively reduce application intrusion to the service end; moreover, the log management device can collect, arrange and store the log data provided by the service end according to the calculation requirement of the calculation end, so that the use of the calculation end is more convenient, and the data collection complexity of the calculation end is reduced.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a block diagram of a data processing system according to an exemplary embodiment of the present application;

FIG. 2 is a schematic diagram of an application scenario provided in an exemplary embodiment of the present application;

fig. 3 is a schematic flow chart of a data processing method according to an exemplary embodiment of the present application;

fig. 4 is a schematic structural diagram of a log management apparatus according to an exemplary embodiment of the present application;

fig. 5 is a schematic structural diagram of a computing device according to an exemplary embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Aiming at the technical problems of high application intrusion risk or excessive acquisition complexity and the like in the existing log data acquisition scheme, some embodiments of the application embodiment: a log management device is added between the service end and the computing end, and can acquire log data provided by the service end; determining at least one target attribute field required by a computing end; and extracting and storing at least one log record matched with the at least one target attribute field from the log data so that the computing end can access the at least one log record to perform data computation. Accordingly, in the embodiment, the service end and the computing end do not directly communicate with each other, which can effectively reduce application intrusion to the service end; moreover, the log management device can collect, arrange and store the log data provided by the service end according to the calculation requirement of the calculation end, so that the use of the calculation end is more convenient, and the data collection complexity of the calculation end is reduced.

The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic structural diagram of a data processing system according to an exemplary embodiment of the present application. As shown in fig. 1, the system includes: the system comprises a service end 10, a log management device 20 and a first computing end 30. The log management device 20 is respectively connected with the service terminal 10 and the first calculation terminal 30 in a communication way.

The data processing system provided by the embodiment can be applied to various scenes in which data calculation needs to be performed according to log data, such as a characteristic calculation scene and the like. The application scenario is not limited in this embodiment.

In this embodiment, the service end 10 may be a personal terminal, an enterprise server, or another type of processing end, and the type of the service end 10 is not limited in this embodiment. In terms of physical implementation, the service end 10 may be a computing device such as a personal computer, a smart phone, a tablet computer, or may be a server device such as a conventional server, a cloud host, and a virtual center. The server device mainly includes a processor, a hard disk, a memory, a system bus, and the like, and is similar to a general computer architecture.

In practical applications, the number of accessible service terminals 10 in the data processing system may be one or more, and only one service terminal 10 is exemplarily shown in fig. 1, which should not cause a loss of the protection scope of the present application. Similarly, the number of the computing terminals in the data processing system may be one or more, and only the first computing terminal 30 is exemplarily shown in fig. 1, but it should be understood that other computing terminals such as the second computing terminal may be further included in the data processing system.

The service end 10 may be configured with a logging function to generate log data. The log data is used to record system events or operation events occurring in the service terminal 10.

In this embodiment, the log management device 20 may obtain log data provided by the service end 10. The log management device 20 may obtain log data from the service end 10 in a kafka manner or a canel manner. Wherein, kafka is a distributed publish-subscribe message system, which can process all action stream data of the service end. In this manner, the log management device 20 can acquire the log data of the service terminal 10 by consuming the kafka message. Canel is an open source project, is realized based on java, and monitors binlog log logs of mysql to acquire data in a mode of simulating slave becoming mysql. In this way, the log management device 20 can pull the log data from the database corresponding to the service end 10.

Of course, the log management device 20 may also use other implementations to obtain log data from the service end 10, for example, obtaining relevant data by directly querying mysql. In addition, for the service end 10, it is only necessary to open log access right to the log management device 20. This can effectively ensure the application security of the service end 10, and the service end 10 is no longer exposed to the application intrusion risk of the first computing end 30.

In this embodiment, the log management device 20 may determine at least one target attribute field required by the first computing end 30 according to the first computing requirement of the first computing end 30. It should be noted that the computing requirements of different computing ends may not be identical, and the computing requirements of the same computing end may change over time.

The first computing end 30 may provide a plurality of service interfaces to the outside, and may be an API interface in actual application. The data services corresponding to different service interfaces may not be identical, and accordingly, the log attribute fields required by different data services may not be identical. In this embodiment, the first computing end 30 may perform data computation for each service interface, and therefore, when the first computing end 30 performs data computation for different service interfaces, the required log attribute fields may not be completely the same.

Based on this, in the present embodiment, the description of the technical solution will be made by taking any one time of data calculation performed by the first calculation terminal 30 as an example.

In this embodiment, the log management device 20 may determine at least one log attribute field required by the first computing end 30 in the data computing process as the first target attribute field. For example, if the first computing end 30 executes the data computation and needs two log attribute fields, namely "query" and "userID", the log management device 20 may determine that the "query" and the "userID" are the first target attribute fields needed by the first computing end 30. In practical applications, the first computing end 30 may generate configuration information reflecting the computing requirements according to the required log attribute fields, and provide the configuration information to the log management device 20, so that the log management device 20 determines at least one first target attribute field required by the first computing end 30, which is not limited to this embodiment.

In this embodiment, the log data typically includes a number of log records. In this embodiment, the log management device 20 may extract and store at least one log record matched with at least one first target attribute field from the log data. In practical applications, the log management device 20 may use the first target attribute field as a search condition, search a log record containing the first target attribute field from the log data, and store the searched log record. The log record includes attribute information corresponding to at least one first target attribute field. Of course, the log management apparatus 20 may also discard other attribute fields than the at least one first target attribute field from the found log record, for example, one log record includes 30 fields, but only 25 fields are the first target attribute fields, so that only 25 first target attribute fields and their corresponding attribute information may be retained in the log record, and the other 5 fields and field values may be discarded. Of course, this embodiment does not limit this.

Accordingly, the first computing terminal 30 can access at least one log record from the log management device 20 to perform data computation. In this embodiment, the log management device 20 may configure an access right to the stored at least one log record for the first computing end 30. In practical applications, when receiving an access request from the first computing end 30, the log management device 20 may first perform authority verification on the first computing end 30, and allow the first computing end 30 to access at least one log record when the first computing end 30 passes the authority verification, so that the first computing end 30 obtains attribute information corresponding to at least one first target attribute field. Optionally, in this embodiment, the log management device 20 may close the access right of the first computing end 30 to other types of logs (including the access right of the log file and the database storing the log) other than the at least one corresponding log record, so that by configuring the access right of the log record, the security of other data and the original log data in the log management device 20 may be effectively ensured.

For the first computing end 30, the required log record can be extracted from the log management device 20 by means of flash or the like. Because the log records in the log management device 20 are sorted according to the computing requirements of the first computing end 30, the first computing end 30 can more conveniently directly access the log records.

On this basis, the first computing terminal 30 may perform data computation by using spark real-time computation, spark stream, flink real-time computation, and store the computation result in the hbase, es, mysql, or other positions.

In this embodiment, a log management device 20 is added between the service end 10 and the first computing end 30, and the log management device 20 can obtain log data provided by the service end 10; determining at least one first target attribute field required by the first computing terminal 30; and extracting and storing at least one log record matched with the at least one first target attribute field from the log data, so that the first computing terminal 30 can access the at least one log record to perform data computation. Accordingly, in this embodiment, the service end 10 and the first computing end 30 do not directly communicate with each other, which can effectively reduce application intrusion to the service end 10; moreover, the log management device 20 can collect, arrange and store the log data provided by the service end 10 according to the calculation requirement of the first calculation end 30, which is more convenient for the first calculation end 30 to use, thereby reducing the data collection complexity of the first calculation end 30.

In the above or below embodiments, the log management device 20 may separate a service operation log from the log data, where the service operation log includes log records of service operation classes; at least one log record matching the at least one first target attribute field is extracted from the business operation log.

In this embodiment, the log management device 20 may sort the log data provided by the service end 10. And separates the service operation log from it.

In practical applications, the service operation log and the service system log may be distinguished by a difference of a special field value in the log data, a special identifier, and the like, for example, the log management device 20 may distinguish the service operation log and the service system log from the log data based on the identifier of the service operation log and the identifier of the service system log. Wherein, the service operation log comprises log records related to the service operation; and the service system log comprises log records related to system processing.

Accordingly, in the embodiment, the log records related to the business operation and the log records related to the system processing can be physically separated, and the two sets of log records are independent and do not influence each other.

Based on this, in this embodiment, the log management device 20 may only open the access right for the service operation log to the first computing end 30, and close the access right for the service system log by the first computing end 30, thereby avoiding the risk of leakage of unrelated data.

In this embodiment, the service operation log may be further sorted, and at least one log record including part or all of the first target attribute field is extracted from the service operation log. In this way, at least one log record matching at least one first target attribute field can be extracted from the full amount of log data step by step.

On the basis, according to a specified log format, structured arrangement can be performed on the attribute information corresponding to the first target attribute field contained in each log record, so as to generate and store an arranged log.

The designated log format may be a log format agreed by the log management device 20 and the first computing end 30, so that the first computing end 30 accesses the sorted log stored in the log management device 20 according to the agreed log format. The specified log format may be a json format or an xml format.

An exemplary log format sample may be:

{"logname":"query","logtime":"2020-07-14 17:25:20","project_name":

"recovery _ feature", "query": perfectly purchased "," user _ id ":192507 }.

The logname, logtime, project _ name are the first target attribute fields required by the first computing end 30. Query represents user search behavior, project _ name represents service line name, and query represents text information searched by the user.

Based on this, the log management device 20 may store the attribute information corresponding to the at least one first target attribute field in a designated log format. Through format conversion of the log record, the first computing end 30 can use the attribute information corresponding to at least one first target attribute field more conveniently, and the data acquisition complexity is reduced.

In this case, in this embodiment, the log management device 20 may only open the access right for the corresponding sorted log to the first computing end 30, and close the access right for the first computing end 30 to other types of logs. Similarly, the log management device 20 may also open access rights of the corresponding consolidated log to other computing terminals, and close access rights to other types of logs. Thereby, the risk of leakage of extraneous data is avoided.

In addition, in this embodiment, the log management device 20 may also generate corresponding sorted logs for other computing ends according to computing requirements of the other computing ends. And moreover, an access relation between the computing end and the sorted logs can be established, different access authorities for the sorted logs at the rear end are configured for each computing end, so that the sorted logs at different computing ends are isolated, and the data security is improved. In this way, in the process of performing the authority verification on the first computing terminal 30 in the foregoing, the authority verification can be performed from at least two dimensions of the access relationship and the log type of access.

In this embodiment, the attribute fields included in the sorted logs sorted by the log management device 20 may be kept synchronous with the computing requirements of the first computing end 30. If the log management device 20 determines that the at least one first target attribute field required by the first computing terminal 30 is increased or decreased, the attribute fields in the sorted log are synchronously increased or decreased. In practical applications, the first computing end 30 may send the second computing requirement to the log management device 20 when the attribute fields need to be increased or decreased, and carry the attribute fields needing to be increased or decreased or at least one second target attribute field in the second computing requirement. For the log management device 20, if the at least one second target attribute field is inconsistent with the at least one first target attribute field, the at least one first target attribute field may be updated with the at least one second target attribute field; or, the at least one first target attribute field may be updated according to the attribute field that needs to be increased or decreased and is carried in the second calculation requirement. And then, performing subsequent log record extraction, structured arrangement and other processes according to the updated at least one first target attribute field to obtain an accurate arranged log.

For example, the original format of the sorted log is { "name": zhang "," age ": 20" }, if the first computing end 30 wishes to collect the gender of the user again, the corresponding attribute field may be added to the sorted log, and the format of the sorted log is changed into: { "name": Zhang III "," age ": 20", "gender": male "}.

Therefore, the log management device 20 executes operations such as log data acquisition, sorting, storage and the like according to the actual calculation requirement of the first calculation terminal 30, and when the calculation requirement of the first calculation terminal 30 changes, the log management device 20 also synchronously and timely performs adaptive adjustment on operations such as log data acquisition, sorting, storage and the like, and the architecture of the whole data processing system does not need to change or rebuild. Thus, the data processing system can respond quickly to different data services and can seamlessly support new data services.

In the above or below embodiments, before acquiring the log data provided by the service end, the log management device 20 may further send at least one first target attribute field required by the first computing end 30 to the service end, so that the service end collects the log data at least including the at least one first target attribute field.

Under the condition that the requirements for data collection and transmission are stricter, the log management device 20 may further determine a requirement range specified by the first computing end 30 in at least one first target attribute field, where the requirement range is used to limit the value range of the attribute information; and sending the at least one first target attribute field and the requirement range to the service end 10, so that the service end 10 collects log data at least containing the requirement range specified under the at least one first target attribute field.

The first computing end 30 may specify a requirement range for part or all of the at least one first target attribute field. For example, the first computing end 30 may specify the demand range for the first target attribute field logtime within 30 days, may specify the demand range 1234-1456 for the first target attribute field user _ id, and so on.

The log management device 20 can configure the specified requirement ranges to the service end 10, and for the service end 10, log data can be collected according to the specified requirement ranges. For example, as mentioned above, the service end 10 may collect log data corresponding to the users 1234-1456 within 30 days, and provide the log data to the log management device 20, and certainly, the service end 10 may also collect other log data at the same time, which is not limited in this embodiment.

Accordingly, in this embodiment, the log management device 20 may synchronize the requirement range specified by the first computing end 30 to the service end 10, so that the service end 10 may collect log data according to the requirement range of the first computing end 30, which may meet the scene requirement with strict requirements on data collection and transmission, ensure the security of other data, and make the processing process of the log data more simplified.

Fig. 2 is a schematic diagram of an application scenario provided in an exemplary embodiment of the present application. With reference to fig. 2, the following description of the technical solution is made in conjunction with an application scenario.

The log management device 20 may acquire the log data provided by the service end 10 by consuming the kafka message or actively pulling data from the mysql of the service end 10, and divide the acquired log data into the service operation log and the service system log. Thereafter, a log record containing some or all of fields A, B, C, D and E is looked up from the business operations log.

Referring to fig. 2, the log management device may sort field values corresponding to the fields A, B, C, D and E included in the searched log record according to a json format to generate a sorted log in the json format, which is shown as the json log in fig. 2, open an access right of the json log to the computing end, and close an access right of the service system log to the computing end.

The computing end can extract the json log from the log management equipment in a flash mode. The calculation end can determine the operation behaviors corresponding to different log records according to the values under the logname field of the json log. And calculating the mode according to the data matched with the operation behaviors.

For logically complex computations, they can be done by flink (real-time), spark (offline). Such as: the query represents the search behavior of the user, and the computing terminal can simply analyze such log records into structured data and store the structured data in a corresponding hive table for subsequent computing. The log records { "name": three and "age": 20} can be stored as a table in hive, the name column stores three and the age column stores 20.

The result can be calculated through flink and stored in hbase.

Logical simplifications can be done by spark sql or flink sql. If tf-idf is calculated for user search behavior, where tf is needed in real time, idf may allow for a delay of (T + 1). For the tf, the calculation can be completed through flink, and the calculation process comprises word segmentation, word frequency calculation and hbase storage of calculation results; the calculation of idf can be done by spark.

The computing end can provide an API interface, the API interface can provide http or rpc services externally, and the requirement ranges under the attribute fields of different data service requirements can be not identical. Referring to fig. 2, the log management device may configure the requirement range under the attribute field of different data service requirements into the service end in a meta-information manner, so that the service end may generate log data as needed. Wherein, the requirement scope may include, but is not limited to, a date scope, a specific function object scope, a user scope, etc.

The data service can extract the calculation results from the hbase or the es of the calculation end, the calculation results are all calculated in advance by the calculation end, and the query performance can be guaranteed when the data volume is large. For example, tf-idf within 30 days of obtaining the user query information, rowkey in hbase is designed as follows: the date + [ tf | idf ] + user id + word segmentation keywords can be used for quickly querying data according to the prefix of the date + user id during query, the prefix removed by the rowkey is the keyword after word segmentation, and the word frequency corresponding to the relevant qualifier is the word frequency. For example, tf correlation data, rowkey is "2020-07-24 _ tf _1234_ perfect", qualifier is INFO: day30 is 3, and a user with id 1234 searches for 3 times containing 'perfect' in nearly 30 days.

An exemplary hbase table structure for storing tf and idf is:

therefore, by means of preparing required log records for the computing end through the log management device, application invasion to the service end can be effectively reduced, and potential risks are reduced, because the service end usually uses a cloud platform of a third party, and if the access right of the database is opened to achieve log data synchronization, the log data synchronization is feasible but has risks.

The log data are converted into a designated log format, and the log data are screened according to the calculation requirement of a calculation end, so that the use of a calculation platform can be facilitated;

the demand can be responded to quickly by relying on the existing computing side, and the data processing system architecture does not need to be changed greatly. The whole structure of the system is simple and easy to build.

Fig. 3 is a flowchart illustrating a data processing method according to an embodiment of the present application. Referring to fig. 3, the method includes:

step 300, determining at least one first target attribute field required by the first computing end according to the first computing requirement of the first computing end;

step 301, obtaining log data provided by a service end;

step 302, extracting at least one log record matched with at least one first target attribute field from log data;

and step 303, storing at least one log record for the first computing terminal to access the log record for data computing.

In step 300, at least one first target attribute field required by the first computing end may be determined according to the first computing requirement of the first computing end.

The first computing end can provide a plurality of service interfaces to the outside, and the first computing end can be an API interface in practical application. The data services corresponding to different service interfaces may not be identical, and accordingly, the log attribute fields required by different data services may not be identical. In this embodiment, the first computing end may perform data computation for each service interface, and therefore, when performing data computation for different service interfaces, the required log attribute fields may not be completely the same.

Based on this, in this embodiment, the description of the technical solution will be made by taking any one time of data calculation executed by the first calculation end as an example.

In this embodiment, at least one log attribute field required by the first computing end in the data computing process may be determined as the first target attribute field. For example, the first computing end executes the data computation, and two log attribute fields, namely "query" and "userID", are needed, so in this embodiment, it may be determined that the first target attribute field needed by the first computing end is "query" and "userID". In practical applications, the first computing end may generate the configuration information according to the required log attribute field, and in this embodiment, the configuration information may be acquired, so as to determine at least one first target attribute field required by the first computing end, which is not limited in this embodiment.

In this embodiment, in step 301, log data provided by the service end may be obtained. The method can adopt a kafka mode or a canel mode to acquire log data from a service end. Wherein, kafka is a distributed publish-subscribe message system, which can process all action stream data of the service end. In this way, the log management device can acquire the log data of the service end by consuming the kafka message. Canel is an open source project, is realized based on java, and monitors binlog log logs of mysql to acquire data in a mode of simulating slave becoming mysql. In this way, the log data can be pulled from the database corresponding to the service end.

Of course, other implementations may also be used to obtain log data from the service end, for example, obtaining related data by directly querying mysql. In addition, for the service end, the log access right can be opened to the log management device. The method can effectively ensure the application safety of the service end, and the service end is not subjected to the application intrusion risk of the first computing end any more.

In this embodiment, the log data typically includes a number of log records. In this embodiment, at least one log record matched with at least one first target attribute field may be extracted from the log data and stored. In practical application, the first target attribute field can be used as a search condition, a log record containing the first target attribute field is searched from log data, and the searched log record is stored. The log record includes attribute information corresponding to at least one first target attribute field. Of course, other attribute fields than the at least one first target attribute field may also be discarded from the located log record, for example, if one log record includes 30 fields, but only 25 of the fields are the first target attribute fields, only 25 first target attribute fields and their corresponding attribute information may be retained in the log record, and the other 5 fields and field values may be discarded. Of course, this embodiment does not limit this.

Accordingly, the first computing side can access at least one log record to perform data computation.

In this embodiment, the first computing end may also be configured with access rights to the stored at least one log record. In practical application, when an access request of a first computing end is received, authority verification can be performed on the first computing end, and the first computing end is allowed to access at least one log record under the condition that the first computing end passes the authority verification, so that the first computing end can obtain attribute information corresponding to at least one first target attribute field. Optionally, in this embodiment, the access right of the first computing end 30 to other types of logs than the corresponding at least one log record may be closed, so that by configuring the access right of the log record, the security of other data may be effectively ensured.

For the first computing end, the required log records can be extracted by way of flash and the like. Because the log records in the embodiment are sorted according to the calculation requirement of the first calculation end, the first calculation end can be used directly more conveniently.

On the basis, the first computing end can perform data computing by adopting spark real-time computing, spark stream, flink real-time computing and other modes, and stores computing results in the positions of hbase, es, mysql and the like.

In this embodiment, log data provided by a service end can be acquired; determining at least one first target attribute field required by a first computing end; and extracting and storing at least one log record matched with at least one first target attribute field from the log data so that the first computing terminal can access the at least one log record to perform data computation. Accordingly, in the embodiment, the service end and the first computing end do not directly communicate with each other, which can effectively reduce application intrusion to the service end; moreover, the log data provided by the service end can be collected, sorted and stored according to the calculation requirement of the first calculation end, so that the first calculation end can be more conveniently used, and the data collection complexity of the first calculation end is reduced.

In the above or below embodiments, the service operation log may be separated from the log data, and the service operation log includes log records of service operation classes; at least one log record matching the at least one first target attribute field is extracted from the business operation log.

In this embodiment, the log data provided by the service end may be sorted. And separates the service operation log from it.

In practical application, the service operation log and the service system log can be distinguished from the log data based on the identification of the service operation log and the identification of the service system log. Wherein, the service operation log comprises log records related to the service operation; and the service system log comprises log records related to system processing.

Based on this, in this embodiment, the access right for the service operation log may be opened only to the first computing end, and the access right for the service system log by the first computing end may be closed, so as to avoid a risk of irrelevant data leakage.

The designated log format may be a log format agreed by the log management device and the first computing terminal, so that the first computing terminal accesses the sorted log stored in the log management device according to the agreed log format. The specified log format may be a json format or an xml format.

In this embodiment, the attribute information corresponding to the at least one first target attribute field may be stored in a designated log format. Through format conversion of the log records, the first computing end can use the attribute information corresponding to at least one first target attribute field more conveniently, and the data acquisition complexity is reduced.

In this case, in this embodiment, the access right for the corresponding sorted log may be opened only to the first computing end, and the access right for the other types of logs by the first computing end may be closed. Similarly, the access right of the corresponding consolidated log can be opened for other computing terminals, and the access right of other types of logs can be closed. Thereby, the risk of leakage of extraneous data is avoided.

In addition, in this embodiment, corresponding sorted logs may be generated for other computing ends according to computing requirements of the other computing ends, respectively. And moreover, an access relation between the computing end and the sorted logs can be established, so that the sorted logs of different computing ends are isolated, and the data security is improved. In this way, in the process of performing the authority verification on the first computing side in the foregoing, the authority verification can be performed from at least two dimensions of the access relationship and the log type (log object) of the access.

In this embodiment, the attribute fields included in the sorted log may be kept synchronous with the computing requirements of the first computing end. And if the fact that the at least one first target attribute field required by the first computing end is increased or decreased is determined, the attribute fields in the sorted log are synchronously increased or decreased. In practical application, the first computing end may initiate a second computing requirement under the condition that the attribute field needs to be increased or decreased, and carry the attribute field needing to be increased or decreased or at least one second target attribute field in the second computing requirement. Thus, in this embodiment, if the at least one second target attribute field is inconsistent with the at least one first target attribute field, the at least one first target attribute field may be updated with the at least one second target attribute field; or, the at least one first target attribute field may be updated according to the attribute field that needs to be increased or decreased and is carried in the second calculation requirement. And then, performing subsequent log record extraction, structured arrangement and other processes according to the updated at least one first target attribute field to obtain an accurate arranged log.

According to the method, the operations such as log data acquisition, arrangement and storage can be executed according to the actual calculation requirement of the first calculation end, when the calculation requirement of the first calculation end changes, the operations such as log data acquisition, arrangement and storage can be synchronously and timely adjusted in a self-adaptive mode, and the framework of the whole data processing system does not need to change or rebuild. Thus, the data processing system can respond quickly to different data services and can seamlessly support new data services.

In the above or following embodiments, before obtaining the log data provided by the service end, at least one first target attribute field required by the first computing end may be sent to the service end, so that the service end collects the log data at least including the at least one first target attribute field.

Under the condition that the requirements for data collection and transmission are stricter, the requirement range specified by the first computing end under at least one first target attribute field can be determined, and the requirement range is used for limiting the value range of the attribute information; and sending the at least one first target attribute field and the requirement range to the service end so that the service end collects log data at least containing the requirement range specified under the at least one first target attribute field.

The first computing end can specify a requirement range for part or all of the at least one first target attribute field. For example, the first computing end may specify the demand range for the first target attribute field logtime within 30 days, may specify the demand range 1234-1456 for the first target attribute field user _ id, and so on.

In this embodiment, the specified requirement ranges may be configured to the service end, and for the service end, the log data may be collected according to the specified requirement ranges. For example, in the above example, the service end may collect log data corresponding to the users 1234- & 1456 within 30 days, and certainly, the service end may also collect other log data, which is not limited in this embodiment.

Accordingly, in this embodiment, the requirement range specified by the first computing end can be synchronized to the service end, so that the service end can collect log data according to the requirement range of the first computing end, which can meet the scene requirement with strict data collection and transmission requirements, and the processing process of the log data is simplified.

It should be noted that, for the sake of brevity, the above description of the technical details of the related embodiments of the data processing method may be referred to the description of the embodiments of the data processing system, which should not be repeated herein, but should not cause a loss of scope of the present application.

It should be noted that the execution subjects of the steps of the methods provided in the above embodiments may be the same device, or different devices may be used as the execution subjects of the methods. For example, the execution subjects of steps 301 to 303 may be device a; for another example, the execution subject of

steps

301 and 302 may be device a, and the execution subject of step 303 may be device B; and so on.

In addition, in some of the flows described in the above embodiments and the drawings, a plurality of operations are included in a specific order, but it should be clearly understood that the operations may be executed out of the order presented herein or in parallel, and the sequence numbers of the operations, such as 301, 302, etc., are merely used for distinguishing different operations, and the sequence numbers do not represent any execution order per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel.

The data processing method provided by the above embodiment can be executed by a log management device. Fig. 4 is a schematic structural diagram of a log management apparatus according to an exemplary embodiment of the present application, and referring to fig. 4, the log management apparatus may include: a first interaction module 40, a second interaction module 41 and a processing module 42.

The first interaction module 40 is configured to determine, according to a first computation requirement of the first computation end, at least one first target attribute field required by the first computation end;

the second interaction module 41 is configured to obtain log data provided by a service end;

a processing module 42, configured to extract at least one log record matching the at least one first target attribute field from the log data; and storing at least one log record for the first computing terminal to access the at least one log record for data computing.

In an alternative embodiment, the processing module 42, when extracting at least one log record matching the at least one first target attribute field from the log data, is configured to: separating a service operation log from log data, wherein the service operation log comprises log records of service operation classes; at least one log record matching the at least one first target attribute field is extracted from the business operation log.

In an optional embodiment, when the processing module 42 extracts at least one log record matching with the at least one first target attribute field from the business operation log, it is configured to: extracting at least one log record containing part or all of the first target attribute fields from the service operation log; and according to a specified log format, performing structured sorting on the attribute information corresponding to the first target attribute field contained in each log record to generate a sorted log.

In an alternative embodiment, the log format is in json format or xml format.

In an alternative embodiment, the processing module 42 is further configured to: determining at least one second target attribute field required by the first computing end according to the second computing requirement of the first computing end; and if the at least one second target attribute field is inconsistent with the at least one first target attribute field, updating the at least one first target attribute field with the at least one second target attribute field.

In an alternative embodiment, the processing module 42 is further configured to: respectively generating corresponding sorted logs for other computing ends according to computing requirements of other computing ends; and constructing an access relation between the computing end and the sorted logs so that the first computing end can access the sorted logs corresponding to the first computing end to obtain the attribute information meeting the computing requirements of the first computing end.

In an alternative embodiment, processor 42 is further configured to: and closing the access rights of the first computing terminal and other computing terminals to other types of logs except the sorted logs.

In an optional embodiment, when acquiring the log data provided by the service end, the first interaction module 40 is configured to: consuming the kafka message provided by the service end to obtain log data; or, the log data is pulled from the database corresponding to the service end.

In an optional embodiment, the processing module 42 is further configured to, before acquiring the log data provided by the service end: determining a requirement range appointed by a first computing end under at least one first target attribute field according to a first computing requirement, wherein the requirement range is used for limiting a value range of attribute information; and sending the at least one target attribute field and the requirement range to a service end so that the service end collects log data at least comprising the requirement range specified under the at least one first target attribute field.

It should be noted that, for the sake of brevity, the above-mentioned technical details regarding the related embodiments of the log management apparatus may refer to the description of the embodiments of the data processing system, which should not be repeated herein, but should not cause a loss of scope of the present application.

The log management apparatus in the above embodiments may be implemented as software or as a combination of software and hardware, and may be integrally provided in the computing device. Fig. 5 is a schematic structural diagram of a computing device according to an exemplary embodiment of the present application. As shown in fig. 5, the computing device includes: memory 50, processor 51 and communication component 52.

Memory 50 is used to store computer programs and may be configured to store other various data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phonebook data, messages, pictures, videos, and so forth.

The memory 50 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 51, coupled to the memory 50 and the communication component 52, for executing computer programs in the memory 50 for: determining at least one target attribute field required by a first computing end according to a first computing requirement of the first computing end; acquiring log data provided by a service end through a communication component 52; extracting at least one log record matched with at least one first target attribute field from log data; and storing at least one log record for the first computing terminal to access the at least one log record for data computing.

In an alternative embodiment, the processor 51, when extracting from the log data at least one log record matching the at least one first target attribute field, is configured to: separating a service operation log from log data, wherein the service operation log comprises log records of service operation classes; at least one log record matching the at least one first target attribute field is extracted from the business operation log.

In an alternative embodiment, the processor 51, when extracting at least one log record matching with the at least one first target attribute field from the business operation log, is configured to: extracting at least one log record containing part or all of the first target attribute fields from the service operation log; and according to a specified log format, performing structured sorting on the attribute information corresponding to the first target attribute field contained in each log record to generate a sorted log.

In an alternative embodiment, the log format is in json format or xml format.

In an alternative embodiment, the processor 51 is further configured to: determining at least one second target attribute field required by the first computing end according to the second computing requirement of the first computing end; and if the at least one second target attribute field is inconsistent with the at least one first target attribute field, updating the at least one first target attribute field with the at least one second target attribute field.

In an alternative embodiment, the processor 51 is further configured to: respectively generating corresponding sorted logs for other computing ends according to computing requirements of other computing ends; and constructing an access relation between the computing end and the sorted logs so that the first computing end can access the sorted logs corresponding to the first computing end to obtain the attribute information meeting the computing requirements of the first computing end.

In an alternative embodiment, the processor 51 is further configured to: and closing the access rights of the first computing terminal and other computing terminals to other types of logs except the sorted logs.

In an alternative embodiment, the processor 51, when acquiring the log data provided by the service end, is configured to: consuming the kafka message provided by the service end to obtain log data; or, the log data is pulled from the database corresponding to the service end.

In an alternative embodiment, the processor 51 is further configured to, before acquiring the log data provided by the service end, include: determining a requirement range appointed by a first computing end under at least one first target attribute field according to a first computing requirement, wherein the requirement range is used for limiting a value range of attribute information; and sending the at least one target attribute field and the requirement range to a service end so that the service end collects log data at least comprising the requirement range specified under the at least one first target attribute field.

It should be noted that, for the sake of brevity, the above description of the technical details of the embodiments related to the computing device may be referred to in the description of the embodiments of the data processing system, which should not be repeated herein, but should not cause a loss of scope of the present application.

Further, as shown in fig. 5, the computing device further includes: power supply components 53, and the like. Only some of the components are schematically shown in fig. 5, and the computing device is not meant to include only the components shown in fig. 5.

Accordingly, the present application further provides a computer-readable storage medium storing a computer program, where the computer program can implement the steps that can be executed by a computing device in the foregoing method embodiments when executed.

The communication component in fig. 5 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a WiFi, a 2G, 3G, 4G/LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

The power supply assembly of fig. 5 described above provides power to the various components of the device in which the power supply assembly is located. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power component is located.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, comprising:

acquiring log data provided by a service end;

2. The method of claim 1, wherein the extracting at least one log record from the log data that matches the at least one first target attribute field comprises:

separating a service operation log from the log data, wherein the service operation log comprises log records of service operation classes;

and extracting at least one log record matched with the at least one first target attribute field from the business operation log.

3. The method of claim 2, wherein the extracting at least one log record matching the at least one first target attribute field from the business operation log comprises:

extracting at least one log record containing part or all of the first target attribute fields from the service operation log;

and according to a specified log format, performing structured arrangement on the attribute information corresponding to the first target attribute field contained in each log record to generate an arranged log.

4. The method of claim 3, further comprising:

determining at least one second target attribute field required by the first computing end according to a second computing requirement of the first computing end;

and if the at least one second target attribute field is not consistent with the at least one first target attribute field, updating the at least one first target attribute field by using the at least one second target attribute field.

5. The method of claim 3, further comprising:

respectively generating corresponding sorted logs for other computing ends according to computing requirements of other computing ends;

and constructing an access relation between the computing end and the sorted logs so that the first computing end can access the sorted logs corresponding to the first computing end to obtain attribute information meeting the computing requirements of the first computing end.

6. The method of claim 5, further comprising:

and closing the access rights of the first computing terminal and the other computing terminals to other types of logs except the sorted logs.

7. The method of claim 1, wherein before obtaining the log data provided by the service end, the method further comprises:

determining a requirement range appointed by the first computing end under the at least one first target attribute field according to the first computing requirement, wherein the requirement range is used for limiting a value range of attribute information;

and sending the at least one target attribute field and the requirement range to the service end so that the service end collects log data at least containing the requirement range specified under the at least one first target attribute field.

8. A log management apparatus, comprising:

9. A computing device comprising a memory, a processor, and a communication component;

the memory is to store one or more computer instructions;

10. A data processing system, comprising: the system comprises a service end, a log management device and a first computing end, wherein the log management device is respectively in communication connection with the service end and the first computing end;

11. A computer-readable storage medium storing computer instructions, which when executed by one or more processors, cause the one or more processors to perform the data processing method of any one of claims 1-7.