CN115098738A

CN115098738A - Service data extraction method and device, storage medium and electronic equipment

Info

Publication number: CN115098738A
Application number: CN202210737128.0A
Authority: CN
Inventors: 韩金涛; 李江涛; 卢永安; 马坤
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-06-27
Filing date: 2022-06-27
Publication date: 2022-09-23

Abstract

The application discloses a method and a device for extracting service data, a storage medium and electronic equipment. The method comprises the steps of obtaining a query request, wherein the query request is used for requesting to query business data with target business attributes; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation; screening a target log file corresponding to a target service from a log file set, wherein the log file set comprises log files generated by executing the service; the service data are extracted from the target log file to obtain the target service data, wherein the target service data are the service data with the target service attribute.

Description

Service data extraction method and device, storage medium and electronic equipment

Technical Field

The present application relates to the field of big data, and in particular, to a method and an apparatus for extracting service data, a storage medium, and an electronic device.

Background

With the rapid development of internet technology, more and more enterprises or service organizations provide business services for users through a network, and more business data are generated randomly, under the circumstance, data value mining is promoted, and data business service support becomes an important work task of the enterprises or service organizations, currently, the enterprises commonly use a way of constructing a data query program for querying different data, namely constructing a query program for querying specific business data, and program information for executing business is written down in the data query program, so that when querying business data, business data is queried by running a corresponding query program, but the way is highly limited, and is not beneficial to flexible expansion of data query, for example, when newly adding statistical logic or newly adding business functions, a new query program needs to be written, the problems of high maintenance cost, poor expansibility and the like of the program query program are caused.

Aiming at the problems of low query efficiency of business data in the related technology and the like, an effective solution is not provided yet.

Disclosure of Invention

The present application mainly aims to provide a method, an apparatus, a storage medium, and an electronic device for extracting service data, so as to solve the problem of low query efficiency of service data in the related art.

In order to achieve the above object, according to an aspect of the present application, a method for extracting service data is provided. The method comprises the following steps: acquiring a query request, wherein the query request is used for requesting to query service data with a target service attribute; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relationship; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

Optionally, the obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service having the corresponding relationship includes: matching a target sub-attribute set with a reference sub-attribute set and each reference sub-attribute set in a reference program identifier, wherein the reference sub-attribute set comprises a plurality of reference service sub-attributes, the plurality of reference service sub-attributes are used for dividing service types of services from a plurality of dimensions, and the target service attributes comprise the target sub-attribute set; and acquiring a target program identifier corresponding to a reference sub-attribute set successfully matched with the target sub-attribute set, wherein the target program identifier is used for uniquely identifying the target service.

Optionally, the matching the target sub-attribute set with each reference sub-attribute set in the reference sub-attribute set and the reference program identifier having a corresponding relationship includes: determining whether a target attribute value of each target business sub-attribute in one or more target business sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference business sub-attribute of a corresponding dimension in each reference sub-attribute set; and determining the reference sub-attribute set with the reference attribute value consistent with the target attribute value as the reference sub-attribute set successfully matched with the target sub-attribute set.

Optionally, the determining whether a target attribute value of each target service sub-attribute of the one or more target service sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference service sub-attribute of a corresponding dimension in each reference sub-attribute set includes: under the condition that the one or more target service sub-attributes comprise a target department type identifier, acquiring a candidate sub-attribute set with a reference identifier value of the reference department type identifier consistent with a target identifier value of the target department type identifier from the reference sub-attribute set; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding candidate identification bit on a candidate service identification string included in each candidate sub-attribute set, wherein the target service identification string includes at least one of: logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding reference identification bit on a reference service identification string included in each reference sub-attribute set or not under the condition that the target department type identification is not included in the one or more target service sub-attributes, wherein the target service identification string includes at least one of the following: the method comprises the steps of logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier.

Optionally, the extracting service data from the target log file to obtain target service data includes: generating a target data extraction script according to a target field corresponding to the target service attribute, wherein the target data extraction script is used for running a program for extracting data corresponding to the target field; and running the target data extraction script on the target log file to obtain the target service data.

Optionally, the generating a target data extraction script according to the target field corresponding to the target service attribute includes: acquiring an initial data extraction script, wherein the initial data extraction script records a program template for extracting data; and editing the target field to a corresponding position in the initial data extraction script to obtain the target data extraction script.

Optionally, before the generating the target data extraction script according to the target field corresponding to the target service attribute, the method further includes one of: extracting a target field of the target service attribute from the query request; and extracting a field of the reference service attribute corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation as the target field.

In order to achieve the above object, according to another aspect of the present application, there is provided an extraction apparatus of service data. The device includes: the system comprises a first acquisition module, a first service module and a second acquisition module, wherein the first acquisition module is used for acquiring a query request, and the query request is used for requesting to query service data with target service attributes; the second acquisition module is used for acquiring the target service corresponding to the target service attribute from the reference service attribute and the reference service which have the corresponding relation; the screening module is used for screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and the first extraction module is used for extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

To achieve the above object, according to another aspect of the present application, there is provided a non-volatile storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform any one of the above methods.

To achieve the above object, according to another aspect of the present application, there is provided an electronic device. The electronic device includes: one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any of the above-described methods.

Through the application, the following steps are adopted: acquiring a query request, wherein the query request is used for requesting to query service data with a target service attribute; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute, namely the reference service attribute and the reference service have a corresponding relation, the query request is used for requesting to query the service data with the target service attribute, the target service attribute of the service data to be queried can be known through the query request, further a target service to be corresponding to the target service attribute can be obtained from the reference service attribute and the reference service with the corresponding relation, namely the target service for generating the service data to be queried is determined through the target service attribute, further the target service data is extracted from the target log file of the target service, the service data to be queried only needs to be known in the query, and the service is queried through the target service attribute, and further extracting target service data corresponding to the service, thereby realizing flexibility and expandability of service data query. By adopting the technical scheme, the problems of low query efficiency of the business data and the like in the related technology are solved, and the technical effect of improving the query efficiency of the business data is realized.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:

fig. 1 is a flowchart of a method for extracting service data according to an embodiment of the present application;

FIG. 2 is a schematic illustration of an alternative query request selection interface according to an embodiment of the present application;

FIG. 3 is an alternative reference business sub-attribute set comparison representation intent according to an embodiment of the present application

Fig. 4 is a flow chart of an alternative service data extraction according to an embodiment of the present application;

fig. 5 is a schematic diagram of a service data extraction apparatus according to an embodiment of the present application;

fig. 6 is a schematic hardware environment diagram of a method for extracting service data according to an embodiment of the present application.

Detailed Description

It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be noted that relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are information and data that are authorized by the user or sufficiently authorized by various parties. For example, an interface is provided between the system and the relevant user or organization, before obtaining the relevant information, an obtaining request needs to be sent to the user or organization through the interface, and after receiving the consent information fed back by the user or organization, the relevant information is obtained.

The scheme can be applied to enterprises or service organizations with statistical requirements on user data, such as banks, hospitals, insurance companies and the like, the enterprises have business data of different users, the data relate to different businesses, the business types of the enterprises have variability, and the statistical requirements also have variability when the data are subjected to statistics.

The present invention is described below with reference to preferred implementation steps, and fig. 1 is a flowchart of a method for extracting service data according to an embodiment of the present application, and as shown in fig. 1, the method includes the following steps:

step S102, obtaining a query request, wherein the query request is used for requesting to query service data with target service attributes;

step S104, obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relationship;

step S106, screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service;

step S108, extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

In the technical solution provided in step S102, the query request carries related information of the target service attribute, and the query request may but is not limited to carry a name and a corresponding attribute value of the target service attribute, and identification information (e.g., a character string, where different character bits in the character string represent attribute values) corresponding to the target service attribute.

Optionally, in this embodiment, the target service attribute may include one or more sub-attributes, and different service attributes are used to divide the service types of the services from different dimensions.

Optionally, in this embodiment, the query request may also carry a data query time condition, that is, service data is queried for a certain period of time, and the log file may be screened according to the data query time condition when the data query is performed.

Optionally, in this embodiment, the query request may be generated according to a service attribute clicked by a data query staff on a front-segment interface, where the front-segment interface is labeled with a corresponding service attribute, and a user performs selection on a front-end interface according to data required by the user, so as to select a target service attribute, fig. 2 is an optional query request selection interface schematic diagram according to the embodiment of the present application, and as shown in fig. 2, the query request selection interface schematic diagram may include, but is not limited to, a login service identifier, a dynamic account service identifier, a service dynamic service identifier, a thread service identifier, an interface service identifier, a logout service identifier, and other sub-service attribute identifiers, and an operator clicks a corresponding attribute identifier according to a data query requirement.

In the technical solution provided in step S104, the reference service attribute having a corresponding relationship with the reference service may be a corresponding relationship between the reference service attribute and the reference service name, or may also be a corresponding relationship between the reference service attribute and a reference program identifier, where the reference program identifier is used to indicate information of a program executing the corresponding reference service, or may also be a corresponding relationship between a plurality of reference sub-attribute sets included in the reference service attribute and the reference service name.

Optionally, in this embodiment, the manner of obtaining the target service corresponding to the target service attribute may be to determine, as the target service, the reference service corresponding to the reference service attribute that is consistent with the target service attribute by determining whether each reference service attribute is consistent with the target service attribute in matching, for example, determine, in multiple reference service attributes, the reference service attribute that is consistent with the attribute name of the target service attribute in matching, or may also determine, in multiple reference service attributes, the reference service attribute that is consistent with the attribute value of the dimensional attribute corresponding to the target service attribute in matching, which is not limited in this scheme.

Optionally, in this embodiment, when a service is newly added, the corresponding relationship between the reference service attribute and the reference service needs to be maintained according to the service attribute of the service, that is, the corresponding relationship between the newly added service attribute and the service is updated to the corresponding relationship between the reference service attribute and the reference service having the corresponding relationship.

In the technical solution provided in step S106, the method for screening out the target log file in the log file set may be a method for searching for identification information corresponding to the target service, for example, the identification information may be identification information used for indicating a service name, or may also be an identification used for indicating a program corresponding to the execution service, for example, the log file may carry a service name of the target service (such as keywords like transfer, data transmission, and the like), and the log file set is searched for the log file containing the service name, so as to obtain the target log file, or the log file may also carry program information of the program (such as a program name, a program serial number, and the like), and the log file set is searched for the log file containing the program information, so as to obtain the target log file.

Optionally, in this embodiment, when the log files of the program are stored again, the log files are classified and stored according to the names of the programs or the functions of the programs, for example, the programs with different names are stored in the same storage space, the programs with different functions are stored in the same storage space, the screening of the target log file may also call the corresponding storage space according to the names of the programs and the functions of the programs, and all the log files stored in the storage space are used as the target log file.

In the technical solution provided in step S108, the manner of extracting the service data may be by using a script program with a data extraction function, where the script program may be pre-constructed, or may also be generated according to the target service attribute, for example, a keyword for the target service attribute is input into an initial script program, where the initial script program is a program with a basic function of data extraction, and the keyword for the target service attribute is inserted into a corresponding position in the initial script program, so as to obtain the script program for extracting the target service data.

It should be noted that, in the related art, a data query program for querying different data is constructed, that is, a query program for querying specific service data is constructed, and program information of an execution service is written in the data query program, so that when the service data is queried, the service data is queried by operating a corresponding query program, but this method has strong limitation and is not favorable for flexible expansion of data query.

Through the steps, the reference service attribute and the reference service have a corresponding relation, the query request is used for requesting to query the service data with the target service attribute, the target service attribute of the service data to be queried can be known through the query request, and then the target service to be corresponding to the target service attribute can be obtained from the reference service attribute and the reference service with the corresponding relation, namely the target service generating the service data to be queried is determined through the target service attribute, and then the target service data is extracted from a target log file of the target service. By adopting the technical scheme, the problems of low query efficiency of the business data and the like in the related technology are solved, and the technical effect of improving the query efficiency of the business data is realized.

As an optional embodiment, the obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service having the corresponding relationship includes:

matching a target sub-attribute set with a reference sub-attribute set and each reference sub-attribute set in a reference program identifier, wherein the reference sub-attribute set comprises a plurality of reference service sub-attributes, the plurality of reference service sub-attributes are used for dividing service types of services from a plurality of dimensions, and the target service attributes comprise the target sub-attribute set;

and acquiring a target program identifier corresponding to a reference sub-attribute set successfully matched with the target sub-attribute set, wherein the target program identifier is used for uniquely identifying the target service.

Optionally, in this embodiment, the program identifier is information for identifying a program, and the program identifier may include, but is not limited to, a program name, a program serial number, and the like, and marks the program by using the program identifier, so that the program has uniqueness, that is, one program identifier uniquely corresponds to one program, that is, one program identifier intelligently finds one corresponding program, and further, one target service can be uniquely identified by using one target program identifier.

Optionally, in this embodiment, in order to ensure uniqueness of the program identifier, but not limited to, the program identifier is generated by using a program identifier generation model, when a program of a newly added service is constructed, the program needs to be registered online, a program construction requirement (and a service attribute) when the program is constructed is imported into the program identifier generation model, the program identifier generation model queries a sequence table, obtains a current sequence value of the program identifier, and returns the program identifier on the basis of +1, where the sequence table pub _ sequence has a table structure as follows: SEQNAME: a sequence name; SEQ VALUE: a current sequence value; MINVALUE: a minimum value; MAXVALUE: a maximum value; STEP: the step length is counted each time. The program identifies the minimum value starting from 100001 and being at most 999999, increasing by 1 per step, i.e.: if a new program identifier is added at this time, and the current sequence value is 100001, the newly generated program identifier is 100002, and the current sequence value is updated to be 100002. And finally, maintaining a corresponding relation table of the reference service attribute and the reference service according to the returned program identifier and the imported service attribute.

Optionally, in this embodiment, the program identifier corresponds to a program for executing a service having a corresponding service attribute.

Optionally, in this embodiment, one service may have service sub-attributes with multiple dimensions, for example, taking a bank transfer service as an example, the service sub-attributes may be divided into an online service and an offline service according to a transaction mode, into a head office service and a branch office service according to a transaction position, and into a large amount service and a small amount service according to an amount of money.

Optionally, in this embodiment, for each target sub-attribute in the target sub-attribute set, if there is a reference sub-attribute in the reference attribute set that matches with the target sub-attribute, it may be determined that the target sub-attribute set and the reference sub-attribute set match successfully, and the number of reference sub-attributes in the reference sub-attribute set that matches with the target sub-attribute set successfully may be equal to the number of target sub-attributes in the target sub-attribute set, or may be more than the number of target sub-attributes in the target sub-attribute set, which is not limited in this embodiment.

As an optional embodiment, the matching the target sub-attribute set with the reference sub-attribute set having the corresponding relationship and each reference sub-attribute set in the reference program identifier includes:

determining whether a target attribute value of each target business sub-attribute in one or more target business sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference business sub-attribute of a corresponding dimension in each reference sub-attribute set;

and determining the reference sub-attribute set with the reference attribute value consistent with the target attribute value as the reference sub-attribute set successfully matched with the target sub-attribute set.

Optionally, in this embodiment, the attribute value is used to indicate whether the attribute exists, a value of the attribute value may be, but is not limited to, a value of "0", "1", or a null value, where "0" indicates that the attribute does not exist, "1" indicates that the attribute exists, and a null value indicates that the attribute may or may not exist, for example, a target attribute value of a target service sub-attribute is controlled, the target service sub-attribute may be matched with a reference service sub-attribute of a corresponding dimension that takes a value of "0" or "1", or the attribute value may also take a specific value, for example, when the service sub-attribute is used to indicate a handling department for handling the service, the attribute value may be a department number of the department.

As an optional embodiment, the determining whether a target attribute value of each target business sub-attribute of the one or more target business sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference business sub-attribute of a corresponding dimension in each reference sub-attribute set includes:

under the condition that the one or more target service sub-attributes comprise a target department type identifier, acquiring a candidate sub-attribute set with a reference identifier value of the reference department type identifier consistent with a target identifier value of the target department type identifier from the reference sub-attribute set; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding candidate identification bit on a candidate service identification string included in each candidate sub-attribute set, wherein the target service identification string includes at least one of: logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier;

determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding reference identification bit on a reference service identification string included in each reference sub-attribute set or not under the condition that the target department type identification is not included in the one or more target service sub-attributes, wherein the target service identification string includes at least one of the following: the system comprises a login service identifier, a dynamic account service identifier, a service dynamic user service identifier, a thread service identifier, an interface service identifier and a logout service identifier.

Optionally, in this embodiment, the department type identifier is used to indicate information of a handling department of the business, taking the business as a banking business as an example, the department type identifier may be used to indicate whether the handled business is a general or branch business, or a loan department or a financing department, where the identifier value may be "0", "1", a null value, or a specific department number.

Optionally, in this embodiment, the login service identifier is used to identify whether the user logs in the service for the account, for example, the user logs in a program corresponding to the service using an account registered by the user.

Optionally, in this embodiment, the action-based service identifier is used to indicate whether the service has a conversion of virtual resources, for example, taking a bank transfer service as an example, the user initiating a transfer transaction or accepting a transfer from another user may become an action-based service.

Optionally, in this embodiment, the service account identifier is used to indicate whether there is a new addition of the user account, for example, the user opens an account in a bank or registers an account with an application.

Optionally, in this embodiment, the thread service identifier is used to indicate whether the service triggers a service of a preset thread program, for example, if the user consumes or transfers money, the thread program that triggers a short message notification is triggered as virtual resources increase or decrease, so that the service is a thread service.

Optionally, in this embodiment, the interface service identifier is used to indicate whether there is interface call of another program during service execution, for example, taking bank transfer as an example, when a user transfers money, user identity authentication is performed, at this time, an interface of a user identity authentication program needs to be called, and the service is an interface service.

The reference service attribute set and the reference service with corresponding relationship comprise a reference service sub-attribute set and a reference program identifier with corresponding relationship, the reference service sub-attribute set comprises a plurality of reference service sub-attributes and attribute values of the reference service sub-attributes, when comparing, the target service sub-attribute and the attribute value of the corresponding reference service sub-attribute are compared, fig. 3 is an optional reference service sub-attribute set comparison representation intention according to the embodiment of the application, as shown in fig. 3, the table correspondingly stores the attribute value of the reference service sub-attribute and the reference program identifier, the reference program identifier is used for executing the program executing the corresponding service, the attribute value of the reference service sub-attribute is used for dividing the service type of the service from a plurality of dimensions, and the reference service sub-attribute can include but is not limited to a reference department type identifier and a login service identifier, the reference service sub-attribute may further include a service column identifier, where the service column identifier is used to indicate a column of the service (such as a transfer service, a financing service, and the like), in the table, characters "0" and "1" indicate values of the attribute, "0" indicates that the attribute is not present, "1" indicates that the attribute is present, and when a service or a program is newly added, the reference service sub-attribute set comparison table needs to be maintained, the program identifier of the newly added program is added to the table, and the service attribute of the service executed by the program is correspondingly added to the table.

As an optional embodiment, the extracting the service data from the target log file to obtain the target service data includes:

generating a target data extraction script according to a target field corresponding to the target service attribute, wherein the target data extraction script is used for running a program for extracting data corresponding to the target field;

and running the target data extraction script on the target log file to obtain the target service data.

Optionally, in this embodiment, the target field is information for indicating a target service attribute, the target field may be an attribute name of the target service attribute or an encoding of the target service attribute, the field is carried in a log file of a corresponding service, and then the corresponding service data can be found in the log file through the target field.

Optionally, in this embodiment, the target data extraction script may be a pre-constructed data extraction script, or may also be a target data extraction script according to the target field.

As an optional embodiment, the generating a target data extraction script according to the target field corresponding to the target service attribute includes:

acquiring an initial data extraction script, wherein the initial data extraction script records a program template for extracting data;

and editing the target field to a corresponding position in the initial data extraction script to obtain the target data extraction script.

Optionally, in the present embodiment, the initial data extraction script may be, but is not limited to, a query statement having a data query function.

Optionally, in this embodiment, the initial data extraction script may be pre-stored, or may also be obtained by editing a target data extraction script having the same data extraction function.

As an optional embodiment, before the generating a target data extraction script according to a target field corresponding to the target service attribute, the method further includes one of:

extracting a target field of the target service attribute from the query request;

and extracting a field of the reference service attribute corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation as the target field.

Optionally, in this embodiment, the query request may be a target field carrying a target service attribute, or may also be a character string carrying an attribute value of a service attribute, where each character bit in the character string corresponds to an attribute value of an attribute, a field having a reference service attribute and a corresponding attribute value in a reference service and a reference service having a corresponding relationship, and a field having a reference service attribute and a corresponding attribute value in the reference service, and a field having a reference service attribute corresponding to the character bit is found as the target field according to a character in the character string.

In order to conveniently and accurately screen the service data of the user and acquire the service information of the related column, the scheme can also use a data lake mode, and can automatically generate a data query strategy according to the service attribute of the service data to be queried by constructing the corresponding relation between the reference service attribute and the reference service, sinking the corresponding relation between the reference service attribute and the reference service into the data lake, sinking the service log in which the user uses a program to handle the service into the data lake, and calling a log file corresponding to the service to query the data from the log file. Fig. 4 is a flow chart of an alternative business data extraction process according to an embodiment of the present application, which may be applied to statistics of banking data, but is not limited to the following steps, as shown in fig. 4:

s401, constructing a corresponding relation between a reference service attribute and a reference service, wherein the corresponding relation can be a reference program identifier and a reference sub-attribute set with the corresponding relation, the reference sub-attribute set comprises a plurality of reference service sub-attributes and is used for dividing service types of the service from a plurality of dimensions, the reference service sub-attributes comprise branch characteristic service identifiers (1 is a branch characteristic transaction, 0 is a general transaction), a transaction identifier string (FLAGS, a first login service identifier 1-is, 0-NO, a second login service identifier 1-is, 0-NO, a third operation account type service identifier 1-is, 0-NO, a fourth thread type account type service identifier 1-is, 0-NO, a fifth interface type account type service identifier 1-is, 0-NO, a sixth login account type service identifier 1-is, 0-no.

S402, downloading a personal transaction log table B to a data lake, wherein the table B comprises the following fields: loggerialno: log running number; LOGDATE: log date, time; TRANCODE: a transaction code; the USERID: user id/account number; AREACODE: a region code; INACCTNO: transferring to an account; OUTACTNO: transferring out the account number; NETCODE: a website number; AMOUNT: a transaction amount; CURTYPE: the currency type; FMENUID: numbering the first-level columns; MENUID: numbering last-stage columns; FEEFAVER: a discount rate of commission fees; RATEFAVER: the rate of exchange; INCURTYPE: transferring into currency; SESSIONID: a customer; TRADEDATE: the online banking transaction date; RATEFAG: positive and negative exchange rate marks; TRANSERIALNO: a transaction number; outproducted: transferring out the product type; OUTPUTRONTNO: transferring out the product serial number; outsubpump: transferring out the sub-product types; outsubroductno: transferring out the serial number of the sub-product; inproducttd: transferring into product types; INPRODUCTNO: transferring the product serial number; INSUBPRODUCTED: transferring into a child product category; INSUBPRODUCTNO: transferring the product number to a secondary product; SUBMENUID: column branch number; IPADR: an IP address IPADR; MACADDR: a MAC address; USERAGENT: a user agent; TRADECOUNT: transaction number; langcode: a language identification; subbankflag: identifying lines; SUMMARY: description is given; custype: authenticating the media type; CHANNELID: a channel unique identifier; CURTYPE _ PRCETYP: the type of the currency and the price; INCURTYPE _ PRCETYP: transferring to a currency price type; TEMP: a spare field; TEMP 1: a spare field 1; TEMP 2: a spare field 2; (ii) a TEMP 3: a spare field 3; ATTESTWAY: an authentication mode; mscount complex: the short message authenticates the country code; MSPHONE: the short message authenticates the mobile phone number; SERVERIP: a server IP; PODNAME: PAAS container name; SERVERTYPE: the server type.

S403, acquiring the target service attribute of the service data requested to be inquired carried in the data inquiry request.

S404, matching the target service attribute with the plurality of reference service attributes, taking the reference program identifier corresponding to the reference service attribute which is matched with the target service attribute as the target program identifier, thereby finding the target service corresponding to the target program identifier and calling the target log file corresponding to the target service.

S405, extracting the target data in the target log file using the target data extraction script.

The data query mode in the process adopts direct data lake query, the data lake and the service database are not the same database instance, the database generating the service data is synchronized with the data of the data lake through daily final day batch, namely, a program is used for generating a file according to the corresponding relation between the newly added reference service attribute and the reference service or the data of the service log file on the same day and sending the file to the data lake for updating.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.

The embodiment of the present application further provides a device for extracting service data, and it should be noted that the device for extracting service data according to the embodiment of the present application may be used to execute the method for extracting service data provided in the embodiment of the present application. The following describes an apparatus for extracting service data provided in an embodiment of the present application.

Fig. 5 is a schematic diagram of a device for extracting service data according to an embodiment of the present application. As shown in fig. 5, the apparatus includes: a first obtaining module 502, a second obtaining module 504, a screening module 506, and a first extracting module 508, wherein:

a first obtaining module 502, configured to obtain a query request, where the query request is used to request to query service data with a target service attribute; the second obtaining module 504 is connected to the first obtaining module 502, and is configured to obtain a target service corresponding to the target service attribute from the reference service attribute and the reference service having a corresponding relationship; the screening module 506 is connected to the second obtaining module 504, and configured to screen a target log file corresponding to the target service from a log file set, where the log file set includes log files generated by executing the service; the first extracting module 508 is connected to the screening module 506, and is configured to extract service data from the target log file to obtain target service data, where the target service data is service data with the target service attribute.

Optionally, in the apparatus for extracting service data provided in this embodiment of the present application, the reference service attribute and the reference service have a corresponding relationship, the query request is used to request to query the service data having the target service attribute, the target service attribute of the service data to be queried can be known through the query request, thereby obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service having the corresponding relationship, namely, the target service for generating the service data to be inquired is determined through the target service attribute, the target service data is extracted from the target log file of the target service, the service is inquired through the target service attribute only by knowing the target service attribute of the service data to be inquired when the inquiry is carried out, and further extracting target service data corresponding to the service, thereby realizing flexibility and expandability of service data query. By adopting the technical scheme, the problems of low query efficiency of the business data in the related technology and the like are solved, and the technical effect of improving the query efficiency of the business data is realized.

Optionally, the second obtaining module includes: a matching unit, configured to match a target sub-attribute set with a reference sub-attribute set having a corresponding relationship and each reference sub-attribute set in a reference program identifier, where the reference sub-attribute set includes multiple reference service sub-attributes, the multiple reference service sub-attributes are used to divide service types of services from multiple dimensions, and the target service attribute includes the target sub-attribute set; and the acquisition unit is used for acquiring a target program identifier corresponding to the reference sub-attribute set successfully matched with the target sub-attribute set, wherein the target program identifier is used for uniquely identifying the target service.

Optionally, the matching unit is configured to: determining whether a target attribute value of each target business sub-attribute in one or more target business sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference business sub-attribute of a corresponding dimension in each reference sub-attribute set; and determining the reference sub-attribute set with the reference attribute value consistent with the target attribute value as the reference sub-attribute set successfully matched with the target sub-attribute set.

Optionally, the matching unit is configured to: under the condition that the one or more target service sub-attributes comprise a target department type identifier, acquiring a candidate sub-attribute set with a reference identifier value of the reference department type identifier consistent with a target identifier value of the target department type identifier from the reference sub-attribute set; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding candidate identification bit on a candidate service identification string included in each candidate sub-attribute set, wherein the target service identification string includes at least one of: logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding reference identification bit on a reference service identification string included in each reference sub-attribute set or not under the condition that the target department type identification is not included in the one or more target service sub-attributes, wherein the target service identification string includes at least one of the following: the method comprises the steps of logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier.

Optionally, the first extracting module includes: a generating unit, configured to generate a target data extraction script according to a target field corresponding to the target service attribute, where the target data extraction script is used to run a program for extracting data corresponding to the target field; and the running unit is used for running the target data extraction script on the target log file to obtain the target service data.

Optionally, the generating unit is configured to: acquiring an initial data extraction script, wherein the initial data extraction script records a program template for extracting data; and editing the target field to a corresponding position in the initial data extraction script to obtain the target data extraction script.

Optionally, the apparatus comprises one of: a second extraction module, configured to extract a target field of the target service attribute from the query request before generating a target data extraction script according to the target field corresponding to the target service attribute; and the third extraction module is used for extracting a field of the reference service attribute corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation as the target field.

The embodiment of the present application further provides a storage medium, where the storage medium includes a stored program, and when the program runs, the method for extracting any service data is executed.

Alternatively, in this embodiment, the storage medium may be configured to store program codes for performing the following steps: acquiring a query request, wherein the query request is used for requesting to query service data with a target service attribute; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

Embodiments of the present application further provide an electronic device, which includes a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any one of the above embodiments of the method for extracting business data.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: acquiring a query request, wherein the query request is used for requesting to query service data with target service attributes; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relationship; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

The device for extracting the service data comprises a processor and a memory, wherein the units and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, and the effect of improving the query efficiency of the service data is realized by adjusting the kernel parameters.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium, on which a program is stored, where the program, when executed by a processor, implements the method for extracting service data.

The embodiment of the invention provides a processor, which is used for running a program, wherein the extraction method of the business data is executed when the program runs.

Fig. 6 is a schematic diagram of a hardware environment of a service data extraction method according to an embodiment of the present application, and as shown in fig. 6, an embodiment of the present invention provides an electronic device, where the device includes a processor, a memory, and a program that is stored in the memory and is executable on the processor, and the processor implements the following steps when executing the program: acquiring a query request, wherein the query request is used for requesting to query service data with target service attributes; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relationship; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

Optionally, the obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service having the corresponding relationship includes: matching a target sub-attribute set with a reference sub-attribute set and each reference sub-attribute set in a reference program identifier, wherein the reference sub-attribute set comprises a plurality of reference service sub-attributes, the plurality of reference service sub-attributes are used for dividing service types of services from a plurality of dimensions, and the target service attribute comprises the target sub-attribute set; and acquiring a target program identifier corresponding to a reference sub-attribute set successfully matched with the target sub-attribute set, wherein the target program identifier is used for uniquely identifying the target service.

The device herein may be a server, a PC, a PAD, a mobile phone, etc.

The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: acquiring a query request, wherein the query request is used for requesting to query service data with target service attributes; acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation; screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service; and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute. .

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: matching a target sub-attribute set with a reference sub-attribute set and each reference sub-attribute set in a reference program identifier, wherein the reference sub-attribute set comprises a plurality of reference service sub-attributes, the plurality of reference service sub-attributes are used for dividing service types of services from a plurality of dimensions, and the target service attributes comprise the target sub-attribute set; and acquiring a target program identifier corresponding to a reference sub-attribute set successfully matched with the target sub-attribute set, wherein the target program identifier is used for uniquely identifying the target service.

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: determining whether a target attribute value of each target business sub-attribute in one or more target business sub-attributes included in the target sub-attribute set is consistent with a reference attribute value of a reference business sub-attribute of a corresponding dimension in each reference sub-attribute set; and determining the reference sub-attribute set with the reference attribute value consistent with the target attribute value as the reference sub-attribute set successfully matched with the target sub-attribute set.

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: under the condition that the one or more target service sub-attributes comprise a target department type identifier, acquiring a candidate sub-attribute set with a reference identifier value of the reference department type identifier consistent with a target identifier value of the target department type identifier from the reference sub-attribute set; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding candidate identification bit on a candidate service identification string included in each candidate sub-attribute set, wherein the target service identification string includes at least one of: logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier; determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding reference identification bit on a reference service identification string included in each reference sub-attribute set or not under the condition that the target department type identification is not included in the one or more target service sub-attributes, wherein the target service identification string includes at least one of the following: the method comprises the steps of logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier.

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: generating a target data extraction script according to a target field corresponding to the target service attribute, wherein the target data extraction script is used for running a program for extracting data corresponding to the target field; and running the target data extraction script on the target log file to obtain the target service data.

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: acquiring an initial data extraction script, wherein the initial data extraction script records a program template for extracting data; and editing the target field to a corresponding position in the initial data extraction script to obtain the target data extraction script.

Optionally, the computer program product is further adapted to execute a program for initializing the following method steps: extracting a target field of the target service attribute from the query request; and extracting a field of the reference service attribute corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation as the target field.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for extracting service data is characterized by comprising the following steps:

acquiring a query request, wherein the query request is used for requesting to query service data with target service attributes;

acquiring a target service corresponding to the target service attribute from the reference service attribute and the reference service with the corresponding relation;

screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service;

and extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

2. The method according to claim 1, wherein said obtaining the target service corresponding to the target service attribute from the reference service attribute and the reference service having the corresponding relationship comprises:

3. The method according to claim 2, wherein matching the target sub-attribute set with each of the reference sub-attribute sets having the correspondence relationship with the reference sub-attribute set and the reference program identifier comprises:

4. The method of claim 3, wherein the determining whether the target attribute value of each of the one or more target business sub-attributes included in the target set of sub-attributes is consistent with the reference attribute value of the reference business sub-attribute of the corresponding dimension in each of the reference set of sub-attributes comprises:

determining whether each target identification bit in a target service identification string included in the target service sub-attribute is consistent with a corresponding reference identification bit on a reference service identification string included in each reference sub-attribute set under the condition that the one or more target service sub-attributes do not include a target department type identification, wherein the target service identification string includes at least one of the following: the method comprises the steps of logging in a service identifier, a dynamic account service identifier, a service dynamic account service identifier, a thread service identifier, an interface service identifier and a log-out service identifier.

5. The method of claim 1, wherein extracting the service data from the target log file to obtain target service data comprises:

6. The method of claim 5, wherein generating a target data extraction script according to the target field corresponding to the target service attribute comprises:

7. The method of claim 5, wherein prior to generating a target data extraction script according to the target field corresponding to the target business attribute, the method further comprises one of:

8. An apparatus for extracting service data, comprising:

the system comprises a first acquisition module, a first service module and a second acquisition module, wherein the first acquisition module is used for acquiring a query request, and the query request is used for requesting to query service data with target service attributes;

the second acquisition module is used for acquiring the target service corresponding to the target service attribute from the reference service attribute and the reference service which have the corresponding relation;

the screening module is used for screening a target log file corresponding to the target service from a log file set, wherein the log file set comprises log files generated by executing the service;

and the first extraction module is used for extracting service data from the target log file to obtain target service data, wherein the target service data is service data with the target service attribute.

9. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any of claims 1 to 7.

10. An electronic device, comprising one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.