WO2020233013A1 - 一种数据处理方法、设备及存储介质 - Google Patents

一种数据处理方法、设备及存储介质 Download PDF

Info

Publication number
WO2020233013A1
WO2020233013A1 PCT/CN2019/117925 CN2019117925W WO2020233013A1 WO 2020233013 A1 WO2020233013 A1 WO 2020233013A1 CN 2019117925 W CN2019117925 W CN 2019117925W WO 2020233013 A1 WO2020233013 A1 WO 2020233013A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
channel
data
business
log channel
Prior art date
Application number
PCT/CN2019/117925
Other languages
English (en)
French (fr)
Inventor
饶发强
Original Assignee
平安普惠企业管理有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安普惠企业管理有限公司 filed Critical 平安普惠企业管理有限公司
Publication of WO2020233013A1 publication Critical patent/WO2020233013A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging

Definitions

  • This application relates to the field of computer technology, and in particular to a data processing method, device and storage medium.
  • log collection is mainly based on proactively collecting logs from the application server for reporting.
  • proxy method Each application deployment server must install an additional agent and perform daily operation and maintenance separately. The agent needs to occupy part of the hardware resources of the application server. The peak period of the business may affect the business.
  • a new application is added to the log analysis platform, it is necessary to manually add a log channel between log collection and storage, and then monitor the agent from the message pipeline. The log is migrated to the log storage, the efficiency of log acquisition is low, and the query and management of the log takes a lot of time to retrieve, and the processing process is cumbersome.
  • the embodiments of the present application provide a data processing method, device, and storage medium, which can improve the efficiency of log data processing.
  • One aspect of the embodiments of the present application provides a data processing method, which may include:
  • the log data is stored in the log storage address corresponding to the log channel according to the log channel list.
  • a data processing device which may include:
  • a data acquisition unit for acquiring log data generated according to business data and log tags corresponding to the log data
  • the channel creation unit is configured to obtain the log channel name corresponding to the log tag from the log channel list, create a log channel corresponding to the log channel name, and store the log data in the created log channel,
  • the log channel is used to temporarily store log data generated based on business data;
  • the log storage unit is configured to store the log data in a log storage address corresponding to the log channel according to the log channel list when log data is detected in the log channel.
  • the third aspect of the embodiments of the present application provides a data processing device, including a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and executed as in the embodiments of the present application.
  • the first method is a data processing device, including a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and executed as in the embodiments of the present application.
  • the fourth aspect of the embodiments of the present application provides a computer non-volatile readable storage medium
  • the computer non-volatile readable storage medium stores a computer program
  • the computer program includes program instructions
  • the program instructions are When executed by the processor, the method in the first aspect of the embodiment of the present application is executed.
  • the self-discovery mechanism can automatically identify newly added log data and automatically monitor the corresponding log channel to realize automatic storage and archiving of logs, avoid the problem of manually pulling logs for storage, and improve the efficiency of log processing .
  • FIG. 1 is a schematic flowchart of a data processing method provided by an embodiment of the present application
  • FIG. 2 is a schematic flowchart of another data processing method provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an example of a data processing method provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of another data processing device provided by an embodiment of the present application.
  • Fig. 6 is a schematic structural diagram of another data processing device provided by an embodiment of the present application.
  • the data processing method provided by the embodiments of this application can be applied to scenarios where a large number of log files are collected, archived, and historical log files are regularly organized.
  • the business processing of an enterprise includes multiple processing modules, and each module corresponds to The business generates different log files.
  • the log files corresponding to different modules require different temporary storage channels.
  • the log files of different modules are stored in the corresponding storage addresses.
  • the data processing equipment obtains the log data generated based on the business data and the The log label corresponding to the log data; obtain the log channel name corresponding to the log label from the log channel list, create a log channel corresponding to the log channel name, and store the log data in the created log channel, so
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, the log data is stored in the log storage address corresponding to the log channel according to the log channel list ,
  • the self-discovery mechanism it can automatically identify the new log data and automatically monitor the corresponding log channel to realize the automatic storage and archiving of the log, avoid the problem of manually pulling the log for storage, and improve the efficiency of log processing.
  • the data processing devices involved in the embodiments of the present application may include: tablet computers, smart phones, personal computers (PCs), notebook computers, palmtop computers, mobile Internet devices (MIDs), and other terminal devices that can process log files.
  • PCs personal computers
  • MIDs mobile Internet devices
  • FIG. 1 provides a schematic flowchart of a data processing method according to an embodiment of this application.
  • the method of the embodiment of the present application may include the following steps S101 to S103.
  • S101 Obtain log data generated according to business data and a log tag corresponding to the log data;
  • the data processing device obtains the log data generated according to the business data and the log tag corresponding to the log data.
  • the business data is the corresponding data generated according to the user's business request for a specific business, and processes various
  • the back-end server for business data generates the log data according to the business data.
  • the server may be a back-end terminal of an insurance company that processes financial services.
  • the server generates corresponding log data in the process of processing user business data, and the log data carries There is a log tag corresponding to the log data, and the log tag is generated according to the business module to which the server processing the business data belongs.
  • the log tags of the log data corresponding to business data generation are the same.
  • One log data corresponds to a log tag.
  • Different log data can have the same log tag. For example, if a user applies for an insurance from an insurance company, the back-end server of the insurance company will apply according to the insurance application.
  • Request to generate the log data corresponding to the application request and the log tag corresponding to the log data the data processing device obtains the log data and the log tag corresponding to the log data, the log tag corresponding to the business request.
  • the log data corresponding to other service requests under the service module has the same log label.
  • the data processing device obtains the log channel name corresponding to the log tag from the log channel list, creates a log channel corresponding to the log channel name, and stores the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data.
  • the log channel list stores the association relationship between the log tag, the log channel name, and the log storage address, and the association relationship is the log tag, A corresponding relationship between the log channel name and the log storage address.
  • the data processing device obtains the log channel name corresponding to the log tag from the log channel list, creates the log channel corresponding to the log channel name, and stores the log Data is stored in the created log channel.
  • the log channel corresponds to the business module one-to-one
  • the log data generated by different business modules are stored in different log channels
  • the log channels correspond to the log tags , That is, different log tags correspond to different log channels.
  • the data processing equipment can periodically detect whether to create a new log channel through an automated script. If a new log channel is created, an application process will be automatically started to log the log channel. For data processing, when an invalid log channel is detected, the automated script will clean up the invalid log channel.
  • the data processing device when the data processing device detects that log data exists in the log channel, it stores the log data in the log storage address corresponding to the log channel according to the log channel list. It can be understood that when the data is processed When the device detects the existence of log data in the log channel, the data processing device obtains the log storage address corresponding to the log channel name of the log channel from the log channel list, and stores the log data in the log storage address.
  • the storage address is used to store log data. Specifically, it can be a storage engine or storage space in a log storage device. It should be noted that the data processing device performs further processing on the log data before storing the log data in the log storage address. Processing, for example, deleting duplicate data in log data.
  • the log data generated according to the business data and the log tag corresponding to the log data are obtained; the log channel name corresponding to the log tag is obtained from the log channel list, and the log channel name corresponding is created And store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, according to the The log channel list stores the log data to the log storage address corresponding to the log channel.
  • FIG. 2 provides a schematic flowchart of a data processing method according to an embodiment of this application.
  • the method of the embodiment of the present application may include the following steps S201 to S209.
  • S202 Insert a business keyword and a globally unique business feature code related to the business data corresponding to the log data into the log data;
  • the data processing device inserts a business keyword and a globally unique business feature code related to the business data corresponding to the log data into the log data.
  • the business keyword is related to the business data
  • the service keyword may specifically be the log generation time, the user’s mobile phone number, user name, or credentials, etc.
  • the globally unique service feature code is the global serial number, and the same service corresponds to a service feature code Different services correspond to different service feature codes.
  • the logs generated by the same service have the same service feature codes. For example, a service needs to be processed on several different service modules, and each of the several service modules Corresponding log data will be generated, and several log data corresponding to this business have the same business feature code.
  • S203 Obtain a log tag corresponding to at least one business module that generates log data, obtain a log channel name corresponding to each log tag and a log storage address corresponding to each log channel name; The label, the association relationship between each log channel name and each log storage address are stored in the log channel list;
  • the data processing device obtains log tags corresponding to at least one business module that generates log data, obtains the log channel name corresponding to each log tag and the log storage address corresponding to each log channel name;
  • the association relationship between the log tags, each log channel name, and each log storage address is stored in a log channel list. It can be understood that the log channel list is used to store log tags, log The association relationship between the channel name and the log storage address, and the association relationship is a correspondence between the log label, the log channel name, and the log storage address.
  • S204 Detect whether there is a log channel corresponding to the log label in the log channel set according to the log channel list; if so, obtain the log channel corresponding to the log label in the log channel set, and store the log data in the log label Corresponding log channel;
  • the data processing device detects whether there is a log channel corresponding to the log label in the log channel set according to the log channel list; if it exists, obtains the log channel corresponding to the log label in the log channel set, and converts the log data Stored to the log channel corresponding to the log tag, it can be understood that the data processing device obtains the log channel name corresponding to the log tag from the log channel list, and detects whether there is a log channel to which the log channel name belongs in the log channel set
  • the log channel set may include multiple log channels. If the log channel to which the log channel name belongs exists in the log channel set, the log data is stored in the log channel.
  • the corresponding relationship is obtained according to the log channel list, the corresponding relationship is set in advance, and the user can manage the existing corresponding relationship.
  • S206 Detect whether there is log data in the log channel according to a preset period; when log data is detected in the log channel, obtain the log storage address corresponding to the log channel name to which the log channel belongs from the log channel list; Store the log data to the log storage address.
  • the data processing device detects whether there is log data in the log channel according to a preset period; when log data is detected in the log channel, it obtains the log channel name corresponding to the log channel from the log channel list.
  • Log storage address the log data is stored to the log storage address. It is understandable that the data processing device detects whether there is log data in the log channel according to a preset period, and the preset detection period can be from an automated script
  • the data processing device acquires the log storage address corresponding to the log channel name to which the log channel belongs from the log channel list, and stores the log data in the log storage In the address, the log storage address is used to store log data. Specifically, it may be a storage engine or storage space in a log storage device. It should be noted that before the data processing device stores the log data in the log storage address, it will Perform further processing on the log data, for example, delete duplicate data in the log data.
  • the data processing device when it detects that there is sensitive information in the log data, it uses special symbols to replace the sensitive information in the log data.
  • the sensitive information includes the user's personal information or is Private information, such as the user's username or contact information
  • the special symbol is an identifier used to hide sensitive information, such as an asterisk, specifically, when the data processing device detects that the user's name is in the log information , The user’s name is replaced by an asterisk, or the mobile phone number is detected in the log information, and the data processing device uses the asterisk to replace some digits in the mobile phone number, which improves the security of the information.
  • S208 Obtain a log search request, and obtain an input keyword and an input feature code according to the log search request;
  • the data processing device obtains the log retrieval request, and obtains the input keyword and input feature code according to the log retrieval request.
  • the log retrieval request is an instruction for log retrieval, and the instruction may be a click Instruction or voice instruction
  • the log retrieval request can be obtained by clicking the log retrieval interface on the touch screen, or clicking the corresponding button of the log retrieval.
  • the data processing device obtains the log retrieval request and obtains the input key according to the log retrieval request Character and input feature code, the input keyword is a business keyword input by the user, and the input feature code is a business feature code input by the user.
  • the data processing device matches the input keyword with the business keyword of the log data in the log storage address, and matches the input feature code with the globally unique business feature code of the log data in the log storage address; from the log storage
  • the log data that successfully matches the input keyword and the input feature code is acquired from the address.
  • the data processing device matches the input keyword with the business keyword of the log data in the log storage address,
  • the input feature code is matched with the globally unique business feature code of the log data in the log storage address, and the log data that simultaneously matches the input keyword and the input feature code is obtained from the log storage address, and the matching
  • the business keyword and the input keyword of the successful log data, and the business feature code of the successfully matched log data is the same as the input feature code, and the successfully matched log data is displayed on a display device.
  • the display device may Is it a display screen or a projector.
  • steps S201 and 205 in the embodiment of the present application please refer to the detailed description of steps S101 and S102 in the embodiment shown in FIG. 1, which will not be repeated here.
  • the log data generated according to the business data and the log tag corresponding to the log data are obtained; the log channel name corresponding to the log tag is obtained from the log channel list, and the log channel name corresponding is created And store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, according to the The log channel list stores the log data to the log storage address corresponding to the log channel.
  • the newly added log data can be automatically identified, and the corresponding log channel can be automatically monitored to realize the automatic storage and archiving of logs, which avoids
  • the problem of manually pulling logs for storage improves the efficiency of log processing, and supports the insertion of a global identification code for marking a complete transaction in each log record, which facilitates the complete retrieval of a single business log and supports log desensitization processing , Improve the security of information.
  • the log data server generated based on business data is a back-end terminal for the insurance company to process financial services.
  • the back-end terminal includes multiple business modules, each The business module can process multiple services.
  • the data processing device obtains the log data generated according to the business data and the log label corresponding to the log data.
  • the business data under the same business module generates the same log label of the corresponding log data
  • the data processing device receives Obtain the log channel name corresponding to the log tag from the log channel list, create a log channel corresponding to the log channel name, and store the log data in the created log channel, which is used for temporary storage according to Log data generated by business data, as shown in Figure 4, business module 1 corresponds to log tag 1, log tag 1 corresponds to log channel 1.
  • the data processing device creates log channel 1 and The log data is stored in log channel 1, business module 2 corresponds to log label 2, and log label 2 corresponds to log channel 2.
  • the data processing device For the log data generated by business module 2 according to the business data, the data processing device creates log channel 2 and sends the log The data is stored in log channel 2.
  • the data processing device stores the log data in the log storage address corresponding to the log channel according to the log channel list, as shown in Figure 4 Shows that when log data is detected in log channel 1, log channel 2 or log channel 3, the data processing device obtains the log storage address corresponding to the log channel name to which the log channel belongs from the log channel list, and stores the log The data is stored in the log storage address.
  • the data processing device 1 of the embodiment of the present application may include: a data acquisition unit 11, a channel creation unit 12, and a log storage unit 13.
  • the data obtaining unit 11 is configured to obtain log data generated according to business data and log tags corresponding to the log data;
  • the data acquisition unit 11 acquires log data generated according to business data and log tags corresponding to the log data.
  • the business data is corresponding data generated according to a user's business request for a specific business, and processes each
  • the back-end server for this kind of business data generates the log data according to the business data.
  • the server may be a back-end terminal of an insurance company that processes financial services.
  • the server generates corresponding log data in the process of processing the user’s business data.
  • the log data Carrying a log tag corresponding to the log data, the log tag is generated according to the business module to which the server processing the business data belongs.
  • the log tags of the corresponding log data generated by the business data are the same.
  • One log data corresponds to a log tag.
  • Different log data can have the same log tag.
  • An application request generates log data corresponding to the application request and a log tag corresponding to the log data
  • the data processing device obtains the log data and a log tag corresponding to the log data
  • the log tag corresponds to the business request
  • log data corresponding to other business requests under the business module have the same log label.
  • the channel creation unit 12 is configured to obtain the log channel name corresponding to the log label from the log channel list, create a log channel corresponding to the log channel name, and store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data;
  • the channel creation unit 12 obtains the log channel name corresponding to the log tag from the log channel list, creates a log channel corresponding to the log channel name, and stores the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data.
  • the log channel list stores the association relationship between the log tag, the log channel name, and the log storage address, and the association relationship is the log tag , A corresponding relationship between the log channel name and the log storage address
  • the data processing device obtains the log channel name corresponding to the log tag from the log channel list, creates the log channel corresponding to the log channel name, and then Log data is stored in the created log channel.
  • the log channel corresponds to the business module one-to-one
  • the log data generated by different business modules are stored in different log channels
  • the log channel corresponds to the log Labels, that is, different log labels correspond to different log channels.
  • the data processing equipment can periodically detect whether to create a new log channel through an automated script. If a new log channel is created, an application process will be automatically started to perform the log channel. In the processing of log data, when an invalid log channel is detected, the automated script will clean up the invalid log channel.
  • the log storage unit 13 is configured to store the log data in the log storage address corresponding to the log channel according to the log channel list when log data is detected in the log channel.
  • the log storage unit 13 when the log storage unit 13 detects that log data exists in the log channel, it stores the log data in the log storage address corresponding to the log channel according to the log channel list. It can be understood that when the data When the processing device detects the existence of log data in the log channel, the log storage unit 13 obtains the log storage address corresponding to the log channel name to which the log channel belongs from the log channel list, and stores the log data in the log storage address.
  • the log storage address is used to store log data, which can be specifically a storage engine or storage space in a log storage device. It should be noted that the data processing device performs processing on the log data before storing the log data in the log storage address. Further processing, for example, delete duplicate data in the log data.
  • the log data generated according to the business data and the log tag corresponding to the log data are obtained; the log channel name corresponding to the log tag is obtained from the log channel list, and the log channel name corresponding is created And store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, according to the The log channel list stores the log data to the log storage address corresponding to the log channel.
  • the data processing device 1 of the embodiment of the present application may include: a data acquisition unit 11, a channel creation unit 12, a log storage unit 13, an association relationship storage and storage unit 14, a channel detection unit 15, and an information replacement unit 16. Insertion unit 17, search request unit 18, matching unit 19.
  • the data obtaining unit 11 is configured to obtain log data generated according to business data and log tags corresponding to the log data;
  • the inserting unit 17 is configured to insert business keywords and globally unique business feature codes related to the business data corresponding to the log data into the log data;
  • the inserting unit 17 inserts a business keyword and a globally unique business feature code related to the business data corresponding to the log data into the log data.
  • the business keyword is related to the business data.
  • the service keyword may specifically be the log generation time, the user’s mobile phone number, user name, or credentials, etc.
  • the globally unique service feature code is the global serial number, and the same service corresponds to a service feature code Different services correspond to different service feature codes.
  • the logs generated by the same service have the same service feature codes. For example, a service needs to be processed on several different service modules, and each of the several service modules Corresponding log data will be generated, and several log data corresponding to this business have the same business feature code.
  • the association relationship storage unit 14 is configured to obtain log tags corresponding to at least one business module that generates log data, obtain the log channel name corresponding to each log tag and the log storage address corresponding to each log channel name ; Store the association relationship between each log tag, each log channel name and each log storage address in the log channel list;
  • the association relationship storage storage unit 14 obtains log tags corresponding to at least one business module that generates log data, obtains the log channel name corresponding to each log tag and the log storage address corresponding to each log channel name. ; Store the association relationship between each log tag, each log channel name, and each log storage address in a log channel list, it is understandable that the log channel list is used to store logs
  • the association relationship between the label, the log channel name and the log storage address, and the association relationship is a correspondence between the log label, the log channel name and the log storage address.
  • the channel detection unit 15 is configured to detect whether there is a log channel corresponding to the log label in the log channel set according to the log channel list; if it exists, obtain the log channel corresponding to the log label in the log channel set, and convert the log Data is stored in the log channel corresponding to the log tag;
  • the channel detection unit 15 detects whether there is a log channel corresponding to the log label in the log channel set according to the log channel list; if it exists, obtains the log channel corresponding to the log label in the log channel set, and converts the log The data is stored in the log channel corresponding to the log tag. It is understandable that the channel detection unit 15 obtains the log channel name corresponding to the log tag from the log channel list, and detects whether there is a log channel name in the log channel set.
  • the log channel, the log channel set may include multiple log channels, and if the log channel to which the log channel name belongs exists in the log channel set, the log data is stored in the log channel, and the log channel and the log The corresponding relationship of data is obtained according to the log channel list, the corresponding relationship is set in advance, and the user can manage the existing corresponding relationship.
  • the channel creation unit 12 is configured to, if the log channel corresponding to the log label does not exist in the log channel set, obtain the log channel name corresponding to the log label from the log channel list, and create a log channel corresponding to the log channel name, And store the log data in the created log channel;
  • the log storage unit 13 is configured to detect whether log data exists in the log channel according to a preset period; when log data is detected in the log channel, obtain the log channel name corresponding to the log channel from the log channel list Log storage address; store the log data to the log storage address.
  • the log storage unit 13 detects whether there is log data in the log channel according to a preset period; when log data is detected in the log channel, obtains the log channel name corresponding to the log channel to which the log channel belongs from the log channel list.
  • the data processing device When the data processing device detects the existence of log data in the log channel, the data processing device obtains the log storage address corresponding to the log channel name to which the log channel belongs from the log channel list, and stores the log data in In the log storage address, the log storage address is used to store log data, which may be a storage engine or storage space in a log storage device. It should be noted that the data processing device stores the log data before the log storage address , The log data will be further processed, for example, to delete duplicate data in the log data.
  • the information replacement unit 16 is configured to replace the sensitive information in the log data with special symbols when it is detected that there is sensitive information in the log data;
  • the information replacement unit 16 uses special symbols to replace the sensitive information in the log data.
  • the sensitive information includes the user's personal information or It is private information, such as the user's username or contact information.
  • the special symbol is an identifier used to hide sensitive information, such as an asterisk.
  • the data processing device detects that the user's name exists in the log information When the user's name is replaced by an asterisk, or the mobile phone number is detected in the log information, the data processing device uses the asterisk to replace some digits in the mobile phone number, which improves the security of the information.
  • the search request unit 18 is configured to obtain a log search request, and obtain an input keyword and an input feature code according to the log search request;
  • the search request unit 18 obtains a log search request, and obtains an input keyword and an input feature code according to the log search request.
  • the log search request is an instruction for log search, and the instruction may be Click command or voice command
  • the log retrieval request can be by clicking the log retrieval interface on the touch screen, or by clicking the button corresponding to the log retrieval
  • the data processing device obtains the log retrieval request, and obtains input according to the log retrieval request
  • a keyword and an input feature code the input keyword is a business keyword input by a user
  • the input feature code is a business feature code input by the user.
  • the matching unit 19 is configured to match the input keyword with the service keyword, and match the input feature code with the service feature code; obtain the input keyword and the service feature code from the log storage address.
  • the above input feature codes all match the log data successfully.
  • the matching unit 19 matches the input keyword with the business keyword of the log data in the log storage address, and matches the input feature code with the globally unique business feature code of the log data in the log storage address; The log data that successfully matches the input keyword and the input feature code is acquired from the address. It is understandable that the matching unit 19 matches the input keyword with the business keyword of the log data in the log storage address.
  • the input feature code is matched with the globally unique business feature code of the log data in the log storage address, and the log data that simultaneously matches the input keyword and the input feature code is obtained from the log storage address.
  • the business keyword and input keyword of the successfully matched log data, and the business feature code of the successfully matched log data is the same as the input feature code, and the successfully matched log data is displayed on a display device, the display device It can be a display screen or a projector.
  • the log data generated according to the business data and the log tag corresponding to the log data are obtained; the log channel name corresponding to the log tag is obtained from the log channel list, and the log channel name corresponding is created And store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, according to the The log channel list stores the log data to the log storage address corresponding to the log channel.
  • the newly added log data can be automatically identified, and the corresponding log channel can be automatically monitored to realize the automatic storage and archiving of logs, which avoids
  • the problem of manually pulling logs for storage improves the efficiency of log processing, and supports the insertion of a global identification code for marking a complete transaction in each log record, which facilitates the complete retrieval of a single business log and supports log desensitization processing , Improve the security of information.
  • the embodiments of the present application also provide a computer non-volatile readable storage medium, the computer non-volatile readable storage medium may store a plurality of instructions, and the instructions are suitable for being loaded by a processor and executed as described above.
  • the method steps of the embodiment shown in FIG. 1 to FIG. 3 for the specific execution process, please refer to the specific description of the embodiment shown in FIG. 1 to FIG. 3, which will not be repeated here.
  • the data processing 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.
  • the communication bus 1002 is used to implement connection and communication between these components.
  • the user interface 1003 may include a display screen (Display), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory.
  • the memory 1005 may also be at least one storage device located far away from the foregoing processor 1001.
  • the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a data processing application program.
  • the user interface 1003 is mainly used to provide an input interface for the user to obtain data input by the user; and the processor 1001 can be used to call the data processing application stored in the memory 1005 and execute The following operations:
  • the log data is stored in the log storage address corresponding to the log channel according to the log channel list.
  • the processor 1001 obtains the log channel name corresponding to the log tag from the log channel list, creates a log channel corresponding to the log channel name, and stores the log data Before the created log channel, perform the following operations:
  • the processor 1001 when the processor 1001 executes the storage of the log data to the log storage address corresponding to the log channel when the log data is detected in the log channel, it specifically executes the following operations :
  • the processor 1001 obtains the log channel name corresponding to the log tag from the log channel list, creates a log channel corresponding to the log channel name, and stores the log data Before the created log channel, perform the following operations:
  • the processor 1001 specifically performs the following operations:
  • the processor 1001 specifically executes the following operations after executing the acquiring log data generated according to the business data and the log tag corresponding to the log data:
  • the processor 1001 specifically performs the following operations:
  • the log data generated according to the business data and the log tag corresponding to the log data are obtained; the log channel name corresponding to the log tag is obtained from the log channel list, and the log channel name corresponding is created And store the log data in the created log channel.
  • the log channel is used to temporarily store log data generated according to business data; when log data is detected in the log channel, according to the The log channel list stores the log data to the log storage address corresponding to the log channel.
  • the program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments.
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本申请实施例公开一种数据处理方法、设备及存储介质，该方法适用于数据存储的应用，其中方法包括如下步骤：获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。采用本申请，可以提高日志数据处理的效率，提高信息的安全性。

Description

一种数据处理方法、设备及存储介质
本申请要求于2019年5月20日提交中国专利局，申请号为201910420243.3、发明名称为“一种数据处理方法、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种数据处理方法、设备及存储介质。
背景技术
随着计算机技术的发展，计算机技术运用的越来越广泛，目前市面上开源的日志分析平台应用最广的属ELK系统了，其主要由三部分组成：日志收集，日志存储与分析，日志检索，其中，日志收集主要是以代理方式主动从应用服务器采集日志进行上报，但基于代理方式存在诸多弊端，每台应用部署的服务器上必须另外安装一个代理并需要单独进行日常运维维护，每个代理需占用一部分应用服务器的硬件资源业务高峰期可能会对业务造成影响，当有新的应用加入到日志分析平台，日志收集与存储之间需要手动添加日志通道，然后监听代理从消息管道中把日志迁移到日志存储器中，日志获取的效率较低，并且日志的查询和管理需要花费大量时间去检索，处理过程比较繁琐。
发明内容
本申请实施例提供一种数据处理方法、设备及存储介质，可以提高日志数据处理的效率。
本申请实施例一方面提供了一种数据处理方法，可包括：
获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
本申请实施例一方面提供了一种数据处理设备，可包括：
数据获取单元，用于获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
通道创建单元，用于从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
日志存储单元，用于当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
本申请实施例第三方面提供了一种数据处理设备，包括处理器和存储器；其中，所述存储器存储有计算机程序，所述计算机程序适于由所述处理器加载并执行如本申请实施例第一方面的方法。
本申请实施例第四方面提供了一种计算机非易失性可读存储介质，所述计算机非易失性可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时，执行如本申请实施例第一方面的方法。
在本申请实施例通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避免了手动拉取日志进行存储的问题，提高了日志处理的效率。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种数据处理方法的流程示意图；
图2是本申请实施例提供的另一种数据处理方法的流程示意图；
图3是本申请实施例提供的一种数据处理方法的举例示意图；
图4是本申请实施例提供的一种数据处理设备的结构示意图；
图5是本申请实施例提供的另一种数据处理设备的结构示意图；
图6是本申请实施例提供的又一种数据处理设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请实施例提供的数据处理方法可以应用于对大量日志文件进行收集、归档处理以及定期对历史日志文件进行整理的场景，例如，在企业的业务处理中包括多个处理模块，各个模块下对应的业务生成不同的日志文件，不同模块对应的日志文件需要不同的临时存储通道，不同模块的日志文件存储在对应的存储地址，具体的，数据处理设备获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归 档，避免了手动拉取日志进行存储的问题，提高了日志处理的效率。
本申请实施例涉及的数据处理设备可以包括：平板电脑、智能手机、个人电脑(PC)、笔记本电脑、掌上电脑以及移动互联网设备(MID)等可以进行日志文件处理的终端设备。
请参见图1，为本申请实施例提供了一种数据处理方法的流程示意图。如图1所示，本申请实施例的所述方法可以包括以下步骤S101-步骤S103。
S101，获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
具体的，数据处理设备获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签，可以理解的是，所述业务数据是根据用户对具体业务的业务请求生成的对应数据，处理各种业务数据的后台服务器根据所述业务数据生成所述日志数据，例如服务器可以是保险公司处理金融服务的后台终端，服务器在处理用户的业务数据过程中会生成对应的日志数据，所述日志数据携带有日志数据对应的日志标签，所述日志标签根据处理所述业务数据的服务器所属的业务模块生成，所述业务模块可以有多个，所述业务模块可以包括多种业务，相同业务模块下的业务数据生成对应的日志数据的日志标签相同，一个日志数据对应一个日志标签，不同的日志数据可以有相同的日志标签，例如，用户申请保险公司的一种保险，保险公司的后台服务器根据保险申请请求生成所述申请请求对应的日志数据以及所述日志数据对应的日志标签，所述数据处理设备获取所述日志数据以及所述日志数据对应的日志标签，所述日志标签对应所述业务请求所属的业务模块，所述业务模块下的其它业务请求对应的日志数据有相同的日志标签。
S102，从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
具体的，数据处理设备从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据，可以理解的是，所述日志通道列表中存储日志标签、日志通道名称和日志存储地址之间的关联关系，所述关联关系是日志标签、日志通道名称和日志存储地址之间的一种对应关系，数据处理设备从日志通道列表中获取所述日志标签对应的日志通道名称，创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所述创建的日志通道，需要说明的是，所述日志通道与业务模块一一对应，不同的业务模块生成的日志数据存储至不同的日志通道，所述日志通道对应所述日志标签，即不同的日志标签对应不同的日志通道，数据处理设备可以通过自动化脚本周期性检测是否创建新的日志通道，如果创建新的日志通道则会自动启动一个应用进程来对所述日志通道进行日志数据的处理，当检 测到有已经失效的日志通道时，自动化脚本会清理所述失效的日志通道。
S103，当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
具体的，数据处理设备当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，可以理解的是，当数据处理设备在日志通道中检测存在日志数据时，数据处理设备从所述日志通道列表获取所述日志通道所属日志通道名称对应的日志存储地址，将所述日志数据存储至日志存储地址中，所述日志存储地址用于存储日志数据，具体可以为存储引擎或者是日志存储设备中的存储空间，需要说明的是，数据处理设备将所述日志数据存储到日志存储地址之前，会对日志数据进行进一步的处理，例如，删除日志数据中的重复数据。
在本申请实施例中，通过获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避免了手动拉取日志进行存储的问题，提高了日志处理的效率。
请参见图2，为本申请实施例提供了一种数据处理方法的流程示意图。如图2所示，本申请实施例的所述方法可以包括以下步骤S201-步骤S209。
S201，获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
S202，在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码；
具体的，数据处理设备在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码，可以理解的是，所述业务关键字是与业务数据相关的信息，所述业务关键字具体可以是日志的生成时间、用户的手机号、用户名、或者证件等信息，所述全局唯一的业务特征码为全局流水号，同一个业务对应一个业务特征码，不同的业务对应不同的业务特征码，同一个业务产生的日志有相同的业务特征码，例如，一个业务需要在几个不同的业务模块上处理，所述几个业务模块中每一个业务模块均会生成对应的日志数据，这个业务对应的几个日志数据拥有相同的业务特征码。
S203，获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中；
具体的，数据处理设备获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中，可以理解的是，所述日志通道列表用于存储日志标签、日志通道名称和日志存储地址之间的关联关系，所述关联关系是日志标签、日志通道名称和日志存储地址之间的一种对应关系。
S204，根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道；
具体的，数据处理设备根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道，可以理解的是，数据处理设备从所述日志通道列表中获取日志标签对应的日志通道名称，并检测日志通道集合中是否存在所述日志通道名称所属的日志通道，所述日志通道集合可以包括多个日志通道，若日志通道集合中存在所述日志通道名称所属的日志通道，则将所述日志数据存储至所述日志通道，所述日志通道与日志数据的对应关系是根据所述日志通道列表获取，所述对应关系是预先进行设置的，用户可以对已存在的对应关系进行管理。
S205，若不存在，则从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道；
S206，根据预设周期检测所述日志通道中是否存在日志数据；当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；将所述日志数据存储至所述日志存储地址。
具体的，数据处理设备根据预设周期检测所述日志通道中是否存在日志数据；当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；将所述日志数据存储至所述日志存储地址，可以理解的是，数据处理设备根据预设周期检测所述日志通道中是否存在日志数据，所述预设检测周期可以从自动化脚本中获取，当数据处理设备在日志通道中检测存在日志数据时，数据处理设备从所述日志通道列表获取所述日志通道所属日志通道名称对应的日志存储地址，将所述日志数据存储至日志存储地址中，所述日志存储地址用于存储日志数据，具体可以为存储引擎或者是日志存储设备中的存储空间，需要说明的是，数据处理设备将所述日志数据存储到日志存储地址之前，会对日志数据进行进一步的处理，例如，删除日志数据中的重复数据。
S207，当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息；
具体的，数据处理设备当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息，可以理解的是，所述敏感信息包括用户的个人信息或者是隐私信息，例如用户的用户名、或者是联系方式信息，所述特殊符号是用于隐藏敏感信息的标识，例如可以为星号，具体的，数据处理设备检测到日志信息中存在用户的名称时，则采用星号将用户的名字替换，或者是检测到日志信息中存在手机号码，数据处理设备采用星号将手机号码中的部分数字替换，提高了信息的安全性。
S208，获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码；
具体的，数据处理设备获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码，可以理解的是，所述日志检索请求是针对日志检索的指令，所述指令可以是点击指令或者是语音指令，所述日志检索请求可以通过点击触屏上的日志检索的接口，或者是点击日志检索对应的按钮，数据处理设备获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码，所述输入关键字是用户输入的业务关键字，所述输入特征码是用户输入的业务特征码。
S209，将所述输入关键字与所述业务关键字进行匹配，将所述输入特征码与所述业务特征码进行匹配；从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据。
具体的，数据处理设备将所述输入关键字与日志存储地址中日志数据的业务关键字进行匹配，将输入特征码与日志存储地址中日志数据的全局唯一的业务特征码进行匹配；从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据，可以理解的是，数据处理设备将所述输入关键字与日志存储地址中日志数据的业务关键字进行匹配，将输入特征码与日志存储地址中日志数据的全局唯一的业务特征码进行匹配，从日志存储地址中获取同时与所述输入关键字和所述输入特征码均匹配成功的日志数据，所述匹配成功的日志数据的业务关键字与输入关键字，且所述匹配成功的日志数据的业务特征码与输入特征码相同，将所述匹配成功的日志数据展示在显示设备中，所述显示设备可以是显示屏或者是投影仪。
其中，本申请实施例的步骤S201和205参见图1所示实施例的步骤S101和S102的具体描述，在此不进行赘述。
在本申请实施例中，通过获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避 免了手动拉取日志进行存储的问题，提高了日志处理的效率，并支持在每条日志记录中插入用于标记一笔完整交易的全局识别码，方便单笔业务日志完整检索，支持日志脱敏处理，提高了信息的安全性。
下面将对本申请实施例提供的一种数据处理方法进行具体说明。
具体的实施场景为用户在保险公司处理金融服务，如图4所示，根据业务数据生成的日志数据的服务器是保险公司处理金融服务的后台终端，所述后台终端包括多个业务模块，每个业务模块可以处理多种业务，数据处理设备获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签，相同业务模块下的业务数据生成对应的日志数据的日志标签相同，数据处理设备从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据，如图4所示，业务模块1对应日志标签1，日志标签1对应日志通道1，针对业务模块1根据业务数据生成的日志数据，数据处理设备创建日志通道1并将所述日志数据存储至日志通道1，业务模块2对应日志标签2，日志标签2对应日志通道2，针对业务模块2根据业务数据生成的日志数据，数据处理设备创建日志通道2并将所述日志数据存储至日志通道2，当检测到所述日志通道中存在日志数据时，数据处理设备根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，如图4所示，当检测到日志通道1或者日志通道2或者日志通道3中存在日志数据时，数据处理设备从所述日志通道列表获取所述日志通道所属日志通道名称对应的日志存储地址，将所述日志数据存储至日志存储地址中。
请参见图4，为本申请实施例提供了一种数据处理设备的结构示意图。如图4所示，本申请实施例的所述数据处理设备1可以包括：数据获取单元11、通道创建单元12、日志存储单元13。
数据获取单元11，用于获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
具体的，数据获取单元11获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签，可以理解的是，所述业务数据是根据用户对具体业务的业务请求生成的对应数据，处理各种业务数据的后台服务器根据所述业务数据生成所述日志数据，例如服务器可以是保险公司处理金融服务的后台终端，服务器在处理用户的业务数据过程中会生成对应的日志数据，所述日志数据携带有日志数据对应的日志标签，所述日志标签根据处理所述业务数据的服务器所属的业务模块生成，所述业务模块可以有多个，所述业务模块可以包括多种业务，相同业务模块下的业务数据生成对应的日志数据的日志标签相同，一个日志数据对应一个日志标签，不同的日志数据可以有相同的日志标签，例如，用户申请保险公司的一种保险，保险公司的后台服务器根据保险申请请求生成 所述申请请求对应的日志数据以及所述日志数据对应的日志标签，所述数据处理设备获取所述日志数据以及所述日志数据对应的日志标签，所述日志标签对应所述业务请求所属的业务模块，所述业务模块下的其它业务请求对应的日志数据有相同的日志标签。
道创建单元12，用于从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
具体的，道创建单元12从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据，可以理解的是，所述日志通道列表中存储日志标签、日志通道名称和日志存储地址之间的关联关系，所述关联关系是日志标签、日志通道名称和日志存储地址之间的一种对应关系，数据处理设备从日志通道列表中获取所述日志标签对应的日志通道名称，创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所述创建的日志通道，需要说明的是，所述日志通道与业务模块一一对应，不同的业务模块生成的日志数据存储至不同的日志通道，所述日志通道对应所述日志标签，即不同的日志标签对应不同的日志通道，数据处理设备可以通过自动化脚本周期性检测是否创建新的日志通道，如果创建新的日志通道则会自动启动一个应用进程来对所述日志通道进行日志数据的处理，当检测到有已经失效的日志通道时，自动化脚本会清理所述失效的日志通道。
日志存储单元13，用于当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
具体的，日志存储单元13当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，可以理解的是，当数据处理设备在日志通道中检测存在日志数据时，日志存储单元13从所述日志通道列表获取所述日志通道所属日志通道名称对应的日志存储地址，将所述日志数据存储至日志存储地址中，所述日志存储地址用于存储日志数据，具体可以为存储引擎或者是日志存储设备中的存储空间，需要说明的是，数据处理设备将所述日志数据存储到日志存储地址之前，会对日志数据进行进一步的处理，例如，删除日志数据中的重复数据。
在本申请实施例中，通过获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避 免了手动拉取日志进行存储的问题，提高了日志处理的效率。
请参见图5，为本申请实施例提供了一种数据处理设备的结构示意图。如图5所示，本申请实施例的所述数据处理设备1可以包括：数据获取单元11、通道创建单元12、日志存储单元13、关联关系存储存储单元14、通道检测单元15、信息替换单元16、插入单元17、检索请求单元18、匹配单元19。
数据获取单元11，用于获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
插入单元17，用于在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码；
具体的，插入单元17在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码，可以理解的是，所述业务关键字是与业务数据相关的信息，所述业务关键字具体可以是日志的生成时间、用户的手机号、用户名、或者证件等信息，所述全局唯一的业务特征码为全局流水号，同一个业务对应一个业务特征码，不同的业务对应不同的业务特征码，同一个业务产生的日志有相同的业务特征码，例如，一个业务需要在几个不同的业务模块上处理，所述几个业务模块中每一个业务模块均会生成对应的日志数据，这个业务对应的几个日志数据拥有相同的业务特征码。
关联关系存储存储单元14，用于获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中；
具体的，关联关系存储存储单元14获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中，可以理解的是，所述日志通道列表用于存储日志标签、日志通道名称和日志存储地址之间的关联关系，所述关联关系是日志标签、日志通道名称和日志存储地址之间的一种对应关系。
通道检测单元15，用于根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道；
具体的，通道检测单元15根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道，可以理解的是，通道检测单元15从所述日志通道列表中获取日志标签对应的日志通道名称，并检测日志通道集合中是否存在所述日志通道名称所属的日志通道，所述日志通道集合可以包括多个日志通道，若日志通道集合中存在所述日 志通道名称所属的日志通道，则将所述日志数据存储至所述日志通道，所述日志通道与日志数据的对应关系是根据所述日志通道列表获取，所述对应关系是预先进行设置的，用户可以对已存在的对应关系进行管理。
通道创建单元12，用于若日志通道集合中不存在日志标签对应的日志通道，则从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道；
日志存储单元13，用于根据预设周期检测所述日志通道中是否存在日志数据；当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；将所述日志数据存储至所述日志存储地址。
具体的，日志存储单元13根据预设周期检测所述日志通道中是否存在日志数据；当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；将所述日志数据存储至所述日志存储地址，可以理解的是，日志存储单元13根据预设周期检测所述日志通道中是否存在日志数据，所述预设检测周期可以从自动化脚本中获取，当数据处理设备在日志通道中检测存在日志数据时，数据处理设备从所述日志通道列表获取所述日志通道所属日志通道名称对应的日志存储地址，将所述日志数据存储至日志存储地址中，所述日志存储地址用于存储日志数据，具体可以为存储引擎或者是日志存储设备中的存储空间，需要说明的是，数据处理设备将所述日志数据存储到日志存储地址之前，会对日志数据进行进一步的处理，例如，删除日志数据中的重复数据。
信息替换单元16，用于当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息；
具体的，信息替换单元16当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息，可以理解的是，所述敏感信息包括用户的个人信息或者是隐私信息，例如用户的用户名、或者是联系方式信息，所述特殊符号是用于隐藏敏感信息的标识，例如可以为星号，具体的，数据处理设备检测到日志信息中存在用户的名称时，则采用星号将用户的名字替换，或者是检测到日志信息中存在手机号码，数据处理设备采用星号将手机号码中的部分数字替换，提高了信息的安全性。
检索请求单元18，用于获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码；
具体的，检索请求单元18获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码，可以理解的是，所述日志检索请求是针对日志检索的指令，所述指令可以是点击指令或者是语音指令，所述日志检索请求可以通过点击触屏上的日志检索的接口，或者是点击日志检索对应的按钮，数据处理设备获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码，所述输入关键字是用户输入的业务关键字，所述输入特征码是用户输 入的业务特征码。
匹配单元19，用于将所述输入关键字与所述业务关键字进行匹配，将所述输入特征码与所述业务特征码进行匹配；从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据。
具体的，匹配单元19将所述输入关键字与日志存储地址中日志数据的业务关键字进行匹配，将输入特征码与日志存储地址中日志数据的全局唯一的业务特征码进行匹配；从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据，可以理解的是，匹配单元19将所述输入关键字与日志存储地址中日志数据的业务关键字进行匹配，将输入特征码与日志存储地址中日志数据的全局唯一的业务特征码进行匹配，从日志存储地址中获取同时与与所述输入关键字和所述输入特征码均匹配成功的日志数据，所述匹配成功的日志数据的业务关键字与输入关键字，且所述匹配成功的日志数据的业务特征码与输入特征码相同，将所述匹配成功的日志数据展示在显示设备中，所述显示设备可以是显示屏或者是投影仪。
其中，本申请实施例的数据获取单元11和通道创建单元12参见图4所示实施例的数据获取单元11和通道创建单元12的具体描述，在此不进行赘述。
在本申请实施例中，通过获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避免了手动拉取日志进行存储的问题，提高了日志处理的效率，并支持在每条日志记录中插入用于标记一笔完整交易的全局识别码，方便单笔业务日志完整检索，支持日志脱敏处理，提高了信息的安全性。
本申请实施例还提供了一种计算机非易失性可读存储介质，所述计算机非易失性可读存储介质可以存储有多条指令，所述指令适于由处理器加载并执行如上述图1-图3所示实施例的方法步骤，具体执行过程可以参见图1-图3所示实施例的具体说明，在此不进行赘述。
请参见图6，为本申请实施例提供了一种数据处理的结构示意图。如图6所示，所述数据处理1000可以包括：至少一个处理器1001，例如CPU，至少一个网络接口1004，用户接口1003，存储器1005，至少一个通信总线1002。其中，通信总线1002用于实现这些组件之间的连接通信。其中，用户接口1003可以包括显示屏(Display)，可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如 WI-FI接口)。存储器1005可以是高速RAM存储器，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。存储器1005可选的还可以是至少一个位于远离前述处理器1001的存储装置。如图6所示，作为一种计算机可读存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及数据处理应用程序。
在图6所示的设备1000中，用户接口1003主要用于为用户提供输入的接口，获取用户输入的数据；而处理器1001可以用于调用存储器1005中存储的数据处理应用程序，并具体执行以下操作：
获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
在一个实施例中，所述处理器1001在执行所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道之前，具体执行以下操作：
获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；
将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中。
在一个实施例中，所述处理器1001在执行所述当检测到所述日志通道中存在日志数据时，将所述日志数据存储至所述日志通道对应的日志存储地址时，具体执行以下操作：
根据预设周期检测所述日志通道中是否存在日志数据；
当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；
将所述日志数据存储至所述日志存储地址。
在一个实施例中，所述处理器1001在执行所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道之前，具体执行以下操作：
根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；
若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道；
若不存在，则执行所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至 所创建的日志通道的步骤。
在一个实施例中，所述处理器1001具体还执行以下操作：
当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息。
在一个实施例中，所述处理器1001在执行所述获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签之后，具体执行以下操作：
在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码。
在一个实施例中，所述处理器1001具体还执行以下操作：
获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码；
将所述输入关键字与所述业务关键字进行匹配，将所述输入特征码与所述业务特征码进行匹配；
从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据。
在本申请实施例中，通过获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址，通过采用自发现机制能够自动识别新增的日志数据，并自动监听相应日志通道，实现日志的自动存储归档，避免了手动拉取日志进行存储的问题，提高了日志处理的效率。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)或随机存储记忆体(Random Access Memory，RAM)等。
以上所揭露的仅为本申请较佳实施例而已，当然不能以此来限定本申请之权利范围，因此依本申请权利要求所作的等同变化，仍属本申请所涵盖的范围。

Claims (20)

  1. 一种数据处理方法，其特征在于，包括：
    获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
    从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
    当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
  2. 根据权利要求1所述的方法，其特征在于，所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道之前，还包括：
    获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；
    将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中。
  3. 根据权利要求1所述的方法，其特征在于，所述获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签，包括：
    获取业务请求指令，所述业务请求指令携带业务模块信息；
    根据所述业务请求指令生成所述业务模块信息对应的业务数据，根据所述业务数据生成日志数据；
    获取所述日志数据对应的业务模块信息所属的日志标签。
  4. 根据权利要求1或2所述的方法，其特征在于，所述当检测到所述日志通道中存在日志数据时，将所述日志数据存储至所述日志通道对应的日志存储地址，包括：
    根据预设周期检测所述日志通道中是否存在日志数据；
    当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；
    将所述日志数据存储至所述日志存储地址。
  5. 根据权利要求1或2所述的方法，其特征在于，所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道之前，还包括：
    根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；
    若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道；
    若不存在，则执行所述从日志通道列表中获取所述日志标签对应的日志通 道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道的步骤。
  6. 根据权利要求1或2所述的方法，其特征在于，还包括：
    启动目标脚本对日志通道环境进行检测；所述日志通道环境用于监控日志通道的状态；
    当检测到所述日志通道环境中创建新的日志通道时，则通过所述目标脚本启动所述新的日志通道对应的目标应用进程，通过所述目标进程对所述新的日志通道中的日志数据进行处理；
    当检测到所述日志通道环境中的日志通道处于失效状态时，通过所述目标脚本会清理所述处于失效状态的日志通道。
  7. 根据权利要求1或2所述的方法，其特征在于，还包括：
    当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息。
  8. 根据权利要求1或2所述的方法，其特征在于，所述获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签之后，还包括：
    在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码。
  9. 根据权利要求8所述的方法，其特征在于，还包括：
    获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码；
    将所述输入关键字与所述业务关键字进行匹配，将所述输入特征码与所述业务特征码进行匹配；
    从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据。
  10. 一种数据处理设备，其特征在于，包括：
    数据获取单元，用于获取根据业务数据生成的日志数据以及所述日志数据对应的日志标签；
    通道创建单元，用于从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道，所述日志通道用于临时存储根据业务数据生成的日志数据；
    日志存储单元，用于当检测到所述日志通道中存在日志数据时，根据所述日志通道列表将所述日志数据存储至所述日志通道对应的日志存储地址。
  11. 根据权利要求10所述的设备，其特征在于，还包括：
    关联关系存储存储单元，用于获取生成日志数据的至少一个业务模块对应的日志标签，获取每个所述日志标签分别对应的日志通道名称以及每个所述日志通道名称分别对应的日志存储地址；
    将每个所述日志标签、每个所述日志通道名称和每个所述日志存储地址之间的关联关系存储至日志通道列表中。
  12. 根据权利要求10所述的设备，其特征在于，所述数据获取单元具体用于：
    获取业务请求指令，所述业务请求指令携带业务模块信息；
    根据所述业务请求指令生成所述业务模块信息对应的业务数据，根据所述业务数据生成日志数据；
    获取所述日志数据对应的业务模块信息所属的日志标签。
  13. 根据权利要求10或11所述的设备，其特征在于，所述日志存储单元具体用于：
    根据预设周期检测所述日志通道中是否存在日志数据；
    当检测到所述日志通道中存在日志数据时，从日志通道列表中获取所述日志通道所属日志通道名称对应的日志存储地址；
    将所述日志数据存储至所述日志存储地址。
  14. 根据权利要求10或11所述的设备，其特征在于，还包括：
    通道检测单元，用于根据所述日志通道列表在日志通道集合中检测是否存在日志标签对应的日志通道；
    若存在，在日志通道集合中获取所述日志标签对应的日志通道，将所述日志数据存储至日志标签对应的日志通道；
    若不存在，则执行所述从日志通道列表中获取所述日志标签对应的日志通道名称，并创建所述日志通道名称对应的日志通道，并将所述日志数据存储至所创建的日志通道的步骤。
  15. 根据权利要求10或11所述的设备，其特征在于，所述通道检测单元，具体还用于：
    启动目标脚本对日志通道环境进行检测；所述日志通道环境用于监控日志通道的状态；
    当检测到所述日志通道环境中创建新的日志通道时，则通过所述目标脚本启动所述新的日志通道对应的目标应用进程，通过所述目标进程对所述新的日志通道中的日志数据进行处理；
    当检测到所述日志通道环境中的日志通道处于失效状态时，通过所述目标脚本会清理所述处于失效状态的日志通道。
  16. 根据权利要求10或11所述的设备，其特征在于，还包括：
    信息替换单元，用于当检测到所述日志数据中存在敏感信息时，采用特殊符号替换所述日志数据中的所述敏感信息。
  17. 根据权利要求10或11所述的设备，其特征在于，还包括：
    插入单元，用于在所述日志数据中插入与所述日志数据对应的业务数据相关的业务关键字和全局唯一的业务特征码。
  18. 根据权利要求17所述的设备，其特征在于，还包括：
    检索请求单元，用于获取日志检索请求，并根据所述日志检索请求获取输入关键字和输入特征码；
    匹配单元，用于将所述输入关键字与所述业务关键字进行匹配，将所述输入特征码与所述业务特征码进行匹配；从日志存储地址中获取与所述输入关键字和所述输入特征码均匹配成功的日志数据。
  19. 一种计算机非易失性可读存储介质，其特征在于，所述计算机非易失性可读存储介质存储有多条指令，所述指令适于由处理器加载并执行如权利要求1-9任意一项的方法步骤。
  20. 一种数据处理设备，其特征在于，其特征在于，包括：处理器和存储器；其中，所述存储器存储有计算机程序，所述计算机程序适于由所述处理器加载并执行如权利要求1-9任意一项的方法步骤。
PCT/CN2019/117925 2019-05-20 2019-11-13 一种数据处理方法、设备及存储介质 WO2020233013A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910420243.3A CN110245059B (zh) 2019-05-20 2019-05-20 一种数据处理方法、设备及存储介质
CN201910420243.3 2019-05-20

Publications (1)

Publication Number Publication Date
WO2020233013A1 true WO2020233013A1 (zh) 2020-11-26

Family

ID=67884455

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/117925 WO2020233013A1 (zh) 2019-05-20 2019-11-13 一种数据处理方法、设备及存储介质

Country Status (2)

Country Link
CN (1) CN110245059B (zh)
WO (1) WO2020233013A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116186785A (zh) * 2023-04-28 2023-05-30 中国工商银行股份有限公司 日志脱敏方法、装置、设备、介质和程序产品

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110245059B (zh) * 2019-05-20 2022-11-08 平安普惠企业管理有限公司 一种数据处理方法、设备及存储介质
CN111104456A (zh) * 2019-10-12 2020-05-05 深圳壹账通智能科技有限公司 数据持久化存储方法、装置、计算机设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020138762A1 (en) * 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
CN106899643A (zh) * 2015-12-21 2017-06-27 阿里巴巴集团控股有限公司 一种用户日志存储方法及设备
CN107239382A (zh) * 2017-06-23 2017-10-10 深圳市冬泉谷信息技术有限公司 一种容器应用的日志处理方法及系统
CN110245059A (zh) * 2019-05-20 2019-09-17 平安普惠企业管理有限公司 一种数据处理方法、设备及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012173752A (ja) * 2011-02-17 2012-09-10 Nec Corp クラスタシステム、データ記録方法、及びプログラム
CN104601489B (zh) * 2015-02-02 2018-03-27 迈普通信技术股份有限公司 一种日志块传输方法及设备
CN106407094B (zh) * 2015-07-29 2020-06-23 深圳市腾讯计算机系统有限公司 日志系统及日志配置方法
CN106559241B (zh) * 2015-09-29 2019-11-08 阿里巴巴集团控股有限公司 应用日志的收集、发送方法、装置、系统及日志服务器
CN105490854B (zh) * 2015-12-11 2019-03-12 传线网络科技（上海）有限公司 实时日志收集方法、系统和应用服务器集群

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020138762A1 (en) * 2000-12-01 2002-09-26 Horne Donald R. Management of log archival and reporting for data network security systems
CN106899643A (zh) * 2015-12-21 2017-06-27 阿里巴巴集团控股有限公司 一种用户日志存储方法及设备
CN107239382A (zh) * 2017-06-23 2017-10-10 深圳市冬泉谷信息技术有限公司 一种容器应用的日志处理方法及系统
CN110245059A (zh) * 2019-05-20 2019-09-17 平安普惠企业管理有限公司 一种数据处理方法、设备及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116186785A (zh) * 2023-04-28 2023-05-30 中国工商银行股份有限公司 日志脱敏方法、装置、设备、介质和程序产品

Also Published As

Publication number Publication date
CN110245059A (zh) 2019-09-17
CN110245059B (zh) 2022-11-08

Similar Documents

Publication Publication Date Title
US10356032B2 (en) System and method for detecting confidential information emails
WO2020233013A1 (zh) 一种数据处理方法、设备及存储介质
US20160085794A1 (en) Data consistency and rollback for cloud analytics
US20180285596A1 (en) System and method for managing sensitive data
CN109522751B (zh) 访问权限控制方法、装置、电子设备及计算机可读介质
US20160162507A1 (en) Automated data duplicate identification
CN108134696A (zh) 一种微服务配置更新方法及装置
US20210165785A1 (en) Remote processing of memory and files residing on endpoint computing devices from a centralized device
US20150286663A1 (en) Remote processing of memory and files residing on endpoint computing devices from a centralized device
CN110895472A (zh) 一种识别业务变更的方法和装置
CN111552895B (zh) 小程序应用中页面路由的解析方法、系统、设备及介质
CN104903865A (zh) 恢复虚拟机映像的在前版本
US9395966B1 (en) Systems and methods for associating installed software components with software products
CN109472540B (zh) 业务处理方法和装置
US9021389B1 (en) Systems and methods for end-user initiated data-loss-prevention content analysis
US10129328B2 (en) Centralized management of webservice resources in an enterprise
WO2021012554A1 (zh) 区块链中数据字段的更新方法、装置、介质、电子设备
US11868412B1 (en) Data enrichment systems and methods for abbreviated domain name classification
CN115544050A (zh) 操作日志记录方法、装置、设备及存储介质
CN115544558A (zh) 敏感信息检测方法、装置、计算机设备及存储介质
EP2819028A2 (en) Content management system
US9632918B1 (en) Creating expected test results using previous test results
CN109547290B (zh) 云平台垃圾数据检测处理方法、装置、设备及存储介质
US20240070319A1 (en) Dynamically updating classifier priority of a classifier model in digital data discovery
CN109525630B (zh) 发送数据分析委托请求的方法、装置、介质及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19929829

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19929829

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 18/03/2022)

122 Ep: pct application non-entry in european phase

Ref document number: 19929829

Country of ref document: EP

Kind code of ref document: A1