WO2022236973A1

WO2022236973A1 - Data backfilling method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022236973A1
Application number: PCT/CN2021/110210
Authority: WO
Inventors: 宋同珍; 万月亮; 火一莽
Original assignee: 北京锐安科技有限公司
Priority date: 2021-05-08
Filing date: 2021-08-03
Publication date: 2022-11-17
Also published as: CN113204531A

Abstract

A data backfilling method and apparatus, an electronic device, and a storage medium. The method comprises: extracting current log data in a data stream to be processed; parsing field information in the current log data according to a log data format; on the basis of the field information in the current log data, detecting whether association information and key attention information exist in the current log data; in response to the existence of the association information in the current log data and no detection result of the key attention information in the current log data, searching for the key attention information in an association relationship database on the basis of the association information; in response to the search of a search result of the key attention information in the association relationship database, backfilling the key attention information into the current log data; and storing the backfilled current log data into the log database.

Description

Data backfill method, device, electronic device and storage medium

This disclosure claims the priority of the Chinese patent application with application number 202110500895.5 submitted to the China Patent Office on May 08, 2021, and the entire content of the above application is incorporated in this disclosure by reference.

technical field

The embodiments of the present application relate to the technical field of data processing, for example, to a data backfilling method, device, electronic equipment, and storage medium.

Background technique

Many Internet applications carry different information in the data generated under different operations. Some operations carry some content of specific concern, but other operations do not carry such content; The information carried by the data generated by different mobile phones under the same operation is different. For example, for mobile phones with the IOS operating system and mobile phones with the Android operating system, the information carried by the data generated under the same operation may be different.

The inconsistency of the information carried in the data generated under the same operation has caused difficulties for the data processing methods in related technologies. For example, some information of specific concern only exists in the data of a certain operation or in the data of a certain terminal. , but other operational data or other terminal data are also very concerned about this information, it can be seen that the processing methods in the related art cannot maximize the value of these key information.

Contents of the invention

This application provides a data backfill method, device, electronic equipment, and storage medium, which can backfill information that lacks key concerns, so that the key information can be reflected in different scenarios, and maximize the value of data. .

In the first aspect, the embodiment of the present application provides a data backfilling method, the method comprising:

Extracting the current log data from the data stream to be processed; parsing field information in the current log data according to the log data format;

Based on the field information in the current log data, detect whether there is associated information and focus information in the current log data;

In response to a detection result that the associated information exists in the current log data and the focused information does not exist in the current log data, searching for the focused information in an association relationship database based on the associated information;

Responding to the finding of the key information in the association relationship database, backfill the key information into the current log data; store the backfilled current log data into the log database.

In the second aspect, the embodiment of the present application also provides a data backfilling device, which includes: an analysis module, a detection module, a search module, and a backfill module; wherein,

The parsing module is configured to extract the current log data in the data stream to be processed; parse the field information in the current log data according to the log data format;

The detection module is configured to detect whether there is associated information and focus information in the current log data based on the field information in the current log data;

The search module is configured to, in response to the detection result that the associated information exists in the current log data and the key attention information does not exist in the current log data, search the association relationship database based on the associated information. State key information;

The backfilling module is configured to backfill the key attention information into the current log data in response to finding the search result of the key attention information in the association relationship database; store the backfilled current log data to the log database.

In a third aspect, the embodiment of the present application provides an electronic device, including:

one or more processors;

memory, arranged to store one or more programs,

The one or more processors are configured to execute the one or more programs to implement the data backfill method described in any embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data backfilling method described in any embodiment of the present application is implemented.

Description of drawings

FIG. 1 is a schematic flow diagram of a data backfill method provided in an embodiment of the present application;

FIG. 2 is a schematic flow diagram of another data backfill method provided in the embodiment of the present application;

FIG. 3 is a schematic flowchart of another data backfill method provided in the embodiment of the present application;

FIG. 4 is a schematic structural diagram of a data backfill device provided in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The application will be described below in conjunction with the accompanying drawings and embodiments.

Embodiment one

Figure 1 is a schematic flow chart of a data backfill method provided by an embodiment of the present application. The method can be performed by a data backfill device or an electronic device, and the device or electronic device can be implemented by software and/or hardware. The device or Electronic devices can be integrated in any smart device with network communication function. As shown in Figure 1, the data backfill method may include the following steps:

S101. Extract current log data from the data stream to be processed; parse field information from the current log data according to the log data format.

In this step, the electronic device can extract the current log data from the data stream to be processed; and parse out multiple field information in the current log data according to the format of the log data. For example, the electronic device can extract the current session from the data stream to be processed; extract the log information from the current session; perform structured processing on the log information to obtain structured processed log information; Output to a structured file; use the structured file as current log data.

S102. Based on the field information in the current log data, detect whether there is associated information and focus information in the current log data.

In this step, the electronic device may detect whether there is associated information and focus information in the current log data based on multiple field information in the current log data. For example, each log data may include multiple fields; these fields at least include fields corresponding to associated information and fields corresponding to focused information. Therefore, based on the field corresponding to the associated information, the electronic device can detect whether there is associated information under the field corresponding to the associated information; at the same time, it can also detect whether there is focused information under the field corresponding to the focused information based on the field corresponding to the focused information. For example, if the value of the current log data in the field corresponding to the associated information is empty, it means that the current log does not have associated information; if the value of the current log data in the field corresponding to the associated information is not empty, it means that the current log Associated information exists.

The focused attention information refers to the attention information preset or defaulted by the user, which may be log information corresponding to at least one field and/or at least one format.

S103. In response to the detection result that there is associated information in the current log data and there is no focused information in the current log data, search for the focused information in the association relationship database based on the associated information.

In this step, if there is associated information in the current log data and there is no focused information in the current log data, the electronic device may search the associated relationship database for the focused information based on the associated information. For example, the associated information in this embodiment of the present application may be an identifier (Identity, ID) of each log data; it may also be other identifiers that uniquely identify each log data. Therefore, the electronic device can use the associated information of each log data to search the associated relationship database for the key information corresponding to the log data.

S104. In response to finding the search result of the key attention information in the relational database, backfill the key attention information into the current log data; store the backfilled current log data into the log database.

In this step, if the focused information is found in the relational database, the electronic device may backfill the focused information into the current log data; and store the backfilled current log data into the log database. For example, the electronic device may backfill the found key attention information to a corresponding position based on the association information of the current log data.

The data backfilling method proposed in the embodiment of the present application extracts the current log data from the data stream to be processed; parses the field information in the current log data according to the log data format; detects the current log data based on the field information in the current log data. Whether there is associated information and focused information; in response to the detection result that there is associated information in the current log data and there is no focused information in the current log data, look up the focused information in the associated relationship database based on the associated information; Find the search results of key attention information in the database, and backfill the key attention information into the current log data; store the backfilled current log data in the log database. That is to say, in the embodiment of the present application, the associated information and key attention information in different log data can be stored in the association relationship database, so that the corresponding key attention information can be found in the association relationship database based on the association information , the found focus information can be backfilled into the log data that lacks the focus information.

The processing method in the related art cannot maximize the value of the focused information. Compared with related technologies, the data backfill method proposed in the embodiment of the present application can backfill information that lacks key concerns, so that the key information can be reflected in different scenarios, and maximize the value of data; In addition, the embodiments of the present application are simple, convenient, popular and applicable to a wider range.

Embodiment two

FIG. 2 is a schematic flowchart of another data backfilling method provided by an embodiment of the present application. The foregoing embodiment may be extended, and Embodiment 2 may also be combined with the foregoing other embodiments. As shown in Figure 2, the data backfill method may include the following steps:

S201. Extract current log data from the data stream to be processed; parse field information from the current log data according to the log data format.

S202. Based on the field information in the current log data, detect whether there is associated information and focus information in the current log data.

S203. In response to the detection result that there is associated information in the current log data and no focused information exists in the current log data, search for the focused information in the association relationship database based on the associated information.

S204. In response to the search result that the key information is not found in the relational database, store the current log data in the log database.

S205. Extract at least one piece of log data that has associated information and does not contain focused information from the log database according to a predetermined period.

In this step, the electronic device may extract from the log database at least one piece of log data that has associated information and does not contain focused information according to a predetermined period. For example, for log data that only has relevant information but no key information, the electronic device can store the log data that only has relevant information but no key information in the log database in advance; Log data that only has associated information but no focused information.

In addition, the electronic device can extract only a predetermined amount of log data from the log database, or extract all log data from the log database.

S206. Based on the association information in the at least one log data, look up the key information in the at least one log data in the association relationship database.

The relationship database is in the process of dynamic data increase and decrease, and the query results obtained by query operations at different times may be different. For example, the key information corresponding to the associated information may be obtained, or the key information corresponding to the associated information may not be obtained.

S207. Responding to the finding of the key information in at least one log data in the relational database, backfill the key information in at least one log data into at least one log data respectively; at least one log after backfilling The data is stored in the log database.

The data backfilling method proposed in the embodiment of the present application extracts the current log data from the data stream to be processed; parses the field information in the current log data according to the log data format; detects the current log data based on the field information in the current log data. Whether there is associated information and focused information; in response to the detection result that there is associated information in the current log data and there is no focused information in the current log data, look up the focused information in the associated relationship database based on the associated information; Find the search results of key attention information in the database, and backfill the key attention information into the current log data; store the backfilled current log data in the log database. That is to say, in the embodiment of the present application, the associated information and key attention information in different log data can be stored in the association relationship database, so that the corresponding key attention information can be found in the association relationship database based on the association information , the found focus information can be backfilled into the log data that lacks the found focus information.

Embodiment Three

FIG. 3 is a schematic flowchart of another data backfilling method provided by an embodiment of the present application. The foregoing embodiments may be extended, and Embodiment 3 may be combined with other foregoing embodiments. As shown in Figure 3, the data backfill method may include the following steps:

S301. Extract the current session from the data stream to be processed; extract log information from the current session.

In this step, the electronic device can extract the current session from the data stream to be processed; extract the log information from the current session. For example, the electronic device may extract multiple sessions from the data to be processed according to the data format corresponding to each session as the current session; extract log information from the current session according to the data format corresponding to the log information.

S302. Perform structured processing on the log information to obtain the structured processed log information; output the structured processed log information to the structured file; use the structured file as the current log data; The field information is parsed from the data.

In this step, the electronic device can perform structured processing on the log information to obtain the structured log information; output the structured processed log information to the structured file; use the structured file as the current log data; according to The log data format parses out multiple field information in the current log data. For example, the data in structured files is structured data. Structured data is also called row data. It is data logically expressed and realized by a two-dimensional table structure. It strictly follows the data format and length specifications, mainly through relational databases. for storage and management.

S303. Based on the field information in the current log data, detect whether there is associated information and focus information in the current log data.

S304. In response to the detection result that the relevant information exists in the current log data and the focused information does not exist in the current log data, search for the focused information in the relational database based on the correlated information.

S305. Responding to the search result of finding the key attention information in the relational database, backfill the key attention information into the current log data; store the backfilled current log data into the log database.

The data backfilling method proposed in the embodiment of the present application extracts the current log data from the data stream to be processed; parses out multiple field information in the current log data according to the log data format; based on the multiple field information in the current log data, detects Whether there is associated information and key attention information in the current log data; if there is associated information and no key attention information in the current log data, then based on the associated information, search for the key attention information in the associated relational database; if found in the associated relational database Focus on the information, backfill the focused information into the current log data; store the backfilled current log data in the log database. That is to say, in the embodiment of the present application, the associated information and key attention information in different log data can be stored in the association relationship database, so that the corresponding key attention information can be found in the association relationship database based on the association information , the found focus information can be backfilled into the log data that lacks the focus information.

Embodiment Four

FIG. 4 is a schematic structural diagram of a data backfilling device provided by an embodiment of the present application. As shown in FIG. 4, the data backfilling device 400 includes: an analysis module 401, a detection module 402, a search module 403 and a backfill module 404; wherein,

The parsing module 401 is configured to extract the current log data in the data stream to be processed; parse the field information in the current log data according to the log data format;

The detection module 402 is configured to detect whether there is associated information and focus information in the current log data based on the field information in the current log data;

The search module 403 is configured to, in response to the detection result that the associated information exists in the current log data and the key attention information does not exist in the current log data, search in an association relationship database based on the associated information Said key focus information;

The backfilling module 404 is configured to, in response to finding the search result of the key attention information in the association relation database, backfill the key attention information into the current log data; the backfilled current log data stored in the log database.

In an embodiment, the backfill module 404 is further configured to store the associated information and the focused information in response to the detection result that the associated information and the focused information exist in the current log data into the relational database; storing the current log data into the log database.

In one embodiment, the backfilling module 404 is further configured to store the current log data in the log database in response to the search result that the key attention information is not found in the relational database.

In one embodiment, the device further includes: an extraction module;

The extracting module is configured to extract at least one log data in the log database that has the associated information and does not have the focused information according to a predetermined period;

The search module 403 is also configured to search the at least one log data for key attention information in the association relationship database based on the associated information in the at least one log data;

The backfilling module 404 is further configured to, in response to finding the key information in the at least one log data in the association relationship database, backfill the key information in the at least one log data into the Among the at least one log data; storing the backfilled at least one log data in the log database.

In one embodiment, the parsing module 401 is configured to extract the current log data from the data stream to be processed in the following manner: extract the current session from the data stream to be processed; extract the log data from the current session Outputting log information; carrying out structured processing on the log information to obtain structured processed log information; outputting the structured processed log information into a structured file; using the structured file as the current log data.

The above-mentioned data backfilling device can execute the method provided by any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in this embodiment, refer to the data backfill method provided in any embodiment of this application.

Embodiment five

FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. FIG. 5 shows a block diagram of an exemplary electronic device suitable for implementing embodiments of the present application. The electronic device 12 shown in FIG. 5 is just one type of example.

As shown in FIG. 5, electronic device 12 takes the form of a general-purpose computing device. Components of electronic device 12 may include: one or more processors or processing units 16 , system memory 28 , bus 18 connecting various system components including system memory 28 and processing unit 16 .

The bus 18 represents one or more types of bus structures, and the bus 18 includes a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in the various bus structures. These architectures include, for example, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) ) Local bus and Peripheral Component Interconnect (PCI) bus.

Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12 and include both volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (Random Access Memory, RAM) 30 and/or cache memory 32 . Electronic device 12 may include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media, not shown in FIG. 5, commonly referred to as a hard disk drive. Although not shown in FIG. 5, a magnetic disk drive for reading and writing a removable nonvolatile magnetic disk such as a floppy disk, and an optical disk drive for reading and writing a removable nonvolatile optical disk may be provided. Wherein, the removable non-volatile optical disc is, for example, a compact disc read-only memory (CD-ROM), a digital video disc (Digital Video Disc Read-Only Memory, DVD-ROM) or other optical media.

In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.

Program/utility tool 40 may be stored in system memory 28 as a set (eg, at least one) of program modules 46, such program modules 46 including an operating system, one or more application programs, other program modules, and program data, which Each or combination of the examples may include the implementation of a network environment. Program modules 46 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the electronic device 12 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interface 22 . Moreover, the electronic device 12 can also communicate with one or more networks through the network adapter 20, one or more networks such as local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN) and/or public network, public network For example the Internet. As shown in FIG. 5 , network adapter 20 communicates with other modules of electronic device 12 via bus 18 . Other hardware and/or software modules can be used in conjunction with electronic device 12, and other hardware and software modules include: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, Tape drives and data backup storage systems, etc.

The processing unit 16 executes a variety of functional applications and data processing by running the programs stored in the system memory 28 , such as implementing the data backfill method provided by the embodiment of the present application.

Embodiment six

Embodiment 6 of the present application provides a computer storage medium.

The computer-readable storage medium in the embodiments of the present application may use any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. Examples of computer readable storage media include: electrical connections with one or more conductors, portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (Erasable Programmable Read-Only Memory, EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal that travels in baseband or as part of a carrier wave with computer readable program code embodied on the computer readable signal medium. Such propagated data signals may take many forms, including electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code contained on the computer readable medium can be transmitted by any appropriate medium, including wireless, electric wire, optical cable, radio frequency (Radio Frequency, RF), etc., or any appropriate combination of the above.

Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages. A programming language such as C language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, part on the user's computer and part on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the remote computer may be connected to an external computer, such as the remote computer using Internet services provider to connect to external computers via the Internet.

Claims

A data backfill method, the method comprising:

Extracting the current log data from the data stream to be processed; parsing field information in the current log data according to the log data format;

Based on the field information in the current log data, detect whether there is associated information and focus information in the current log data;

In response to a detection result that the associated information exists in the current log data and the focused information does not exist in the current log data, searching for the focused information in an association relationship database based on the associated information;

Responding to the finding of the key information in the association relationship database, backfill the key information into the current log data; store the backfilled current log data into the log database.
The method according to claim 1, said method further comprising:

In response to the detection result that the associated information and the focused attention information exist in the current log data, the associated information and the focused attention information are stored in an association relationship database; the current log data is stored in the in the log database.
The method according to claim 1, said method further comprising:

In response to a search result that the key information is not found in the association relationship database, the current log data is stored in the log database.
The method of claim 3, further comprising:

Extracting at least one log data in the log database that has the associated information and does not have the focused information according to a predetermined period;

Searching for the key information in the at least one log data in the association relationship database based on the association information in the at least one log data;

Responding to the search result of finding the focus information in the at least one log data in the association relationship database, backfill the focus information in the at least one log data into the at least one log data respectively; The backfilled at least one log data is stored in the log database.
The method according to claim 1, wherein said extracting current log data from the data stream to be processed comprises:

Extracting the current session from the data stream to be processed; extracting log information from the current session;

Carrying out structured processing on the log information to obtain structured processed log information; outputting the structured processed log information into a structured file; using the structured file as the current log data.
A data backfilling device, the device comprising: an analysis module, a detection module, a search module and a backfill module; wherein,

The parsing module is configured to extract the current log data in the data stream to be processed; parse the field information in the current log data according to the log data format;

The detection module is configured to detect whether there is associated information and focus information in the current log data based on the field information in the current log data;

The search module is configured to, in response to the detection result that the associated information exists in the current log data and the key attention information does not exist in the current log data, search the association relationship database based on the associated information. State key information;

The backfilling module is configured to backfill the key attention information into the current log data in response to finding the search result of the key attention information in the association relationship database; store the backfilled current log data to the log database.
The device according to claim 6, wherein the backfill module is further configured to combine the associated information with the The key attention information is stored in the relational database; the current log data is stored in the log database.
The device according to claim 6, wherein the backfilling module is further configured to store the current log data in the in the log database.
The device according to claim 8, further comprising: an extraction module;

The extracting module is configured to extract at least one log data in the log database that has the associated information and does not have the focused information according to a predetermined period;

The search module is also configured to search the at least one log data for key attention information in the association relationship database based on the associated information in the at least one log data;

The backfill module is further configured to, in response to finding the key information in the at least one log data in the association relation database, backfill the key information in the at least one log data to the the at least one log data; and store the backfilled at least one log data in the log database.
The device according to claim 6, wherein the parsing module is configured to extract the current log data from the data stream to be processed in the following manner: extract the current session from the data stream to be processed; Extract log information in the current session; carry out structured processing on the log information to obtain structured processed log information; output the structured processed log information into a structured file; as the current log data.
An electronic device comprising:

one or more processors;

memory, arranged to store one or more programs,

The one or more processors are configured to execute the one or more programs to implement the data backfilling method according to any one of claims 1 to 5.
A storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data backfilling method according to any one of claims 1 to 5 is realized.