WO2023000785A1

WO2023000785A1 - Data processing method, device and system, and server and medium

Info

Publication number: WO2023000785A1
Application number: PCT/CN2022/092645
Authority: WO
Inventors: 李启坤; 刘圣杰
Original assignee: 京东科技控股股份有限公司
Priority date: 2021-07-19
Filing date: 2022-05-13
Publication date: 2023-01-26
Also published as: CN113468196A; CN113468196B

Abstract

Disclosed in embodiments of the present disclosure are a data processing method and apparatus, and a server and a medium. A specific embodiment of the method comprises: obtaining an analysis result set of a target log file based on parallel processing, wherein the target log file is used for recording changes of a first database; sorting at least one piece of data according to time information of data in the first database corresponding to analysis results in the analysis result set; and generating, on the basis of latest data indicated by the time information, a logical view associated with the data in the first database.

Description

Method, device, system, server and medium for processing data

Cross References to Related Applications

This patent application claims the priority of the Chinese patent application with the application number 202110813220.6 and the title of the invention "method, device, system, server and medium for processing data" filed on July 19, 2021. The full text of the application Incorporated by reference into this disclosure.

technical field

Embodiments of the present disclosure relate to the field of computer technologies, and in particular to methods, devices, systems, servers and media for processing data.

Background technique

With the rapid development of the Internet, the scale of data that needs to be processed is also increasing. For the technical requirement of synchronizing the relational database to the big data (Big Data) environment, the existing technology provides a method of analyzing the Binlog (binary log) log file of the relational database and directly writing the analysis result into the big data. method of the file system.

In order to ensure the processing sequence and timeliness of downstream data during the Binlog parsing process of the relational database, and in order to ensure data consistency as much as possible, the prior art often adopts a serial method when processing the Binlog process, and after Binlog parsing, all The use of asynchronous message queues for subsequent data processing file generation will result in the constraints of serial processing in massive Binlog parsing scenarios, causing delays in data synchronization.

Contents of the invention

Embodiments of the present disclosure propose methods, apparatuses, systems, servers, media and computer program products for processing data.

In a first aspect, an embodiment of the present disclosure provides a method for processing data, the method comprising: acquiring a parsing result set of a target log file based on parallel processing, wherein the target log file is used to record changes to the first database ; According to the time information of the data in the first database corresponding to the analysis results in the analysis result set, sort at least one piece of data; based on the latest data indicated by the time information, generate logic associated with the data in the first database view.

In some embodiments, the acquisition of the parsing result set of the target log file based on parallel processing includes: receiving the parsing result of the target log file concurrently written by at least two consumers in the target distributed message queuing system; The resulting analysis results are generated to generate a set of analysis results.

In some embodiments, the above parsing results are respectively written into corresponding files in the preset big data file system according to the extracted message subject.

In some embodiments, the above-mentioned sorting of at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set includes: generating files and preset files in the preset big data file system Association relationship between metadata tables; data windowing processing is performed according to the data time association field in the preset metadata table, wherein the data time association field matches the time information of the data in the first database; The data after window processing are sorted by time.

In some embodiments, the method further includes: based on the data in the logical view and the time information corresponding to the data, creating a temporary data table of the target time period, wherein the target time period matches the time information corresponding to the data, and the temporary data table The data in includes the results of the last data processing and the results of this incremental processing.

In a second aspect, an embodiment of the present disclosure provides an apparatus for processing data, the apparatus including: an acquisition unit configured to acquire a parsing result set of a target log file based on parallel processing, wherein the target log file is used for Record the change of the first database; the sorting unit is configured to sort at least one piece of data according to the time information of the data in the first database corresponding to the parsing results in the parsing result set; the generating unit is configured to sort at least one piece of data based on the time information Indicates the most recent data, generating a logical view associated with the data in the first database.

In some embodiments, the acquisition unit is further configured to: receive the parsing results of the target log files concurrently written by at least two consumers in the target distributed message queue system; generate the parsing results based on the received parsing results gather.

In some embodiments, the above sorting unit may be further configured to: generate a preset association relationship between files in the big data file system and preset metadata tables; associate fields according to data time in preset metadata tables performing data windowing processing, wherein the data time correlation field matches the time information of the data in the first database; performing time sorting on the data after the windowing processing.

In some embodiments, the device further includes: a creation unit configured to create a temporary data table of the target time period based on the data in the logical view and the time information corresponding to the data, wherein the target time period and the time information corresponding to the data Matching, the data in the temporary data table includes the result of the last data processing and the result of this incremental processing.

In a third aspect, an embodiment of the present disclosure provides a system for processing data, the system includes: an analysis end configured to write the analysis result of the acquired target log file into the target file based on parallel processing The system, wherein the target log file is used to record changes in the first database; the data processing end is configured to sort at least one piece of data according to the time information of the data in the first database corresponding to the analysis result; based on the time information indicated The most recent data in time, generating a logical view associated with the data in the first database.

In some embodiments, the analysis end includes: an acquisition unit configured to acquire a log file used to record changes in the first database as a target log file; an analysis unit configured to use at least two threads to process the target log file parsing, generating a parsing result; a writing unit configured to write the generated parsing result into a target file system.

In some embodiments, the parsing unit is further configured to: use at least two threads to filter statements that hit preset filter words in the target log file to generate a filtered target log file; use at least two threads to filter the target log file The target log file is parsed and the parsing result is generated.

In some embodiments, the parsing end is further configured to: extract the message subject from the parsed result of the acquired target log file; write the parsing result concurrently to the preset big data The corresponding file in the file system.

In a fourth aspect, an embodiment of the present disclosure provides a server, which includes: one or more processors; a storage device, on which one or more programs are stored; when one or more programs are processed by one or more executed by a processor, so that one or more processors implement the method described in any implementation manner of the first aspect.

In a fifth aspect, an embodiment of the present disclosure provides a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

In a sixth aspect, the embodiments of the present disclosure provide a computer program product including a computer program. When the computer program is executed by a processor, the method described in any implementation manner in the first aspect can be implemented.

Description of drawings

Other characteristics, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present disclosure can be applied;

Figure 2 is a flowchart of one embodiment of a method for processing data according to the present disclosure;

FIG. 3 is a schematic diagram of an application scenario of a method for processing data according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of yet another embodiment of a method for processing data according to the present disclosure;

Fig. 5 is a schematic structural diagram of an embodiment of an apparatus for processing data according to the present disclosure;

Fig. 6 is a sequence diagram of interaction between various devices in an embodiment of the system for processing data according to the present disclosure.

FIG. 7 is a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure.

detailed description

The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that, in the case of no conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and embodiments.

FIG. 1 shows an exemplary architecture 100 to which the method for processing data or the apparatus for processing data of the present disclosure can be applied.

As shown in FIG. 1 , the system architecture 100 may include

servers

101 , 102 , 103 and

networks

104 , 105 . The

networks

104 and 105 are used to provide communication link media between the server 101 and the server 102 and between the server 102 and the server 103 respectively. The

networks

104, 105 may include various connection types such as wires, wireless communication links, or fiber optic cables, among others.

The server 102 can interact with the server 101 and the server 103 through the

networks

104, 105, respectively, to receive or send messages and the like.

The

servers

101, 102, 103 may be servers that provide various services. As an example, server 101 may be a database server. Server 102 may be a server for performing log parsing. The server 103 may be a big data server, which may provide file system services and computing services in a big data environment.

It should be noted that the server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

It should be noted that the method for processing data provided by the embodiments of the present disclosure is generally executed by the server 103 , and correspondingly, the device for processing data is generally disposed in the server 103 .

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

The method, device, system, server, and medium for processing data provided by the embodiments of the present disclosure open the parallel processing of data in the log parsing stage, and perform data sorting according to the time information of the data in the subsequent processing link, and then based on the time The most recent data generates a logical view, which not only improves the processing efficiency of data, but also ensures the consistency of data by "trading space for time".

Continuing to refer to FIG. 2 , a flow 200 of one embodiment of the method for processing data according to the present disclosure is shown. The method for processing data includes the following steps:

Step 201, acquiring a parsing result set of a target log file based on parallel processing.

In this embodiment, the execution body of the method for processing data (the server 103 shown in FIG. 1 ) can obtain the parsing result set of the target log file based on parallel processing through a wired connection or a wireless connection. Wherein, the above-mentioned target log file may be used to record changes of the first database. The change of the above-mentioned first database may include at least one of structure change (such as CREATE, ALTER TABLE, etc.) and data modification (such as INSERT, UPDATE, DELETE, etc.) of the database table. As an example, the above target log file may be a Binlog file.

In this embodiment, the aforementioned execution subject (server 103 as shown in FIG. 1 ) can obtain the parsing result set of the target log file based on parallel processing that is stored locally in advance, and can also obtain the electronic device (such as The parsing result set of the target log file based on parallel processing sent by the server 102) shown in FIG. 1 .

In some optional implementations of this embodiment, the above execution subject may also obtain a parsing result set of the target log file based on parallel processing through the following steps:

The first step is to receive parsing results of target log files written concurrently by at least two consumers in the target distributed message queuing system.

In these implementation manners, as an example, the above-mentioned target distributed message queue system may be Kafka. Specifically, the execution subject may enable concurrent consumption at the consumer end in the target distributed message queue system, so as to allow at least two consumer ends to concurrently write the parsing results of the target log file thereinto.

Based on the above optional implementation methods, this solution can ignore the order of consumption and the limitation of repeated consumption, so as to maximize the concurrent processing speed and greatly reduce the backlog of parsed messages.

Optionally, the above parsing results may be respectively written into corresponding files in the preset big data file system according to the extracted message subject.

In these implementations, different files can be set according to different message topics in the above preset big data file system. Therefore, the execution subject for writing the above analysis results can write the analysis results into corresponding files in the preset big data file system according to the extracted message subject. As an example, the above-mentioned big data file system may be HDFS (Hadoop Distributed File System, Hadoop Distributed File System).

Based on the above optional implementation methods, this solution can directly obtain the analysis results written into the corresponding files, thereby providing a data basis for subsequent fast writing and loading of files.

The second step is to generate a set of analysis results based on the received analysis results.

In these implementations, based on the analysis results received in the first step above, the execution subject may generate a set of analysis results in various ways. As an example, the execution subject may directly form a plurality of received parsing results into a parsing result set.

Step 202, sort at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set.

In this embodiment, according to the time information of the data in the first database corresponding to the analysis results in the analysis result set based on the parallel processing of the target log file obtained in step 201, the above-mentioned execution subject can use various methods for at least one The data is sorted.

In this embodiment, the execution subject may directly follow the chronological order indicated by the time information of the data in the first database corresponding to the analysis results in the analysis result set of the target log file based on parallel processing acquired in the above step 201 Sort the data in the above-mentioned first database.

In some optional implementations of this embodiment, according to the time information of the data in the first database corresponding to the analysis results in the analysis result set, the execution subject may sort at least one piece of data through the following steps:

The first step is to generate the association relationship between the files in the preset big data file system and the preset metadata table.

In these implementations, the execution subject may generate the association relationship between the files in the preset big data file system and the preset metadata table in various ways. As an example, the above execution subject may create an association relationship between a file and a preset metadata table through metadata management of hive (a Hadoop-based data warehouse tool).

The second step is to perform data windowing processing according to the data time correlation field in the preset metadata table.

In these implementations, according to the data time association field in the preset metadata table, the execution subject may perform data windowing processing in various ways. Wherein, the above-mentioned data time association field usually matches the time information of the data in the above-mentioned first database. As an example, the execution subject may perform data windowing processing according to the primary key of the preset metadata table (for example, the id of the data). Wherein, the id of the above data may be the time associated field of the above data (eg modified_date).

The third step is to perform time sorting on the data after windowing processing.

In these implementation manners, the execution subject may perform time sorting on the data after the second step of windowing processing in various ways. As an example, the above execution subject can use the function row_number()over(partition by id order by modified_date desc) to sort the data by time, so as to obtain the data results arranged according to the time order of the data.

Based on the above optional implementation, this solution can sort the data based on the preset relationship between various files and tables in the big data file system and the windowing process of the data, because the sorting is transferred to the preset Processing in the big data file system avoids serial bottlenecks caused by forced sequential writing when data is written, thereby improving data processing efficiency.

Step 203, based on the latest data indicated by the time information, a logical view associated with the data in the first database is generated.

In this embodiment, based on the latest data indicated by the time information in step 202, the execution subject may generate a logical view associated with the data in the first database in various ways. As an example, the execution subject may first save the latest data indicated by the time information in the time sorting result obtained in step 202 . Then, the above-mentioned executive body can use the saved data to create a logical view, so that the file data can be read and queried through SQL (Structured Query Language, Structured Query Language).

In this embodiment, the above-mentioned executive body may provide the client to query and read data through the above-mentioned logical view.

Continuing to refer to FIG. 3 , FIG. 3 is a schematic diagram of an application scenario of a method for processing data according to an embodiment of the present disclosure. In the application scenario of FIG. 3 , the server 301 can acquire the parsing result set 302 of the target log file 304 based on parallel processing. Optionally, the above parsing result set 302 may be generated by the server 303 parsing and parsing the target log file 304 in parallel using multiple threads. Then, the server 301 may sort according to the time information of the data in the first database corresponding to the analysis results in the analysis result set 302 to generate the sorting result 303 . Next, based on the latest data indicated by the time information (such as "data x" shown in the figure), the server 301 can generate a logical view 305 associated with the data in the above-mentioned first database for other devices to read data. fetch and query.

At present, one of the existing technologies usually adopts a serial method when processing the Binlog process, and uses an asynchronous message queue to generate subsequent data processing files after Binlog parsing, so as to ensure that the downstream of the relational database Binlog parsing process The sequence and timeliness of data processing lead to the restriction of serial processing in order to ensure the consistency of data in the massive Binlog parsing scenario, which causes the delay of data synchronization. However, in the method provided by the above-mentioned embodiments of the present disclosure, data is sorted according to the time information of the data in the subsequent processing link by opening data parallel processing in the log parsing stage, and then a logical view is generated based on the data with the latest time. "Space for time" realizes the effect of not only improving the processing efficiency of data, but also ensuring the consistency of data.

Further referring to FIG. 4 , it shows a flow 400 of still another embodiment of a method for processing data. The flow 400 of the method for processing data includes the following steps:

Step 401 , acquiring a parsing result set of a target log file based on parallel processing.

Step 402: sort at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set.

Step 403, based on the latest data indicated by the time information, a logical view associated with the data in the first database is generated.

The above step 401, step 402, and step 403 are respectively consistent with step 201, step 202, step 203 and their optional implementations in the aforementioned embodiments, and the above is directed to step 201, step 202, step 203 and their optional implementations The description of the manner is also applicable to step 401, step 402 and step 403, which will not be repeated here.

Step 404, based on the data in the logical view and the time information corresponding to the data, create a temporary data table for the target time period.

In this embodiment, based on the data in the logical view and the time information corresponding to the data, the execution subject of the method for processing data (such as the server 103 shown in FIG. 1 ) can create temporary data of the target time period in various ways surface. Wherein, the above-mentioned target time period usually matches the time information corresponding to the above-mentioned data, and the data in the above-mentioned temporary data table usually includes the result after last data processing and the result of this incremental processing.

In this embodiment, as an example, the execution subject may periodically create a temporary table according to the data in the logical view generated in step 403 above. Wherein, the execution subject can use the time window to match the target time period corresponding to the temporary table with the time information corresponding to the data in the temporary table, that is, the time indicated by the time information corresponding to the data in the temporary table usually falls within above target time period.

It can be seen from FIG. 4 that the process 400 of the method for processing data in this embodiment embodies the step of creating a temporary data table for a target time period based on the data in the logical view and the time information corresponding to the data. Therefore, the scheme described in this embodiment can avoid excessive data from affecting the speed of data reading and query through the created temporary table of the target time period. At the same time, since each query is based on the last data processing The result set can improve the query speed of later data and improve the efficiency of use. In the case of frequent data reading and query, the efficiency can be significantly improved.

Further referring to FIG. 5 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for processing data, and the device embodiment corresponds to the method embodiment shown in FIG. 2 or FIG. 4 , The device can be specifically applied to various electronic devices.

As shown in FIG. 5 , an apparatus 500 for processing data provided in this embodiment includes an acquiring unit 501 , a sorting unit 502 and a generating unit 503 . Wherein, the acquiring unit 501 is configured to acquire the parsing result set of the target log file based on parallel processing, wherein the target log file is used to record the change of the first database; According to the time information of the data in the first database corresponding to the results, at least one piece of data is sorted; the generating unit 503 is configured to generate a logical view associated with the data in the first database based on the latest data indicated by the time information .

In this embodiment, in the device 500 for processing data: the specific processing of the acquiring unit 501, the sorting unit 502 and the generating unit 503 and the technical effects brought about by them can refer to the

steps

201, 201 and Relevant descriptions of step 202 and step 203 will not be repeated here.

In some optional implementations of this embodiment, the acquisition unit 501 may be further configured to: receive the parsing results of the target log files concurrently written by at least two consumers in the target distributed message queue system; The received parsing result generates a parsing result set.

In some optional implementation manners of this embodiment, the above parsing results may be respectively written into corresponding files in the preset big data file system according to the extracted message subject.

In some optional implementations of this embodiment, the above sorting unit 502 may be further configured to: generate a preset association relationship between files in the big data file system and preset metadata tables; The data time correlation field in the data table performs data windowing processing, wherein the data time correlation field matches the time information of the data in the first database; and performs time sorting on the data after the windowing processing.

In some optional implementations of this embodiment, the above-mentioned apparatus 500 for processing data may further include: a creation unit (not shown in the figure), configured to , to create a temporary data table for the target time period. Wherein, the above-mentioned target time period may be matched with the time information corresponding to the above-mentioned data. The data in the above-mentioned temporary data table may include the result after the last data processing and the result of this incremental processing.

In the device provided by the above-mentioned embodiments of the present disclosure, the analysis result set of the log files of the open data parallel processing obtained by the acquisition unit 501 in the log analysis stage, the sorting unit 502 sorts the data according to the time information of the data, and then the generating unit 503 based on The most recent data generates a logical view, which not only improves the processing efficiency of data, but also ensures the consistency of data by "trading space for time".

Further referring to FIG. 6 , it shows a sequence 600 of interaction between various devices in one embodiment of the system for processing data. The system for processing data may include: an analysis end (such as the server 102 shown in FIG. 1 ), and a data processing end (such as the server 103 shown in FIG. 1 ). Wherein, the above analysis terminal can be configured to write the analysis result of the obtained target log file into the target file system based on parallel processing, wherein the target log file is used to record the change of the first database. The above-mentioned data processing end may be configured to sort at least one piece of data according to the time information of the data in the first database corresponding to the analysis result; The associated logical view writes the parsed results to the target file system.

In some optional implementations of this embodiment, the parsing end may include: an acquisition unit configured to acquire a log file used to record changes in the first database as a target log file; an parsing unit configured to use at least The two threads parse the target log file to generate a parsing result; the writing unit is configured to write the generated parsing result into the target file system.

In some optional implementations of this embodiment, the parsing unit may be further configured to: use at least two threads to filter statements matching preset filter words in the target log file to generate a filtered target log file; Using at least two threads to analyze the filtered target log file to generate an analysis result.

In some optional implementations of this embodiment, the parsing end may be further configured to: extract the message topic from the parsing result obtained after parsing the target log file; Write concurrently to corresponding files in the preset big data file system.

As shown in FIG. 6 , in step 601 , based on parallel processing, the parsing end writes the parsing result of the acquired target log file into the target file system.

In this embodiment, the parsing end (for example, the server 102 in FIG. 1 ) may write the parsing result of the obtained target log file into the target file system based on various parallel processing methods. Wherein, the above-mentioned target log file may be consistent with the description in the foregoing embodiments, and will not be repeated here.

In some optional implementations of this embodiment, the above analysis end may write the analysis result of the obtained target log file into the target file system through the following steps:

Step 6011, acquire the log file for recording the change of the first database as the target log file.

In these implementation manners, the above-mentioned analyzing end may obtain a log file for recording changes of the first database from a communication-connected electronic device or locally as a target log file through a wired or wireless connection.

Step 6012, utilize at least two threads to analyze the target log file, and generate an analysis result.

In these implementation manners, the analysis end may use at least two threads to analyze the target log file acquired in step 6011 to generate an analysis result. As an example, the above parsing end can enable multi-threaded mode for data parsing, thereby modifying the existing sequential file parsing to concurrent file parsing.

It should be noted that the above parsing end may also determine to parse multiple files according to pre-configured parameters.

Based on the above optional implementation methods, this solution can parse the target log files in parallel on the parsing end, thereby significantly improving the efficiency of log parsing.

Optionally, the above parsing end can generate parsing results through the following steps:

Step 60121, use at least two threads to filter the statements matching the preset filter words in the target log file, and generate a filtered target log file.

In these implementations, the above-mentioned analyzing end may use the above-mentioned at least two threads to filter the sentences matching the preset filter words in the target log file acquired in step 6011, and generate a filtered target log file. As an example, the above parsing end may filter the statements according to the preset filter word settings during the processing of each target log file to be parsed. Wherein, the aforementioned preset filter words may include but not limited to at least one of the following: INSERT, UPDATE, DELETE, ALERT.

Based on the above optional implementation methods, this solution can shield unnecessary sentences during the sentence parsing process according to the needs of actual application scenarios, thereby significantly reducing the amount of parsed data and significantly improving the real-time performance of parsed file processing.

Step 60122, use at least two threads to analyze the filtered target log file, and generate an analysis result.

In these implementation manners, the above analysis end may use at least two threads to analyze the target log file filtered in the above step 60121 in a corresponding log analysis manner, and generate an analysis result.

Step 6013, write the generated parsing result into the target file system.

In these implementation manners, the parsing end may write the parsing result generated in step 6012 into the target file system in various ways. As an example, the above parsing end may directly write the generated parsing results into the target file system using multiple threads. As another example, the above analysis end may also be a Kafka system with concurrent consumption enabled. Then, the producer (producer) of the above-mentioned Kafka system may first store the generated parsing result in the message queue. Then, multiple consumers (consumers) of the above-mentioned Kafka system can concurrently consume the parsing result writing tasks in the above-mentioned message queue.

In some optional implementations of this embodiment, the above parsing end may write the generated parsing results into the above target file system through the following steps:

Step 60131, extract the message topic from the parsed result of the acquired target log file.

In these implementation manners, the above parsing end may extract the message topic from the parsing result after parsing the acquired target log file in various ways. Wherein, the above-mentioned message subject may be a field included in the above-mentioned parsing result. The above parsing end may also use at least two threads to extract the above message body in parallel.

Step 60132, according to the extracted message subject, concurrently write the parsing result to the corresponding file in the preset big data file system.

In these implementations, according to the extracted message subject, the analysis end may concurrently write the analysis results to corresponding files in the preset big data file system in various ways. Wherein, different files may be set in the preset big data file system according to different message topics. Therefore, the analysis end can write the analysis results into corresponding files in the preset big data file system according to the message subject extracted in step 60131. As an example, the above-mentioned big data file system may be HDFS (Hadoop Distributed File System, Hadoop Distributed File System).

In step 602, according to the time information of the data in the first database corresponding to the parsing result, the data processing end sorts at least one piece of data.

In step 603, based on the latest data indicated by the time information, the data processing end generates a logical view associated with the data in the first database.

The above-mentioned steps 602 and 603 are respectively consistent with steps 201 to 203 and their optional implementations in the foregoing embodiments, and the above descriptions for steps 201 to 203 and their optional implementations are also applicable to steps 602 and 602. Step 603, which will not be repeated here.

In the system for processing data provided by the above-mentioned embodiments of the present disclosure, firstly, based on parallel processing, the parsing end writes the parsing result of the obtained target log file into the target file system, wherein the above-mentioned target log file is used for Record the change of the first database; then, the data processing end sorts at least one piece of data according to the time information of the data in the first database corresponding to the analysis result; finally, based on the latest data indicated by the time information, the data processing end A logical view associated with data in the first database is generated. In this way, the parsing end can be used to write the parsing results in parallel to improve processing efficiency; and the data processing end sorts the data in chronological order to generate a logical view to avoid data inconsistency caused by writing data out of order, so as to realize the exchange of space for parallelism Data processing time, improve data processing efficiency.

Referring now to FIG. 7 , it shows a schematic structural diagram of an electronic device (such as the server 103 in FIG. 1 ) 700 suitable for implementing embodiments of the present disclosure. The server shown in FIG. 7 is only an example, and should not limit the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 7 , an electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) Various appropriate actions and processes are executed by programs in the memory (RAM) 703 . In the RAM 703, various programs and data necessary for the operation of the electronic device 700 are also stored. The processing device 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to the bus 704 .

Generally, the following devices can be connected to the I/O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device including, for example, a liquid crystal display (LCD, Liquid Crystal Display), a speaker, a vibrator, etc. 707; storage means 708 including eg magnetic tape, hard disk, etc.; and communication means 709. The communication means 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 7 shows electronic device 700 having various means, it should be understood that implementing or having all of the means shown is not a requirement. Additional or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 7 may represent one device, or may represent multiple devices as required.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 709, or from storage means 708, or from ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . The program code contained on the computer readable medium can be transmitted by any appropriate medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned server; it may also exist independently without being assembled into the server. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the server, the server: obtains a parsing result set of the target log file based on parallel processing, wherein the target log file is used for Record the change of the first database; sort at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set; generate the latest data corresponding to the first database based on the time information indicated A logical view of data association in

Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, Also included are conventional procedural programming languages—such as "C," the Python language, or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units can also be set in a processor, for example, can be described as: a processor, including an acquisition unit, a sorting unit, and a generation unit. Among them, the names of these units do not constitute a limitation of the unit itself in some cases. For example, the acquisition unit can also be described as "a unit that acquires the parsing result set of the target log file based on parallel processing, wherein the target log file is used to record changes to the first database".

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the above-mentioned invention without departing from the above-mentioned inventive concept. Other technical solutions formed by any combination of technical features or equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features having similar functions disclosed in (but not limited to) the embodiments of the present disclosure.

Claims

A method for processing data comprising:

Acquiring a parsing result set of a target log file based on parallel processing, wherein the target log file is used to record changes of the first database;

Sorting at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set;

Based on the latest data indicated by the time information, a logical view associated with the data in the first database is generated.
The method according to claim 1, wherein said obtaining the analysis result set of the target log file based on parallel processing comprises:

receiving parsing results of the target log file written concurrently by at least two consumers in the target distributed message queuing system;

Based on the received analysis results, the analysis result set is generated.
The method according to claim 2, wherein the parsing results are respectively written into corresponding files in the preset big data file system according to the extracted message topics.
The method according to claim 3, wherein said sorting at least one piece of data according to the time information of the data in the first database corresponding to the analysis results in the analysis result set includes:

Generate the association relationship between the files in the preset big data file system and the preset metadata table;

performing data windowing processing according to the data time correlation field in the preset metadata table, wherein the data time correlation field matches the time information of the data in the first database;

Perform time sorting on the data after the windowing processing.
The method according to any one of claims 1-4, wherein the method further comprises:

Based on the data in the logical view and the time information corresponding to the data, create a temporary data table for the target time period, wherein the target time period matches the time information corresponding to the data, and the data in the temporary data table Including the results after the last data processing and the results of this incremental processing.
A device for processing data, comprising:

An acquisition unit configured to acquire a parsing result set based on a parallel processing target log file, wherein the target log file is used to record changes of the first database;

The sorting unit is configured to sort at least one piece of data according to the time information of the data in the first database corresponding to the parsing results in the parsing result set;

A generating unit configured to generate a logical view associated with the data in the first database based on the latest data indicated by the time information.
A system for processing data comprising:

The parsing end is configured to write the parsing result of the obtained target log file into the target file system based on parallel processing, wherein the target log file is used to record changes of the first database;

The data processing end is configured to sort at least one piece of data according to the time information of the data in the first database corresponding to the parsing result; A logical view of data associations in a database.
The system according to claim 7, wherein the analysis end comprises:

an acquisition unit configured to acquire a log file for recording changes of the first database as a target log file;

The analysis unit is configured to use at least two threads to analyze the target log file and generate an analysis result;

The writing unit is configured to write the generated parsing result into the target file system.
The system according to claim 8, wherein the parsing unit is further configured to:

Utilize the at least two threads to filter the statements that hit the preset filter word in the target log file, and generate the filtered target log file;

Utilize the at least two threads to analyze the filtered target log file to generate an analysis result.
The system according to claim 7, wherein the parsing end is further configured to:

Extracting the message topic from the parsing result after parsing the acquired target log file;

According to the extracted message subject, the parsing result is concurrently written to a corresponding file in the preset big data file system.
A server comprising:

one or more processors;

a storage device having one or more programs stored thereon;

When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method in any one of claims 1-5.
A computer-readable medium, on which a computer program is stored, wherein, when the program is executed by a processor, the method according to any one of claims 1-5 is realized.
A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-5.