CN113641640A

CN113641640A - Data processing method, device, equipment and medium for streaming computing system

Info

Publication number: CN113641640A
Application number: CN202110970525.8A
Authority: CN
Inventors: 孙英富; 汪婷; 石然; 朱良昌; 田勇; 龚宓
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-08-23
Filing date: 2021-08-23
Publication date: 2021-11-12
Anticipated expiration: 2041-08-23
Also published as: CN113641640B

Abstract

The present disclosure provides a data processing method, apparatus, device and medium for a streaming computing system, and relates to the field of computers, in particular to the field of distributed computing and the field of streaming computing. The method comprises the following steps: reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating the data file and can indicate the storage information of the data file; reading the data file into an internal memory according to the log file; generating a second operation instruction, wherein the second operation instruction has the same function as the at least one first operation instruction; requesting to write a second operation instruction into a log file; and executing the second operation instruction in response to determining that the second operation instruction has been written into the log file.

Description

Data processing method, device, equipment and medium for streaming computing system

Technical Field

The present disclosure relates to the field of computers, and in particular, to the field of distributed computing and streaming computing, and in particular, to a data processing method and apparatus for a streaming computing system, an electronic device, a computer-readable storage medium, and a computer program product.

Background

The streaming computing system needs to store state data and progress information in a persistent mode during running, all data can be completely recovered when a process exits unexpectedly, and data are guaranteed not to be lost again. The streaming computing job is composed of a plurality of distributed computing nodes, each node is an indispensable computing node in the whole job, and has no redundant part, so that the streaming computing system needs to ensure that each node is in a normal survival state.

Cloud computing (cloud computing) refers to a technology architecture that accesses a flexibly extensible shared physical or virtual resource pool through a network, where resources may include servers, operating systems, networks, software, applications, storage devices, and the like, and may be deployed and managed in an on-demand, self-service manner. Through the cloud computing technology, high-efficiency and strong data processing capacity can be provided for technical application and model training of artificial intelligence, block chains and the like.

The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered as having been acknowledged in any prior art.

Disclosure of Invention

The present disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for a streaming computing system.

According to an aspect of the present disclosure, there is provided a data processing method for a streaming computing system, including: reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating the data file and can indicate the storage information of the data file; reading the data file into an internal memory according to the log file; generating a second operation instruction, wherein the second operation instruction has the same function as the at least one first operation instruction; requesting to write a second operation instruction into a log file; and executing the second operation instruction in response to determining that the second operation instruction has been written into the log file.

According to another aspect of the present disclosure, there is provided a data processing apparatus for a streaming computing system, comprising: the reading unit is configured to read a log file, wherein the log file comprises at least one executed first operation instruction, the first operation instruction is used for operating the data file and can indicate storage information of the data file, and the reading unit is further configured to read the data file into an internal memory according to the log file; an instruction unit configured to generate a second operation instruction, the second operation instruction being functionally identical to the at least one first operation instruction; a request unit configured to request writing of a second operation instruction into the log file; and an execution unit configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the data processing method for a streaming computing system described above.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the above-described data processing method for a streaming computing system.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described data processing method for a streaming computing system.

According to one or more embodiments of the present disclosure, by recording operations on all data files using a log file, storage information (e.g., file name, address, etc.) of the data files can be recorded, so that when performing failover, compared to requesting a return data file directory to a file system, the method of the present disclosure can quickly read the data files to continue a corresponding streaming computing job while reducing the burden on the file system. In addition, by trying to write data into the log file and executing the corresponding operation instruction on the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is solved, the system stability is improved, and the system failure rate is reduced.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the embodiments and, together with the description, serve to explain the exemplary implementations of the embodiments. The illustrated embodiments are for purposes of illustration only and do not limit the scope of the claims. Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements.

Fig. 1 shows a flow diagram of a data processing method for a streaming computing system according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a file system according to an embodiment of the present disclosure;

3A-3B illustrate schematic diagrams of executing an operation instruction on a data file according to an embodiment of the disclosure;

FIG. 4 illustrates a flow diagram of a file system according to an embodiment of the present disclosure;

5A-5B illustrate schematic diagrams of executing an operation instruction on a data file according to an embodiment of the disclosure;

FIG. 6 shows a flow diagram of a data processing method for a streaming computing system according to an embodiment of the present disclosure;

FIG. 7 illustrates a flowchart for executing a third operational instruction according to an embodiment of the present disclosure;

FIG. 8 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 9 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

fig. 10 shows a block diagram of the structure of an execution unit of a data processing apparatus according to an embodiment of the present disclosure; and

FIG. 11 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, the timing relationship, or the importance relationship of the elements, and such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, based on the context, they may also refer to different instances.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the elements may be one or more. Furthermore, the term "and/or" as used in this disclosure is intended to encompass any and all possible combinations of the listed items.

In the related art, when the abnormal condition of the computing node is detected, the computing node can be replaced by a new computing node to take over the specific task of the abnormal node under the streaming computing operation; or the streaming computing system is reloaded to the latest checkpoint in its entirety, thereby restoring the system integrity to the last normal state. During the process of replacing or reloading the computing node as a whole, a request is made to the file system to return a directory of data files, so as to read the data of the recording point. However, when the failure rate is high, the file system needs to frequently traverse the data file directory, thereby causing a large burden to the file system and even the streaming computing system.

To solve the above problem, by recording operations on all data files using a log file, storage information (e.g., file name, address, etc.) of the data files can be recorded, so that when performing failover, the method of the present disclosure can quickly read the data files to continue the corresponding streaming computing job, while reducing the load on the file system, compared to requesting a return of a data file directory to the file system. In addition, by trying to write data into the log file and executing the corresponding operation instruction on the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is solved, the system stability is improved, and the system failure rate is reduced.

According to an aspect of the present disclosure, a data processing method for a streaming computing system is provided. As shown in fig. 1, the data processing method includes: step S101, reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file; step S102, reading the data file into a memory according to the log file; step S103, generating a second operation instruction, wherein the second operation instruction has the same function as the at least one first operation instruction; step S104, requesting to write a second operation instruction into a log file; and step S105, in response to determining that the second operation instruction has been written into the log file, executing the second operation instruction.

Thus, by recording operations on all data files by using the log file, the storage information (such as file name, address and the like) of the data files can be recorded, so that when the failover is carried out, compared with the request for returning the data file directory to the file system, the method disclosed by the invention can rapidly read the data files to continue the corresponding streaming computing operation, and simultaneously reduce the load of the file system. In addition, by trying to write data into the log file and executing the corresponding operation instruction on the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is solved, the system stability is improved, and the system failure rate is reduced.

According to some embodiments, a streaming computing system may include a plurality of running compute nodes and a file system. Multiple computing nodes work together to complete a streaming computing job, with each computing node completing a portion of the tasks in the streaming computing job. The compute nodes interact with the file system to read information from and write information to the file system. The compute node may include memory units for temporary reading and writing and processing units for performing tasks. It is understood that the data processing method of the present disclosure can be combined with cloud computing technology to implement a data processing method for a streaming computing system based on cloud computing.

According to some embodiments, a set of log files and data files corresponding to each partial task are included in the file system. In an exemplary embodiment, the log file and the data file may be stored in a log directory and a data directory, respectively, and the log directory and the data directory may be located under the same level directory. The streaming computing system may further comprise a storage unit for storing files in a file system such as log files, data files, etc.

According to some embodiments, at least one first operation instruction that was previously executed may be included in the log file. The instructions may be generated and executed by a currently normally operating computing node that executes the data processing method, or may be generated and executed by other computing nodes that have terminated operation or abnormally operate, which is not limited herein.

According to some embodiments, the data File may include a plurality of data files of different types, and the data File may include at least one of a Sequence File (Sequence File) and a Random File (Random File). The data in the sequence file may be, for example, yield information with sequence numbers, and the data in the random file may be, for example, randomly accessed status information. Each type of data file may also include a plurality of files of different names, and each file may also include a plurality of attribute fields, e.g., fields public, private, etc., representing access rights. During operation of the compute node, the necessary information is written to the data files, and during failover, the compute node reads the valid information into the compute node's memory unit, as will be described below.

The file system will be described below by way of an exemplary embodiment. As shown in fig. 2, the log file (log) is stored in a log directory (MetaFile), and the Sequence file and the Random file are stored in two data file directories (Sequence, Random), respectively. The sequence file comprises two files of 'sfile 1' and 'sfile 2', and the random file comprises 'rfile 1' and one file.

According to some embodiments, the first operation instruction may be for operating a data file and may be capable of indicating storage information of the data file. It should be noted that, in the present disclosure, the first operation instruction, the second operation instruction, and the fourth operation instruction may all have the same function, that is, all may be used for operating the data file, and can indicate the storage information of the data file, and for convenience of describing the scheme of the present disclosure, they are distinguished according to the difference of their sources and generation times. The first operation instruction, the second operation instruction, and the fourth operation instruction may be, for example, instructions generated by different computing nodes, or may be instructions generated by the same computing node, which is not limited herein.

It is to be noted that the third operation instruction in the present disclosure has a function different from the above-described operation instruction for performing an operation within the log directory, as will be described later.

According to some embodiments, the operation instruction (MetaEntry) for the data file may include at least one of a create data file instruction (createenentry) and a delete data file instruction (DeleteEntry). The create data file instruction may create a data file with a specified file name, and the delete data file instruction may delete the specified data file. In an exemplary embodiment, after creating a sequence data file with the file name "sfile 3" is performed on the file system shown in FIG. 2, the results shown in FIG. 3A may be obtained. In another exemplary embodiment, the result shown in FIG. 3B may be obtained after deleting the sequence data file having the file name "sfile 2" from the file system shown in FIG. 2. Thus, functions of creating a data file and deleting a data file can be realized.

According to some embodiments, the data file may include at least one data block, and a size of each of the at least one data block is not greater than a preset value. In an exemplary embodiment, as shown in FIG. 4, the sequence data file "sfile 1" includes two data blocks numbered "0" and "1", the sequence data file "sfile 2" includes one data block numbered "0", and the random data file "rfile 1" includes three data blocks numbered "0", "1", and "2".

Therefore, the data file is divided into the data blocks, the problem that the single data file is too large and is not easy to read and write is solved, the load pressure of the file system is reduced, and the stability of the computing system is improved. It is understood that, a person skilled in the art can set the preset value of the size of the data block according to the requirement, and is not limited herein.

According to some embodiments, the operation instruction on the data file may further include at least one of a switch data block instruction and a delete data block instruction. The switch data block instruction may indicate to add a data block in a corresponding data file and to switch data writes to the data file to the added data block. In one exemplary embodiment, after executing the switch data block instruction for "sfile 2" to the file system as shown in fig. 4, the sequence data file "sfile 2" adds a new data block numbered "1" while setting other data blocks of the file (i.e., data block numbered "0") to read only (readonly) to switch the subsequent data writing to the file to the newly added data block numbered "1", as shown in fig. 5A. The delete data block instruction indicates deletion of at least one data block of the corresponding data file. The delete data block command may delete a single data block, multiple consecutive data blocks, or multiple non-consecutive data blocks of the corresponding data file, which is not limited herein. In an exemplary embodiment, after executing an instruction to the file system as shown in fig. 4 to delete data blocks numbered "0" and "1" for "rfile 1", data blocks numbered "0" and "1" in the random data file "rfile 1" are deleted, and only data block numbered "2" is retained, as shown in fig. 5B. Thus, switching and deleting of data blocks in the data file can be realized.

According to some embodiments, in step S102, reading the data file into the memory according to the log file, and in step S103, generating a second operation instruction, where the second operation instruction has the same function as the at least one first operation instruction, the computing node reads the data file into the memory, and executes a task assigned to the computing node with the streaming computing job. During the process of executing the task, the computing node further generates data and needs to write the data into a data file. It is noted that for the data writing process, the second operation instruction may be generated only when generating a new data file and switching to a new data block, while writing to an existing data file and an existing data block does not require generation of the second operation instruction.

As described above, there are typically multiple running compute nodes in a streaming computing system. When the abnormal condition occurs to the computing node, a new logic node is needed to replace the abnormal condition, so that the flow computing job can be rapidly resumed under the condition that other computing nodes are not influenced. However, in some cases, an anomalous compute node may not be able to be terminated such that multiple compute nodes are doing the same thing at the same time. Nodes that should not survive in such a forecast but are in a running state are often referred to as "zombie nodes".

In order to solve the problem of the zombie nodes, when the abnormal condition of the nodes is detected, the whole streaming computing system can be reloaded to the nearest check point, so that the system is completely restored to the last normal state. By using the method, the influence of botnet nodes on the system can be eliminated, but when the failure rate is high, frequent restarting of all the instances can cause a large burden on the flow type computing system.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S601, obtaining exclusive write authority of the log file. The operations of step S603 to step S607 in fig. 6 are similar to the operations of step S101 to step S105 in fig. 1, and are not described again here. Therefore, by acquiring the exclusive write-in permission of the log file, other computing nodes included in the stream computing system do not have the write-in permission of the log file, so that the interference of bots on the log file is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, the TruncateLease semantic may be used to obtain exclusive write permissions to log files.

According to some embodiments, step S601 may include obtaining exclusive write rights to the data file. Therefore, by acquiring the exclusive write-in permission of the data file, other computing nodes included in the streaming computing system do not have the write-in permission of the data file, so that the interference of a zombie node on the data file is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, TruncateLease semantics may likewise be used to obtain exclusive write rights to a data file.

According to some embodiments, the acquisition of the exclusive write permission of the log file and the data file can be performed immediately after the computing node is started, so that the operation of the files by the zombie node in a time period from the starting of the computing node to the acquisition permission can be avoided. And after the current node is abnormal, the new computing node replacing the current node can immediately acquire the exclusive write-in permission of the files, so that the write-in permission of the current node which becomes a zombie node to the files is released, the correctness of the log files and the data files is ensured, and the normal operation of the streaming computing system is ensured.

According to some embodiments, the step S104 of requesting to write the second operation instruction to the log file may include requesting to write the second operation instruction to the log file from the file system. The file system may determine whether the current compute node still has write rights. In some embodiments, when the file system determines that the current compute node has write permission, the second operation instruction is written to the log file. In an exemplary embodiment, after writing the second operation instruction, a response message may be sent to the current compute node to indicate that the second operation instruction has successfully written the log file. In other embodiments, when the file system determines that the current computing node no longer has the write permission, that is, the write permission of the current computing node has been replaced by another computing node, the second operation instruction is not written into the log file, so as to avoid abnormal writing of the log file by the zombie node.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S608, in response to detecting that the preset condition is met, executing a third operation instruction, where the third operation instruction is used to perform an operation in the log directory where the log file is located. It is to be understood that the present disclosure does not limit the sequence relationship between step S608 and steps S603 to S607. In an exemplary embodiment, if it is detected that the preset condition is satisfied before the steps S603 to S607 are performed, the third operation instruction may be performed before the steps S603 to S607 are performed. In another exemplary embodiment, step S608 and its possible sub-steps may also be interspersed or alternated with step S603-step S607, which is not limited herein.

According to some embodiments, the third operational instruction includes at least one of a create log file instruction, a move log file instruction, and a delete log file instruction. Therefore, when the preset condition is met, operations such as creating the log file, moving the log file, deleting the log file and the like in the rerog directory are realized. It is to be understood that, a person skilled in the art may set the preset condition corresponding to the specific third operation instruction according to a requirement, and the preset condition is not limited herein.

According to some embodiments, as shown in fig. 7, the step S608 of executing the third operation instruction may include, for example: step S701, compressing the log file to create a temporary log file; step S702, indicating blocking waiting, and suspending writing of the log file during the blocking waiting period; step S703, in response to determining that one or more fourth operation instructions are written into the log file during compression, writing one or more fourth operation instructions into the temporary log file; step S704, deleting the log file, and taking the temporary log file as a compressed log file; and step S705, instructing the release of the blocking wait. Therefore, by compressing the log file, the size of the log file can be reduced, the efficiency of failover is improved, and the utilization rate of storage resources of the streaming computing system is improved.

According to some embodiments, the step S701 of compressing the log file to create the temporary log file may include, for example, reading the log file into a memory of the computing node, and then compressing the operation instruction included in the log file to generate the temporary log file. The compression of the operation instructions included in the log file may include, for example, removing all operation instructions related to the deleted data file. The generated temporary log file may be stored in a temporary file subdirectory under the log directory.

According to some embodiments, step S702, indicating blocking waiting, may comprise, for example, instructing the file system to suspend writing to the log file, and may comprise placing the blocked write request in a waiting queue.

According to some embodiments, step S703, in response to determining that the one or more fourth operation instructions are written to the log file during compression, writing the one or more fourth operation instructions to the temporary log file may include, for example, recording the operation instructions written to the log file during compression of the log file. Since these operation instructions occur after the log file is read, these operation instructions are not included in the memory of the compute node. By writing these operation instructions into the temporary log file, the accuracy of the compressed log file can be ensured.

According to some embodiments, the step S704 of deleting the log file, and taking the temporary log file as a compressed log file may include, for example, deleting an original log file, and moving the temporary log file from the temporary file subdirectory into the log directory.

According to some embodiments, the step S705 of instructing unblocking waiting may include, for example, instructing the file system to unblock and start processing a write request in the waiting sequence, and then writing a subsequent operation instruction into the compressed log file, thereby completing the compression of the log file.

According to some embodiments, when the third operation instruction is a compression instruction for the log file, the preset condition may be, for example, that the time length from the last compression exceeds a preset time length, that the number of operation instructions of the operation data file recorded in the current log file exceeds a preset threshold, or another condition for triggering execution of the third operation instruction, which is not limited herein.

According to some embodiments, when writing operation instructions for operating the data file, the log file records sequence numbers of the operation instructions so as to order and search the operation instructions.

According to some embodiments, the file name of the log file may include the sequence number of the last operation instruction recorded by the previous log file when it was last compressed, i.e., when the log file was generated. Thus, whether the preset condition for executing the third operation instruction for compression is met can be judged by comparing the sequence number of the operation instruction currently written in the log file with the sequence number in the file name of the log file.

According to some embodiments, the file name of the log file may also include a timestamp of the last compression, i.e., when the log file was generated. Thus, it is possible to determine whether a preset condition for executing the third operation instruction for compression is satisfied by comparing the time stamp of the current time and the time stamp in the file name of the log file. In an exemplary embodiment, as shown in fig. 4, the file name "metallic.20.1624285613" of the log file includes the sequence number "20" of the last operation instruction recorded by the previous log file at the last compression time, and includes the time stamp "1624285613" at the last compression time.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S602, acquiring an exclusive operation authority for performing an operation in the log directory. Therefore, by acquiring the exclusive operation authority of the log directory, other computing nodes included in the stream computing system do not have the operation authority of the log directory, so that the interference of bots on log files and the log directory is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, SetOwnDir semantics may be used to obtain exclusive operating permissions to operate within a log directory.

According to some embodiments, similar to the exclusive write permission to the log file, the acquisition of the exclusive operation permission to operate in the log directory may be performed immediately after the computing node is started, so that the botnet node can be prevented from operating the log directory in a time period between the computing node being started and the acquisition permission. And after the current node is abnormal, the new computing node replacing the current node can immediately acquire the exclusive operation authority in the log directory, so that the exclusive operation authority of the current node which becomes the zombie node in the log directory is removed, the log file and the log directory are ensured not to be influenced by the zombie node, and the normal operation of the streaming computing system is ensured.

According to another aspect of the present disclosure, there is also provided a data processing apparatus for a streaming computing system. As shown in fig. 8, the data processing apparatus 800 includes: the reading unit 810 is configured to read a log file, where the log file includes at least one executed first operation instruction, and the first operation instruction is used to operate the data file and can indicate storage information of the data file, and the reading unit 810 is further configured to read the data file into an internal memory according to the log file; an instruction unit 820 configured to generate a second operation instruction, the second operation instruction being functionally identical to the at least one first operation instruction; a requesting unit 830 configured to request writing of the second operation instruction to the log file; and an execution unit 840 configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

The operation of the unit 810 of the data processing apparatus 800 is similar to the operation of the steps S101 to S102 of the data processing method, and the operation of the units 820 to 840 of the data processing apparatus 800 is similar to the operation of the steps S103 to S105 of the data processing method, respectively, and is not repeated herein.

According to some embodiments, the second operation instruction comprises at least one of a create data file instruction and a delete data file instruction.

According to some embodiments, the data file includes at least one data block, and a size of each of the at least one data block is not greater than a preset value. The second operation instruction includes at least one of a switch data block instruction and a delete data block instruction. The switch data block instruction instructs to add a data block in a corresponding data file and instructs to switch data writing to the data file to the added data block. The delete data block instruction indicates deletion of at least one data block of the corresponding data file.

According to some embodiments, as shown in fig. 9, the data processing apparatus 900 further comprises: an obtaining unit 910 configured to obtain an exclusive write permission to the log file. The operations of the units 920 to 950 in fig. 9 are similar to the operations of the units 810 to 840 in fig. 8, respectively, and are not repeated herein.

According to some embodiments, the execution unit 950 is further configured to, in response to detecting that the preset condition is met, execute a third operation instruction, where the third operation instruction is used to perform an operation in the log directory where the log file is located.

According to some embodiments, the third operational instruction includes at least one of a create log file instruction, a move log file instruction, and a delete log file instruction. As shown in fig. 10, the execution unit 950 includes: a compression subunit 1051 configured to compress the log file to create a temporary log file; an indication subunit 1052 configured to indicate a blocking wait during which the log file suspends writing; a write subunit 1053 configured to write one or more fourth operation instructions to the temporary log file in response to determining that the one or more fourth operation instructions are written to the log file during compression; and a delete subunit 1054 that deletes the log file and takes the temporary log file as a compressed log file. The indication subunit 1052 is further configured to indicate unblocking waiting.

The operations of the units 1052-1054 in fig. 10 are similar to the operations of steps S701-S704 in fig. 7, respectively, and the operation of the unit 1052 is further similar to the operation of step S705 in fig. 7, which is not repeated here.

According to some embodiments, the obtaining unit 910 is further configured to obtain an exclusive operation right to operate within the log directory.

According to an embodiment of the present disclosure, there is also provided an electronic device, a readable storage medium, and a computer program product.

Referring to fig. 11, a block diagram of a structure of an electronic device 1100, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 11, the device 1100 comprises a computing unit 1101, which may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM1103, various programs and data necessary for the operation of the device 1100 may also be stored. The calculation unit 1101, the ROM 1102, and the RAM1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

A number of components in device 1100 connect to I/O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the device 1100, and the input unit 1106 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote control. Output unit 1107 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. Storage unit 1108 may include, but is not limited to, a magnetic or optical disk. The communication unit 1109 allows the device 1100 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, 1302.11 devices, WiFi devices, WiMax devices, cellular communication devices, and/or the like.

The computing unit 1101 can be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The calculation unit 1101 performs the respective methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1100 via ROM 1102 and/or communication unit 1109. When the computer program is loaded into RAM1103 and executed by the computing unit 1101, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be performed in parallel, sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the above-described methods, systems and apparatus are merely exemplary embodiments or examples and that the scope of the present invention is not limited by these embodiments or examples, but only by the claims as issued and their equivalents. Various elements in the embodiments or examples may be omitted or may be replaced with equivalents thereof. Further, the steps may be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced with equivalent elements that appear after the present disclosure.

Claims

1. A data processing method for a streaming computing system, comprising:

reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file;

reading the data file into an internal memory according to the log file;

generating a second operation instruction, wherein the second operation instruction has the same function as the at least one first operation instruction;

requesting to write the second operation instruction into the log file; and

executing the second operation instruction in response to determining that the second operation instruction has been written to the log file.

2. The method of claim 1, further comprising:

and acquiring exclusive write permission of the log file.

3. The method of claim 1, wherein the second operation instruction comprises at least one of a create data file instruction and a delete data file instruction.

4. The method of claim 1, wherein the data file comprises at least one data block, each of the at least one data block having a size not greater than a preset value.

5. The method of claim 4, wherein the second operation instruction comprises at least one of a switch data block instruction and a delete data block instruction,

wherein the switch data block instruction instructs to add a data block in a corresponding data file and instructs to switch data writing to the data file to the added data block, and

wherein the delete data block instruction indicates to delete at least one data block of the corresponding data file.

6. The method of claim 1, wherein the data file comprises at least one of a sequence file and a random file.

7. The method of claim 1, further comprising:

and executing a third operation instruction in response to the detection that the preset condition is met, wherein the third operation instruction is used for operating in the log directory where the log file is located.

8. The method of claim 7, further comprising:

and acquiring exclusive operation authority for operating in the log directory.

9. The method of claim 7, wherein the third operational instruction comprises at least one of a create log file instruction, a move log file instruction, and a delete log file instruction.

10. The method of claim 7, wherein executing the third operational instruction comprises:

compressing the log file to create a temporary log file;

indicating a blocking wait during which the log file suspends writes;

in response to determining that one or more fourth operational instructions were written to the log file during the compressing, writing the one or more fourth operational instructions to the temporary log file;

deleting the log file, and taking the temporary log file as a compressed log file; and

indicating to release the blocking wait.

11. A data processing apparatus for a streaming computing system, comprising:

a reading unit configured to read a log file, the log file including at least one executed first operation instruction, the first operation instruction being used to operate a data file and being capable of indicating storage information of the data file,

the reading unit is further configured to read the data file into a memory according to the log file;

an instruction unit configured to generate a second operation instruction, the second operation instruction being functionally identical to the at least one first operation instruction;

a request unit configured to request writing of the second operation instruction to the log file; and

an execution unit configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

12. The apparatus of claim 11, further comprising:

an acquisition unit configured to acquire an exclusive write permission to the log file.

13. The apparatus of claim 11, wherein the second operation instruction comprises at least one of a create data file instruction and a delete data file instruction.

14. The apparatus of claim 11, wherein the data file comprises at least one data block, a size of each of the at least one data block is not greater than a preset value,

wherein the second operation instruction comprises at least one of a switch data block instruction and a delete data block instruction,

15. The apparatus according to claim 11, wherein the execution unit is further configured to execute, in response to detecting that a preset condition is met, a third operation instruction for operating in a log directory in which the log file is located.

16. The apparatus according to claim 15, wherein the acquiring unit is further configured to acquire an exclusive operation authority to operate within the log directory.

17. The apparatus of claim 15, wherein the third operational instruction comprises at least one of a create log file instruction, a move log file instruction, and a delete log file instruction, and wherein the execution unit comprises:

a compression subunit configured to compress the log file to create a temporary log file;

an indication subunit configured to indicate a blocking wait during which the log file suspends writing;

a write subunit configured to write one or more fourth operational instructions to the temporary log file in response to determining that the one or more fourth operational instructions were written to the log file during the compressing; and

a deletion subunit that deletes the log file and takes the temporary log file as a compressed log file,

wherein the indication subunit is further configured to indicate to release the blocking wait.

18. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

19. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-10.

20. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-10 when executed by a processor.