CN113641640B

CN113641640B - Data processing method, device, equipment and medium for stream type computing system

Info

Publication number: CN113641640B
Application number: CN202110970525.8A
Authority: CN
Inventors: 孙英富; 汪婷; 石然; 朱良昌; 田勇; 龚宓
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-08-23
Filing date: 2021-08-23
Publication date: 2023-07-07
Anticipated expiration: 2041-08-23
Also published as: CN113641640A

Abstract

The present disclosure provides a data processing method, apparatus, device and medium for a streaming computing system, and relates to the field of computers, in particular to the distributed computing field and the streaming computing field. The method comprises the following steps: reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file; reading the data file into the memory according to the log file; generating a second operation instruction, wherein the second operation instruction has the same function as at least one first operation instruction; requesting to write a second operation instruction into the log file; and in response to determining that the second operation instruction has been written to the log file, executing the second operation instruction.

Description

Data processing method, device, equipment and medium for stream type computing system

Technical Field

The present disclosure relates to the field of computers, and more particularly to the field of distributed computing and the field of streaming computing, and in particular to a data processing method, apparatus, electronic device, computer readable storage medium and computer program product for a streaming computing system.

Background

The streaming computing system needs to store state data and progress information in a lasting mode when running, and can still completely recover all data when a process accidentally exits, so that the data is ensured not to be heavy and lost. The streaming computing job is composed of a plurality of distributed computing nodes, each of which is an indispensable computing node in the whole job, and has no redundant part, so that the streaming computing system needs to ensure that each node is in a normal survival state.

Cloud computing (cloud computing) refers to a technical system that accesses an elastically extensible shared physical or virtual resource pool through a network, wherein resources can include servers, operating systems, networks, software, applications, storage devices and the like, and can be deployed and managed in an on-demand and self-service manner. Through cloud computing technology, high-efficiency and powerful data processing capability can be provided for technical application such as artificial intelligence and blockchain, and model training.

The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.

Disclosure of Invention

The present disclosure provides a data processing method, apparatus, electronic device, computer readable storage medium and computer program product for a streaming computing system.

According to an aspect of the present disclosure, there is provided a data processing method for a streaming computing system, including: reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file; reading the data file into the memory according to the log file; generating a second operation instruction, wherein the second operation instruction has the same function as at least one first operation instruction; requesting to write a second operation instruction into the log file; and in response to determining that the second operation instruction has been written to the log file, executing the second operation instruction.

According to another aspect of the present disclosure, there is provided a data processing apparatus for a streaming computing system, comprising: a reading unit configured to read a log file, the log file including at least one executed first operation instruction, the first operation instruction being used for operating the data file and being capable of indicating storage information of the data file, wherein the reading unit is further configured to read the data file into the memory according to the log file; an instruction unit configured to generate a second operation instruction, the second operation instruction having the same function as the at least one first operation instruction; a request unit configured to request writing of the second operation instruction to the log file; and an execution unit configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method for a streaming computing system described above.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the above-described data processing method for a streaming computing system.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the data processing method for a streaming computing system described above.

According to one or more embodiments of the present disclosure, by recording operations on all data files using log files, storage information (e.g., file names, addresses, etc.) of the data files can be recorded, so that upon failover, the method of the present disclosure can quickly read the data files to continue with the corresponding streaming computing job while reducing the burden on the file system, as compared to requesting return of the data file directory to the file system. In addition, by firstly attempting to write the data into the log file and then executing the corresponding operation instruction of the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is avoided, the stability of the system is improved, and the failure rate of the system is reduced.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.

FIG. 1 illustrates a flow chart of a data processing method for a streaming computing system in accordance with an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a file system according to an embodiment of the present disclosure;

3A-3B illustrate diagrams of executing operational instructions for a data file according to embodiments of the present disclosure;

FIG. 4 illustrates a flow chart of a file system according to an embodiment of the present disclosure;

5A-5B illustrate diagrams of executing operational instructions for a data file according to embodiments of the present disclosure;

FIG. 6 illustrates a flow chart of a data processing method for a streaming computing system in accordance with an embodiment of the present disclosure;

FIG. 7 illustrates a flowchart of executing a third operational instruction according to an embodiment of the present disclosure;

FIG. 8 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 9 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 10 shows a block diagram of the structure of an execution unit of a data processing apparatus according to an embodiment of the present disclosure; and

fig. 11 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.

The terminology used in the description of the various illustrated examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.

In the related technology, when the abnormal condition of the computing node is detected, the computing node can be replaced by a new computing node to take over the abnormal node to continue to execute the specific task under the streaming computing job; or reloading the whole streaming computing system to the latest check point, thereby completely restoring the system to the last normal state. During the replacement or overall reloading of a compute node, it is necessary to request a return directory of data files from the file system to read the data of the record point. However, at higher failure rates, the file system needs to traverse the data file directory frequently, thus placing a large burden on the file system and even on the streaming computing system.

In order to solve the above problems, by recording operations on all data files using log files, storage information (e.g., file names, addresses, etc.) of the data files can be recorded, so that the method of the present disclosure can rapidly read the data files to continue corresponding streaming computation operations while reducing the burden of the file system, compared to requesting return of the data file directory to the file system, when performing failover. In addition, by firstly attempting to write the data into the log file and then executing the corresponding operation instruction of the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is avoided, the stability of the system is improved, and the failure rate of the system is reduced.

According to one aspect of the present disclosure, a data processing method for a streaming computing system is provided. As shown in fig. 1, the data processing method includes: step S101, reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file; step S102, reading a data file into a memory according to the log file; step S103, generating a second operation instruction, wherein the second operation instruction has the same function as at least one first operation instruction; step S104, a second operation instruction is requested to be written into the log file; and step S105, in response to determining that the second operation instruction has been written into the log file, executing the second operation instruction.

Thus, by using the log file to record the operations on all the data files, the storage information (such as file name, address, etc.) of the data files can be recorded, so that the method of the disclosure can quickly read the data files to continue the corresponding streaming calculation operation, and simultaneously reduce the burden of the file system compared with the request of returning the data file directory to the file system when performing the failover. In addition, by firstly attempting to write the data into the log file and then executing the corresponding operation instruction of the data file after successful writing is detected, the problem that the log file is inconsistent with the data file due to the fact that the operation instruction is still executed on the data file when the log file cannot be written is avoided, the stability of the system is improved, and the failure rate of the system is reduced.

According to some embodiments, a streaming computing system may include a plurality of running computing nodes and a file system. Multiple compute nodes cooperate to complete a streaming computing job, with each compute node completing a portion of the tasks in the streaming computing job. The compute nodes interact with the file system to read and write information from and to the file system. The compute node may include a memory unit for temporary reading and writing and a processing unit for performing tasks. It is to be appreciated that the data processing methods of the present disclosure can be combined with cloud computing techniques to implement data processing methods for cloud computing-based streaming computing systems.

According to some embodiments, a set of log files and data files corresponding to each portion of the task are included in the file system. In one exemplary embodiment, the log file and the data file may be stored in a log directory and a data directory, respectively, and the log directory and the data directory may be located under the same level directory. The streaming computing system may further comprise a storage unit for storing files in a file system, such as log files, data files, etc.

According to some embodiments, the log file may include at least one first operation instruction previously executed therein. These instructions may be previously generated and executed by a computing node that is currently executing the data processing method described above that is operating normally, or may be previously generated and executed by other computing nodes that have terminated or are operating abnormally, without limitation.

According to some embodiments, the data File may include a plurality of data files of different types, and the data File may include at least one of a Sequence File (Sequence File) and a Random File (Random File). The data in the sequence file may be, for example, yield information with a sequence number, and the data in the random file may be, for example, random access status information. Each type of data file may also include a plurality of differently named files, and each file may also include a plurality of attribute fields, such as field public, private, etc., that represent access rights. When the computing node is running, the necessary information is written into these data files, and when the failover occurs, the computing node reads the valid information into the memory unit of the computing node, as will be described below.

The file system will be described below by way of an exemplary embodiment. As shown in fig. 2, a log file (metal) is stored in a log directory (MetaFile), and a sequence file and a random file are respectively stored in two data file directories (Sequence, random). The sequence file comprises two files of "sfile1" and "sfile2", and the random file comprises one file of "rfile 1".

According to some embodiments, the first operation instruction may be used to operate on the data file and may be capable of indicating stored information of the data file. It should be noted that, in the present disclosure, the first operation instruction, the second operation instruction, and the fourth operation instruction may all have the same function, that is, may all be used to operate the data file, and may indicate the storage information of the data file, and in order to facilitate description of the solution of the present disclosure, they are distinguished according to different sources and generation times thereof. The first operation instruction, the second operation instruction, and the fourth operation instruction may be, for example, instructions generated by different computing nodes, or may be instructions generated by the same computing node, which is not limited herein.

It should be noted that the third operation instruction in the present disclosure has a function different from that of the operation instruction described above for performing an operation within the log directory, as will be described below.

According to some embodiments, the operation instruction (MetaEntry) for the data file may include at least one of a create data file instruction (CreateEntry) and a delete data file instruction (deleeentry). The create data file instruction may create a data file with a specified file name, and the delete data file instruction may delete the specified data file. In one exemplary embodiment, the results shown in FIG. 3A may be obtained after creating a sequence data file with a file name of "sfile3" performed on the file system shown in FIG. 2. In another exemplary embodiment, after deleting the sequence data file with the file name "sfile2" is performed on the file system shown in fig. 2, the result as shown in fig. 3B may be obtained. Thus, the functions of creating a data file and deleting a data file can be realized.

According to some embodiments, the data file may include at least one data block, each of the at least one data block having a size not greater than a preset value. In one exemplary embodiment, as shown in fig. 4, the sequence data file "sfile1" includes two data blocks numbered "0" and "1", the sequence data file "sfile2" includes one data block numbered "0", and the random data file "rfile1" includes three data blocks numbered "0", "1", and "2".

Therefore, by dividing the data file into data blocks, the problem that a single data file is too large to read and write is avoided, so that the load pressure of a file system is reduced, and the stability of a computing system is improved. It is understood that the preset value of the data block size can be set by a person skilled in the art according to the requirement, and the method is not limited herein.

According to some embodiments, the operation instructions for the data file may further include at least one of a switch data block instruction and a delete data block instruction. The switch data block instruction may instruct to add a data block in a corresponding data file and instruct to switch data writing to the data file to the added data block. In one exemplary embodiment, after executing the switch data block instruction for "sfile2" on the file system as shown in fig. 4, the sequence data file "sfile2" is incremented by one new data block numbered "1", while the other data blocks of the file (i.e., the data blocks numbered "0") are set to read-only (read only) to switch the subsequent data write to the file to the newly incremented data block numbered "1", as shown in fig. 5A. The delete data block instruction indicates deletion of at least one data block of the corresponding data file. The delete data block instruction may delete a single data block, a plurality of consecutive data blocks, or a plurality of non-consecutive data blocks of the corresponding data file, without limitation. In one exemplary embodiment, after executing the instruction for deleting the data blocks numbered "0" and "1" for "rfile1" on the file system as shown in fig. 4, the data blocks numbered "0" and "1" in the random data file "rfile1" are deleted, and only the data block numbered "2" remains as shown in fig. 5B. Thereby, switching and deleting of data blocks in the data file can be achieved.

According to some embodiments, in step S102, the data file is read into the memory according to the log file, and in step S103, a second operation instruction is generated, where the second operation instruction has the same function as the at least one first operation instruction, the computing node reads the data file into the memory, and performs a task assigned to the computing node with the streaming computing job. During execution of the task, the computing node may further generate data and may need to write the data to the data file. It is noted that for the data writing process, the second operation instruction may be generated only when generating a new data file and switching to a new data block, whereas writing an existing data file and an existing data block does not require generating the second operation instruction.

As described above, there are typically multiple running computing nodes in a streaming computing system. When an abnormal condition occurs to a computing node, a new logic node needs to be used for replacing the abnormal condition, so that the running of the streaming computing job can be quickly resumed without affecting other computing nodes. However, in some cases, the anomalous computing node may not be terminated such that multiple computing nodes are doing the same at the same time. Nodes in such a forecast that should not survive but be in operation are often referred to as "zombie nodes".

In order to solve the zombie node problem, when an abnormal condition of the node is detected, the whole streaming computing system can be reloaded to the latest check point, so that the system is completely restored to the last normal state. By using the method, the influence of the zombie nodes on the system can be eliminated, but when the failure rate is high, frequent restarting of all the examples can cause a great burden on the streaming computing system.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S601, obtaining exclusive write permission for the log file. The operations of step S603 to step S607 in fig. 6 are similar to those of step S101 to step S105 in fig. 1, and are not described here. Therefore, by acquiring the exclusive write-in authority of the log file, other computing nodes included in the streaming computing system do not have the write-in authority of the log file, so that interference of the zombie nodes on the log file is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, exclusive write rights to log files may be obtained using the mount release semantics.

According to some embodiments, step S601 may include obtaining an exclusive write right to a data file. By acquiring the exclusive write-in authority of the data file, other computing nodes included in the streaming computing system do not have the write-in authority of the data file, so that interference of the zombie nodes on the data file is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, exclusive write rights to a data file may also be obtained using the mount release semantics.

According to some embodiments, the acquisition of exclusive write rights to log files and data files may be performed immediately after the computing node is started, so that zombie nodes can be prevented from operating on the files in the time period between the starting of the computing node and the acquisition of the rights. After the current node is abnormal, the new computing node which replaces the current node can immediately acquire exclusive write-in permission of the files, so that the write-in permission of the current node which becomes a zombie node to the files is relieved, the correctness of the log files and the data files is ensured, and the normal operation of the streaming computing system is ensured.

According to some embodiments, step S104, requesting to write the second operation instruction to the log file may include requesting to write the second operation instruction to the log file from the file system. The file system may determine whether the current computing node still has write rights. In some embodiments, the second operation instruction is written to the log file when the file system determines that the current computing node has write permission. In one exemplary embodiment, after writing the second operation instruction, a receipt message may be sent to the current computing node to indicate that the second operation instruction has been successfully written to the log file. In other embodiments, when the file system determines that the current computing node no longer has write permission, i.e., the write permission of the current computing node has been replaced by another computing node, the second operation instruction is not written to the log file, so as to avoid abnormal writing of the log file by the zombie node.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S608, in response to detecting that the preset condition is met, executing a third operation instruction, where the third operation instruction is used to perform an operation in the log directory where the log file is located. It is to be understood that the present disclosure does not limit the sequential relationship between step S608 and steps S603-S607. In an exemplary embodiment, when it is detected that the preset condition is satisfied before performing the steps S603 to S607, the third operation instruction may be performed before performing the steps S603 to S607. In another exemplary embodiment, the step S608 and the possible sub-steps thereof may also be interleaved or alternated with the steps S603-S607, which is not limited herein.

According to some embodiments, the third operational instructions include at least one of a create log file instruction, a move log file instruction, and a delete log file instruction. Thus, when the preset condition is met, operations such as creating the log file, moving the log file and deleting the log file in the log directory are realized. It is understood that, the preset conditions corresponding to the specific third operation instruction can be set by a person skilled in the art according to the requirement, and the preset conditions are not limited herein.

According to some embodiments, as shown in fig. 7, step S608, executing the third operation instruction may include, for example: step S701, compressing the log file to create a temporary log file; step S702, indicating blocking waiting, wherein the log file pauses writing during the blocking waiting; step S703, in response to determining that the one or more fourth operation instructions write the log file during compression, writing the one or more fourth operation instructions to the temporary log file; step S704, deleting the log file, and taking the temporary log file as a compressed log file; and step S705, instruction to unblock waiting. Therefore, the size of the log file can be reduced by compressing the log file, the efficiency of failure transfer is improved, and the storage resource utilization rate of the streaming computing system is improved.

According to some embodiments, the compressing the log file to create the temporary log file in step S701 may include, for example, reading the log file into a memory of the computing node, and then compressing an operation instruction included in the log file to generate the temporary log file. The compression of the operation instructions included in the log file may include, for example, eliminating all operation instructions related to the deleted data file. The generated temporary log file may be stored in a temporary file subdirectory under the log directory.

According to some embodiments, step S702, indicating to block waiting may include, for example, indicating that the file system is suspending writing to the log file, and may include placing the blocked write request in a waiting queue.

According to some embodiments, step S703, in response to determining that the one or more fourth operation instructions were written to the log file during the compression, writing the one or more fourth operation instructions to the temporary log file may include, for example, recording the operation instructions written to the log file during the compression of the log file. Since these operational instructions occur after reading the log file, they are not included in the memory of the compute node. By writing these operation instructions into the temporary log file, the accuracy of the compressed log file can be ensured.

According to some embodiments, step S704, deleting the log file, and taking the temporary log file as a compressed log file may include, for example, deleting the original log file, and moving the temporary log file from the temporary file sub-directory into the log directory.

According to some embodiments, step S705, indicating unblock waiting may include, for example, indicating a file system unblock state and beginning to process a write request in a waiting sequence, and further writing a subsequent operation instruction to the compressed log file, thereby completing the compression of the log file.

According to some embodiments, when the third operation instruction is a compression instruction for the log file, the preset condition may be, for example, a time period longer than the last compression, or the number of operation instructions for detecting the operation data file recorded by the current log file exceeds a preset threshold, or other conditions for triggering execution of the third operation instruction, which are not limited herein.

According to some embodiments, the log file, when writing the operation instructions for operating the data file, records the sequence numbers of the operation instructions to facilitate ordering and searching of the operation instructions.

According to some embodiments, the file name of the log file may include the sequence number of the last operation instruction of the previous log file record at the last compression, i.e., at the time the log file was generated. Thus, it is possible to judge whether or not the preset condition for executing the third operation instruction for compression is satisfied by comparing the sequence number of the operation instruction currently written in the log file with the sequence number in the file name of the log file.

According to some embodiments, the file name of the log file may also include a timestamp of the last compression, i.e., when the log file was generated. Thus, it is possible to determine whether the preset condition for executing the third operation instruction for compression is satisfied by comparing the timestamp of the current time with the timestamp in the file name of the log file. In one exemplary embodiment, as shown in FIG. 4, the file name "metal.20.1624285613" of the log file includes the sequence number "20" of the last operation instruction of the previous log file record at the last compression, and includes the timestamp "1624285613" at the last compression.

According to some embodiments, as shown in fig. 6, the data processing method may further include: step S602, obtaining an exclusive operation right for operating in the log directory. Therefore, by acquiring the exclusive operation authority to the log directory, other computing nodes included in the streaming computing system do not have the operation authority to the log directory, so that interference of zombie nodes to the log file and the log directory is avoided, the stability of the system is improved, and the failure rate of the system is reduced. In one exemplary embodiment, the setownedir semantics may be used to obtain exclusive operation rights to operate within a journal directory.

According to some embodiments, similar to the exclusive write permission to the log file, the acquisition of the exclusive operation permission to operate in the log directory may be performed immediately after the start of the computing node, so that the operation of the log directory by the zombie node in a period between the start of the computing node and the acquisition of the permission can be avoided. After the current node is abnormal, the new computing node which replaces the current node can immediately acquire the exclusive operation authority in the log directory, so that the exclusive operation authority of the current node which becomes the zombie node in the log directory is relieved, the fact that the log file and the log directory are not affected by the zombie node is ensured, and the normal operation of the streaming computing system is ensured.

According to another aspect of the present disclosure, there is also provided a data processing apparatus for a streaming computing system. As shown in fig. 8, the data processing apparatus 800 includes: a reading unit 810 configured to read a log file, the log file including at least one executed first operation instruction, the first operation instruction being used for operating the data file and being capable of indicating storage information of the data file, wherein the reading unit 810 is further configured to read the data file into the memory according to the log file; an instruction unit 820 configured to generate a second operation instruction, the second operation instruction being functionally identical to the at least one first operation instruction; a request unit 830 configured to request writing of the second operation instruction to the log file; and an execution unit 840 configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

The operations of the unit 810 of the data processing apparatus 800 are similar to those of the above-described steps S101 to S102 of the data processing method, and the operations of the unit 820 to the unit 840 of the data processing apparatus 800 are similar to those of the above-described steps S103 to S105 of the data processing method, respectively, and are not described herein.

According to some embodiments, the second operation instructions include at least one of a create data file instruction and a delete data file instruction.

According to some embodiments, the data file comprises at least one data block, each of the at least one data block having a size not greater than a preset value. The second operation instruction includes at least one of a switch data block instruction and a delete data block instruction. The switch data block instruction instructs to add a data block in the corresponding data file and instructs to switch the writing of data to the data file to the added data block. The delete data block instruction indicates deletion of at least one data block of the corresponding data file.

According to some embodiments, as shown in fig. 9, the data processing apparatus 900 further comprises: an acquisition unit 910 configured to acquire exclusive write rights to the log file. The operations of the units 920-950 in fig. 9 are similar to the operations of the units 810-840 in fig. 8, respectively, and are not described here again.

According to some embodiments, the execution unit 950 is further configured to execute, in response to detecting that the preset condition is met, a third operation instruction for performing an operation within the log directory in which the log file is located.

According to some embodiments, the third operational instructions include at least one of a create log file instruction, a move log file instruction, and a delete log file instruction. As shown in fig. 10, the execution unit 950 includes: a compression subunit 1051 configured to compress the log file to create a temporary log file; an indication subunit 1052 configured to indicate a blocking wait during which the log file pauses writing; a write subunit 1053 configured to write the one or more fourth operation instructions to the temporary log file in response to determining that the one or more fourth operation instructions were written to the log file during compression; and a deletion subunit 1054 that deletes the log file, and takes the temporary log file as a compressed log file. The indication subunit 1052 is further configured to indicate a unblock wait.

The operations of the units 1052 to 1054 in fig. 10 are similar to those of the steps S701 to S704 in fig. 7, respectively, and the operation of the unit 1052 is further similar to the step S705 in fig. 7, and a detailed description thereof will be omitted.

According to some embodiments, the obtaining unit 910 is further configured to obtain an exclusive operation right to operate within the log directory.

According to embodiments of the present disclosure, there is also provided an electronic device, a readable storage medium and a computer program product.

Referring to fig. 11, a block diagram of an electronic device 1100 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 11, the apparatus 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, ROM 1102, and RAM1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

Various components in device 1100 are connected to I/O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the device 1100, the input unit 1106 may receive input numeric or character information as well as generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and/or a remote control. The output unit 1107 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 1108 may include, but is not limited to, magnetic disks, optical disks. The communication unit 1109 allows the device 1100 to exchange information/data with other devices through computer networks such as the internet and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, 1302.11 devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.

The computing unit 1101 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the respective methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1108. In some embodiments, some or all of the computer programs may be loaded and/or installed onto device 1100 via ROM 1102 and/or communication unit 1109. When a computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the data processing method by any other suitable means (e.g. by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.

Claims

1. A data processing method for a streaming computing system, comprising:

Reading a log file, wherein the log file comprises at least one first operation instruction, and the first operation instruction is used for operating a data file and can indicate storage information of the data file;

reading the data file into a memory according to the log file;

generating a second operation instruction aiming at the data writing process, responding to the determination that a new data file needs to be generated, wherein the second operation instruction has the same function as the at least one first operation instruction, and the second operation instruction is used for continuing to perform streaming calculation operation corresponding to the at least one first operation instruction;

requesting writing the second operation instruction into the log file; and

the second operation instruction is executed in response to determining that the second operation instruction has been written to the log file.

2. The method of claim 1, further comprising:

and acquiring exclusive write permission of the log file.

3. The method of claim 1, wherein the second operation instruction comprises a create data file instruction.

4. The method of claim 1, wherein the data file comprises at least one data block, each of the at least one data block having a size not greater than a preset value.

5. The method of claim 4, wherein the second operation instruction comprises at least one of a switch data block instruction and a delete data block instruction,

wherein the switch data block instruction instructs to add a data block in a corresponding data file and instructs to switch data writing to the data file to the added data block, and

wherein the delete data block instruction indicates deletion of at least one data block of the corresponding data file.

6. The method of claim 1, wherein the data file comprises at least one of a sequence file and a random file.

7. The method of claim 1, further comprising:

and responding to the detection that the preset condition is met, executing a third operation instruction, wherein the third operation instruction is used for operating in the log catalog where the log file is located.

8. The method of claim 7, further comprising:

and acquiring the exclusive operation authority for operating in the log catalog.

9. The method of claim 7, wherein the third operation instruction comprises at least one of a create log file instruction, a move log file instruction, and a delete log file instruction.

10. The method of claim 7, wherein executing the third operational instruction comprises:

compressing the log file to create a temporary log file;

indicating a blocking wait during which the log file pauses writing;

writing one or more fourth operating instructions to the temporary log file in response to determining that the one or more fourth operating instructions were written to the log file during the compression;

deleting the log file and taking the temporary log file as a compressed log file; and

indicating to unblock the waiting.

11. A data processing apparatus for a streaming computing system, comprising:

a reading unit configured to read a log file including at least one executed first operation instruction for operating a data file and capable of indicating stored information of the data file,

the reading unit is further configured to read the data file into a memory according to the log file;

an instruction unit configured to generate, for a data writing process, in response to determining that a new data file needs to be generated, a second operation instruction, the second operation instruction being functionally identical to the at least one first operation instruction and for continuing a streaming computation job corresponding to the at least one first operation instruction;

A request unit configured to request writing of the second operation instruction to the log file; and

and an execution unit configured to execute the second operation instruction in response to determining that the second operation instruction has been written to the log file.

12. The apparatus of claim 11, further comprising:

and an acquisition unit configured to acquire exclusive write rights to the log file.

13. The apparatus of claim 11, wherein the second operation instructions comprise create data file instructions.

14. The apparatus of claim 11, wherein the data file comprises at least one data block, each of the at least one data block having a size not greater than a preset value,

wherein the second operation instruction includes at least one of a switch data block instruction and a delete data block instruction,

15. The apparatus of claim 11, wherein the execution unit is further configured to execute a third operation instruction for operating within a log directory in which the log file is located in response to detecting that a preset condition is satisfied.

16. The apparatus of claim 15, wherein the obtaining unit is further configured to obtain exclusive operation rights for operations within the log directory.

17. The apparatus of claim 15, wherein the third operation instruction comprises at least one of a create log file instruction, a move log file instruction, and a delete log file instruction, and wherein the execution unit comprises:

a compression subunit configured to compress the log file to create a temporary log file;

an indication subunit configured to indicate a blocking wait during which the log file suspends writing;

a writing subunit configured to write one or more fourth operating instructions to the temporary log file in response to determining that the one or more fourth operating instructions were written to the log file during the compression; and

A deletion subunit deleting the log file and taking the temporary log file as a compressed log file,

wherein the indication subunit is further configured to indicate to unblock the waiting.

18. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the method comprises the steps of

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

19. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-10.

20. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-10.