CN114650287B - Breakpoint continuous method, device and system for data file - Google Patents
Breakpoint continuous method, device and system for data file Download PDFInfo
- Publication number
- CN114650287B CN114650287B CN202210354068.4A CN202210354068A CN114650287B CN 114650287 B CN114650287 B CN 114650287B CN 202210354068 A CN202210354068 A CN 202210354068A CN 114650287 B CN114650287 B CN 114650287B
- Authority
- CN
- China
- Prior art keywords
- data
- file
- breakpoint
- target
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011437 continuous method Methods 0.000 title claims description 4
- 238000012545 processing Methods 0.000 claims abstract description 142
- 238000000034 method Methods 0.000 claims abstract description 86
- 230000008569 process Effects 0.000 claims abstract description 45
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012795 verification Methods 0.000 claims description 16
- 230000002093 peripheral effect Effects 0.000 claims description 14
- 230000005540 biological transmission Effects 0.000 abstract description 22
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 230000006854 communication Effects 0.000 description 19
- 238000004891 communication Methods 0.000 description 18
- 238000003860 storage Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 230000004044 response Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a breakpoint continuing method, device and system of a data file, which can be used in the technical field of artificial intelligence, and the method comprises the following steps: when a breakpoint is detected to exist in the processing process of the data file to be processed, inquiring a breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to the field of artificial intelligence technologies, and in particular, to a method, an apparatus, and a system for breakpoint continuous of a data file.
Background
The traditional bank adopts a large host as a core business system so as to improve the system operation capability and meet the transaction requirement of the core business. In the related art, a peripheral platform system sequentially performs service processing on a plurality of data files through a general file transmission application, a large host side acquires the processed data files at fixed time to realize file transmission, when the service processing on a certain data file fails due to rapid increase of data volume, the service processing needs to be performed again from a first data file, the processing efficiency of the data file is low, and the waiting time for acquiring the processed data file at the large host side is prolonged, so that the transmission efficiency of the data file is low.
Disclosure of Invention
The invention aims to provide a breakpoint continuous method for a data file, which can continuously process business from a breakpoint in the data file, improve the processing efficiency of the data file, reduce the waiting time of a host side and further improve the transmission efficiency of the data file. Another object of the present invention is to provide a breakpoint resume device for a data file. It is yet another object of the present invention to provide a breakpoint resume system for a data file. It is yet another object of the present invention to provide a computer readable medium. It is a further object of the invention to provide a computer device.
In order to achieve the above purpose, the present invention discloses a breakpoint continuing method for a data file, wherein the data file includes a plurality of data units; comprising the following steps:
when a breakpoint is detected to exist in the processing process of the data file to be processed, inquiring a breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed;
And starting to perform service processing from the breakpoint data unit to generate a processed target data unit.
Preferably, according to the identification value of the data unit in the data file to be processed, the breakpoint data unit is queried from the index file, including:
inquiring the processing state of the data unit from the index file according to the identification value of the data unit;
and determining the data unit with the processing state to be processed as a breakpoint data unit.
Preferably, the method further comprises:
when all data units in the data file to be processed complete service processing, verifying the attribute information of all generated target data units;
if the verification is passed, generating a target file according to all the generated target data units;
storing the target file to a preset target address, and sending a processing completion message to the host server, wherein the processing completion message comprises the target address so that the host server can acquire the target file according to the target address.
Preferably, the attribute information includes a data size and a data length;
Verifying attribute information of all generated target data units, including:
Counting the data size and the data length of the target data unit;
Matching the identification value with the position sequence number of the file list in the shared memory, and inquiring the target size and the target length of the corresponding data unit from the shared memory;
If the data size of each target data unit is the same as the corresponding target size and the data length is the same as the corresponding target length, checking is passed;
if the data size of at least one target data unit is different from the corresponding target size or the data length is different from the corresponding target length, the verification fails.
Preferably, after performing service processing from the breakpoint data unit and generating the processed target data unit, the method further includes:
updating the processing state corresponding to the breakpoint data unit in the index file to processed state;
The identification value is matched with the position serial number of the file list in the shared memory, and the processing state of the corresponding data unit is inquired from the shared memory;
The processing state of the data unit is updated to processed.
Preferably, after generating the target file according to all the generated target data units, the method further comprises:
deleting the corresponding data unit from the index file according to the identification value;
and deleting the corresponding data unit from the shared memory by matching the identification value with the position sequence number of the file list in the shared memory.
The invention also discloses a breakpoint continuous system of the data file, wherein the data file comprises a plurality of data units; the system comprises: a peripheral platform and a scheduling module; comprising the following steps:
The peripheral platform is used for sending the data file to be processed to the scheduling module;
the scheduling module is used for inquiring breakpoint data units from the index file according to the identification values of the data units in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed; and starting to perform service processing from the breakpoint data unit to generate a processed target data unit.
Preferably, the system further comprises: a host server;
The scheduling module is also used for checking the attribute information of all the generated target data units when all the data units in the data file to be processed complete service processing; if the verification is passed, generating a target file according to all the generated target data units; storing the target file to a preset target address;
The host server is used for receiving the processing completion message sent by the scheduling module, and the processing completion message comprises a target address; and acquiring the target file according to the target address.
The invention also discloses a breakpoint continuous device of the data file, wherein the data file comprises a plurality of data units; the device comprises:
the inquiry unit is used for inquiring the breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed;
And the service processing unit is used for performing service processing from the breakpoint data unit and generating a processed target data unit.
The invention also discloses a computer readable medium having stored thereon a computer program which when executed by a processor implements a method as described above.
The invention also discloses a computer device comprising a memory for storing information comprising program instructions and a processor for controlling the execution of the program instructions, the processor implementing the method as described above when executing the program.
The invention also discloses a computer program product comprising a computer program/instruction which, when executed by a processor, implements a method as described above.
When a breakpoint is detected to exist in the processing process of a data file to be processed, inquiring a breakpoint data unit from an index file according to an identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a breakpoint resume system of a data file according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for resuming a breakpoint of a data file according to an embodiment of the present invention;
FIG. 3 is a flowchart of another method for breakpoint resume of a data file according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a breakpoint resume device for a data file according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the breakpoint continuing method, device and system for data files disclosed by the application can be used in the technical field of artificial intelligence and can also be used in any field except the technical field of artificial intelligence, and the application field of the breakpoint continuing method, device and system for data files disclosed by the application is not limited.
In order to facilitate understanding of the technical scheme provided by the application, the following description will explain relevant contents of the technical scheme of the application. The peripheral platform system comprises a plurality of peripheral platforms, and each platform carries out service processing on different data files through a universal file transmission application according to self configuration to obtain target files. The business process may perform different processes according to different business requirements, including but not limited to splitting, decoding, and translating the data file. The large host side comprises a plurality of hosts, and the hosts are high-performance hosts for processing core service data in target files, so that the price is high, and resources are not easy to expand. Compared with the related art, when the data file processing abnormality occurs, the embodiment of the application keeps the temporary file so as to realize the function of breakpoint continuous operation. The embodiment of the application is based on a record-level breakpoint sequence, namely: when the system is restarted, the pointer offset position of the source file corresponding to the breakpoint data unit in the data file at the time of abnormal termination is restarted. The breakpoint of the data file is the pointer offset position of the source file element corresponding to the breakpoint data unit.
Fig. 1 is a schematic structural diagram of a breakpoint resume system of a data file according to an embodiment of the present invention, as shown in fig. 1, a peripheral platform 100 and a scheduling module 200.
The peripheral platform 100 is used to send the pending data file to the scheduling module 200. As an alternative, the data file to be processed is a transmission control protocol (Transmission Control Protocol, abbreviated as TCP) message. The first communication module is integrated in the peripheral platform 100, the second communication module is integrated in the scheduling module 200, the peripheral platform 100 sends the data file to be processed to the scheduling module 200 through the first communication module, and the scheduling module 200 receives the data file to be processed sent by the peripheral platform 100 through the second communication module. As an alternative, the protocol that the communication module communicates with is the TCP protocol, and the TCP protocol adopts a transmission response mechanism, namely: each TCP message sent by the sending end receives the response of the receiving end and considers that the TCP message is successfully transmitted, the TCP protocol adopts a timeout retransmission mechanism, the sending end starts a timer when sending a TCP message segment, and if the response is not received within the timing time, the sending end retransmits the message segment; TCP messages are sent in the form of Internet Protocol (IP) datagrams, which may be out of order and repeated to the receiving end, so that the TCP protocol also rearranges and sorts the received messages before delivery to the subsequent application layer. Therefore, communication between the peripheral platform 100 and the scheduling module 200 is realized through the TCP protocol, and reliability of communication and security of a data file to be processed can be ensured.
In the embodiment of the present invention, file information of a data file to be processed and file information of other external connection module files are stored in a database, and the database type may be a relational database, for example: oracle (Oracle) database.
The scheduling module 200 is configured to query a breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed; and starting to perform service processing from the breakpoint data unit to generate a processed target data unit.
In the embodiment of the invention, the system further comprises: the host server 300.
The scheduling module 200 is further configured to verify attribute information of all generated target data units when all data units in the data file to be processed complete service processing; if the verification is passed, generating a target file according to all the generated target data units; and storing the target file to a preset target address.
The host server 300 is configured to receive a processing completion message sent by the scheduling module, where the processing completion message includes a target address; and acquiring the target file according to the target address. The host server 300 completes the communication process with the peripheral platform 100 through the scheduling module 200.
In the embodiment of the present invention, a third communication module is integrated in the host server 300, the scheduling module 200 sends the processing completion message to the host server 300 through the second communication module, and the host server 300 receives the processing completion message sent by the scheduling module 200 through the third communication module. As an alternative, the protocol that the communication module communicates with is the TCP protocol, and the TCP protocol adopts a transmission response mechanism, namely: each TCP message sent by the sending end receives the response of the receiving end and considers that the TCP message is successfully transmitted, the TCP protocol adopts a timeout retransmission mechanism, the sending end starts a timer when sending a TCP message segment, and if the response is not received within the timing time, the sending end retransmits the message segment; TCP messages are sent in the form of Internet Protocol (IP) datagrams, which may be out of order and repeated to the receiving end, so that the TCP protocol also rearranges and sorts the received messages before delivery to the subsequent application layer. Therefore, the communication between the scheduling module 200 and the host server 300 is realized through the TCP protocol, and the reliability of the communication and the security of the process completion message can be ensured.
In the technical scheme provided by the embodiment of the invention, when the existence of a breakpoint is detected in the processing process of the data file to be processed, the breakpoint data unit is inquired from the index file according to the identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
It should be noted that the breakpoint resume system of the data file shown in fig. 1 is also applicable to the breakpoint resume method of the data file in fig. 2 or fig. 3, which is not repeated herein.
The implementation process of the breakpoint continuing method of the data file provided by the embodiment of the invention is described below by taking the breakpoint continuing device of the data file as an execution body as an example. It can be understood that the execution body of the breakpoint resume method of the data file provided in the embodiment of the present invention includes, but is not limited to, a breakpoint resume device of the data file.
Fig. 2 is a flowchart of a breakpoint resume method of a data file according to an embodiment of the present invention, where, as shown in fig. 2, the method includes:
step 101, when a breakpoint is detected in the processing process of the data file to be processed, inquiring the breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed.
In an embodiment of the present invention, the data file includes a plurality of data units.
And 102, starting business processing from the breakpoint data unit, and generating a processed target data unit.
According to the technical scheme provided by the embodiment of the invention, when the existence of the breakpoint is detected in the processing process of the data file to be processed according to the identification value of the data unit in the data file to be processed, the breakpoint data unit is inquired from the index file according to the identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
FIG. 3 is a flowchart of another method for breakpoint resume of a data file according to an embodiment of the present invention, as shown in FIG. 3, the method includes:
step 201, according to the identification value of the data unit, the processing state of the data unit is queried from the index file.
In the embodiment of the invention, each step is executed by the breakpoint continuous device of the data file, and the scheduling module is integrated in the breakpoint continuous device of the data file.
Specifically, when a breakpoint is detected to exist in the processing process of the data file to be processed, the position of the breakpoint in the data file to be processed needs to be queried.
In the embodiment of the invention, the data files are stored in a pre-divided shared memory, and the data file information stored in the shared memory comprises, but is not limited to, service types, processing states, storage paths and file list position sequence numbers of the data files. Each data file is provided with a storage path and a corresponding file list position serial number, wherein the file list position serial number is the storage position of the data file in the shared memory, and the file list position serial number can uniquely identify one data file.
In the embodiment of the invention, the data file comprises a plurality of data units, and the plurality of data units are orderly arranged in the data file. In order to ensure the data safety, an index file is created, the data units are stored in the index file in a linked list form, the data units in the data file to be processed are copied into the index file in sequence along with the business processing of the data units, and the data units are processed in the index file, so that the data safety can be ensured, and the loss caused by the irrecoverable data loss can be avoided. The index file includes, but is not limited to, a processing state of a data unit and an identification value, each identification value being capable of uniquely identifying a data unit, the processing state of the data unit being synchronized with the processing state of a corresponding data unit in the data file in the shared memory.
Further, since there are multiple processes in parallel processing in the breakpoint continuous process of the data file, when accessing the shared memory and the index file, mutual exclusive access is required through PV operation, that is: only one process is allowed to access the shared memory and index file at a time, where P operates as a request for resources (wait) and V operates as a release resource (signal). Specifically, for accessing a shared memory and an index file to enable a plurality of processes to mutually exclusive, setting a mutual exclusion semaphore (mutex) for the shared memory and the index file, and setting an initial value of the mutex semaphore (mutex) to be 1; and placing the critical sections of the shared memory and the index file accessed by each process between wait (mutex) and signal (mutex) operations. For example: two processes access the shared memory and index file, the initial value of mutex signal (mutex) is 1, and the range of values is (-1, 0, 1). When mutex=1, it means that neither process has access to the shared memory and index file; when mutex=0, it means that one process is ready to access the shared memory and index file, and the other process enters the set blocking queue to wait; when mutex = -1, it means that one process is accessing the shared memory and index file, and another process needs to wake up when the process currently accessing the shared memory and index file exits because it is waiting to exist in the blocking queue.
In the embodiment of the invention, the content and the number of the upstream files of the data units processed each time are fixed, and the identification values of the data units are fixedly ordered, so that the sequences of the data units obtained each time are completely consistent, and based on the sequences, the complete function of breakpoint continuous operation can be realized. As an alternative, the identification value of the data unit is the file name and directory information of the upstream data unit.
Step 202, determining the data unit with the processing state to be processed as a breakpoint data unit.
In the embodiment of the invention, the processing state of the data unit in the data file is inquired, wherein the processing state comprises to-be-processed or processed; and determining the data unit with the processing state to be processed as a breakpoint data unit, wherein the breakpoint data unit is the position of the breakpoint of the data file to be processed in the processing process. Specifically, if the processing state of the data unit is to be processed, indicating that the data unit is not processed for service, interrupting the data processing process at the data unit, and determining the data unit as a breakpoint data unit; if the processing state of the data unit is processed, the data unit is indicated to finish business processing, a target data unit obtained after business processing is generated, and the processing state of the next data unit is inquired from the index file until the breakpoint data unit is determined.
In the embodiment of the invention, the data unit corresponds to a service type, corresponding service processing is carried out on the data unit according to the actual requirement of the service type, a processed target data unit is generated, and the target data unit is a processing result obtained after the service processing of the data unit. It should be noted that different service types correspond to different service processing manners, and the specific service processing manner in the embodiment of the invention is not limited.
Further, the processing state corresponding to the breakpoint data unit in the index file is updated to be processed, so that repeated processing or number jumping processing of the breakpoint data unit when the interrupt is generated in subsequent processing and re-inquiry is avoided, and the data processing efficiency can be improved.
Further, the identification value of the data unit is matched with the position serial number of the file list in the shared memory, and the processing state of the corresponding data unit is inquired from the shared memory; the processing state of the data unit is updated to be processed, so that the processing state of the data unit in the index file and the processing state of the corresponding data unit in the shared memory can be ensured to be kept synchronous.
Step 203, when all data units in the data file to be processed complete service processing, verifying the attribute information of all generated target data units; if the verification is passed, executing step 204; if the verification fails, the process is ended.
In the embodiment of the invention, the attribute information comprises a data size and a data length.
In the embodiment of the present invention, step 203 specifically includes:
step 2031, counting the data size and the data length of the target data unit.
Step 2032, matching the identification value with the file list position sequence number in the shared memory, and querying the target size and the target length of the corresponding data unit from the shared memory.
In the embodiment of the invention, the shared memory also records a target size and a target length corresponding to the data unit, wherein the target size is the size which should be provided after the data unit is subjected to service processing, and the target length is the length which should be provided after the data unit is subjected to service processing. It should be noted that, the target size and the target length are preset according to the service condition of the data unit, which is not limited in the embodiment of the present invention.
Step 2033, judging whether the data size of each target data unit is the same as the corresponding target size and the data length is the same as the corresponding target length, if yes, executing step 204; if not, the process ends.
In the embodiment of the present invention, if the data size of each target data unit is the same as the corresponding target size and the data length is the same as the corresponding target length, indicating that the verification is passed, executing step 204; if the data size of at least one target data unit is different from the corresponding target size or the data length is different from the corresponding target length, the verification is failed, and the process is ended. If the verification fails, the data file processing is abnormal, and the processing and transmission are required to be carried out again.
In the embodiment of the invention, the target data unit is checked, so that the abnormality in the file processing process can be timely detected, and the subsequent host server is prevented from acquiring the abnormal target file.
And 204, generating a target file according to all the generated target data units.
In the embodiment of the invention, the target data units are arranged and spliced according to the sequence of the data units in the data file to generate the target file.
Further, according to the identification value, the corresponding data unit is deleted from the index file. Generating a target file, indicating that the data file is successfully processed, continuing to carry out subsequent transmission, emptying the data unit in the index file, and providing a copying space for the data unit of the next file to be processed.
Further, the corresponding data unit is deleted from the shared memory by matching the identification value with the file list position sequence number in the shared memory. And clearing the shared memory, and keeping consistency of the index file and the data content in the shared memory so as to carry out service processing on the next file to be processed.
Step 205, storing the target file to a preset target address, and sending a processing completion message to the host server, where the processing completion message includes the target address, so that the host server obtains the target file according to the target address.
In the embodiment of the invention, a processing completion message is sent to the host server so as to inform the host server that the processing of the data file is completed. The target address is preset and used for storing the generated target file so that the host server can actively acquire the target file, data transmission is completed, and the acquisition timeliness of the host server is improved.
In the technical scheme of the breakpoint continuing method of the data file, when the existence of the breakpoint is detected in the processing process of the data file to be processed, the breakpoint data unit is inquired from the index file according to the identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
Fig. 4 is a schematic structural diagram of a breakpoint resume device of a data file according to an embodiment of the present invention, where the device is configured to execute the breakpoint resume method of the data file, as shown in fig. 4, and the device includes: a first querying unit 11 and a service processing unit 12.
The query unit 11 is configured to query the index file for a breakpoint data unit according to the identification value of the data unit in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed.
The service processing unit 12 is configured to perform service processing from the breakpoint data unit, and generate a processed target data unit.
In the embodiment of the present invention, the query unit 11 is specifically configured to query, according to the identification value of the data unit, the processing state of the data unit from the index file; and determining the data unit with the processing state to be processed as a breakpoint data unit.
In the embodiment of the invention, the device further comprises: a verification unit 13, a generation unit 14 and a transmission unit 15.
The verification unit 13 is configured to verify attribute information of all generated target data units when all data units in the data file to be processed complete service processing.
The generating unit 14 is configured to generate a target file according to all the generated target data units if the verification passes.
The sending unit 15 is configured to store the target file to a preset target address, and send a processing completion message to the host server, where the processing completion message includes the target address, so that the host server obtains the target file according to the target address.
In the embodiment of the invention, the attribute information comprises data size and data length; the checking unit 14 is specifically configured to count the data size and the data length of the target data unit; matching the identification value with the position sequence number of the file list in the shared memory, and inquiring the target size and the target length of the corresponding file entry from the shared memory; if the data size of each target data unit is the same as the corresponding target size and the data length is the same as the corresponding target length, checking is passed; if the data size of at least one target data unit is different from the corresponding target size or the data length is different from the corresponding target length, the verification fails.
In the embodiment of the present invention, the updating unit 16 and the matching unit 17.
The updating unit 16 is configured to update a processing state corresponding to the breakpoint data unit of the index file to processed; the processing status of the file entry is updated to processed.
The matching unit 17 is configured to query the processing state of the corresponding file entry from the shared memory by matching the identification value with the file list location number in the shared memory.
In the embodiment of the invention, the device further comprises: the deletion unit 18.
The deleting unit 18 is configured to delete the corresponding data unit from the index file according to the identification value; and deleting the corresponding file entry from the shared memory by matching the identification value with the file list position serial number in the shared memory.
In the scheme of the embodiment of the invention, when the existence of the breakpoint is detected in the processing process of the data file to be processed, the breakpoint data unit is inquired from the index file according to the identification value of the data unit in the data file to be processed; and starting to perform service processing from the breakpoint data unit, generating a processed target data unit, and continuing to perform service processing from the breakpoint in the data file, so that the processing efficiency of the data file is improved, the waiting time of a host side is reduced, and the transmission efficiency of the data file is improved.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer device, which may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
The embodiment of the invention provides a computer device, which comprises a memory and a processor, wherein the memory is used for storing information comprising program instructions, the processor is used for controlling the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the steps of the embodiment of the breakpoint continuing method of the data file are realized.
Referring now to FIG. 5, there is illustrated a schematic diagram of a computer device 600 suitable for use in implementing embodiments of the present application.
As shown in fig. 5, the computer apparatus 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate works and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM)) 603. In the RAM603, various programs and data required for the operation of the computer device 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a liquid crystal feedback device (LCD), and the like, and a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on drive 610 as needed, so that a computer program read therefrom is mounted as needed as storage section 608.
In particular, according to embodiments of the present invention, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The technical scheme of the application obtains, stores, uses, processes and the like the data, which all meet the relevant regulations of national laws and regulations.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (11)
1. A breakpoint continuous method of a data file is characterized in that the data file comprises a plurality of data units which are orderly arranged; the method comprises the following steps:
When a breakpoint is detected in the processing process of a data file to be processed, inquiring breakpoint data units from an index file according to identification values of the data units in the data file to be processed, wherein the index file stores the data units in a linked list form, and the identification values of the data units are in fixed ordering;
And in the index file, starting to perform service processing from the breakpoint data unit, and generating a processed target data unit.
2. The method for breakpoint resume of data file according to claim 1, wherein said searching the breakpoint data unit from the index file according to the identification value of the data unit in the data file to be processed comprises:
inquiring the processing state of the data unit from an index file according to the identification value of the data unit;
and determining the data unit with the processing state to be processed as the breakpoint data unit.
3. The method of breakpoint resume of a data file according to claim 1, further comprising:
When all data units in the data file to be processed complete service processing, verifying the attribute information of all generated target data units;
if the verification is passed, generating a target file according to all the generated target data units;
Storing the target file to a preset target address, and sending a processing completion message to a host server, wherein the processing completion message comprises the target address so that the host server can acquire the target file according to the target address.
4. A breakpoint resume method for a data file according to claim 3, wherein the attribute information includes a data size and a data length;
The verifying the attribute information of all the generated target data units comprises the following steps:
Counting the data size and the data length of the target data unit;
matching the identification value with the position sequence number of the file list in the shared memory, and inquiring the target size and the target length of the corresponding data unit from the shared memory;
If the data size of each target data unit is the same as the corresponding target size and the data length is the same as the corresponding target length, checking is passed;
if the data size of at least one target data unit is different from the corresponding target size or the data length is different from the corresponding target length, the verification fails.
5. The method for breakpoint resume of a data file according to claim 1, wherein after said performing service processing from said breakpoint data unit to generate a processed target data unit, further comprising:
updating the processing state corresponding to the breakpoint data unit in the index file to processed state;
The identification value is matched with the position serial number of the file list in the shared memory, and the processing state of the corresponding data unit is inquired from the shared memory;
updating the processing state of the data unit to processed.
6. A method of breakpoint resume of a data file according to claim 3, further comprising, after said generating a target file from all of the generated target data units:
Deleting the corresponding data unit from the index file according to the identification value;
And deleting the corresponding data unit from the shared memory by matching the identification value with the file list position serial number in the shared memory.
7. A breakpoint continuous system of a data file, wherein the data file comprises a plurality of data units which are orderly arranged; the system comprises: a peripheral platform and a scheduling module;
The peripheral platform is used for sending a data file to be processed to the scheduling module;
The scheduling module is used for inquiring breakpoint data units from an index file according to identification values of the data units in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed, the data units are stored in the index file in a linked list form, and the identification values of the data units are fixedly ordered; and in the index file, starting to perform service processing from the breakpoint data unit, and generating a processed target data unit.
8. The breakpoint resume system of a data file according to claim 7, further comprising: a host server;
the scheduling module is further used for checking attribute information of all generated target data units when all data units in the data file to be processed complete service processing; if the verification is passed, generating a target file according to all the generated target data units; storing the target file to a preset target address;
The host server is configured to receive a processing completion message sent by the scheduling module, where the processing completion message includes the target address; and acquiring a target file according to the target address.
9. The breakpoint continuous making device of the data file is characterized in that the data file comprises a plurality of orderly arranged data units; the device comprises:
The query unit is used for querying breakpoint data units from the index file according to the identification values of the data units in the data file to be processed when the breakpoint is detected to exist in the processing process of the data file to be processed, wherein the data units are stored in the index file in a linked list form, and the identification values of the data units are fixedly ordered;
And the service processing unit is used for performing service processing from the breakpoint data unit in the index file and generating a processed target data unit.
10. A computer readable medium on which a computer program is stored, characterized in that the program, when executed by a processor, implements a breakpoint resume method for a data file according to any of claims 1 to 6.
11. A computer device comprising a memory for storing information including program instructions and a processor for controlling execution of the program instructions, wherein the program instructions when loaded and executed by the processor implement the breakpoint continued method of the data file according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210354068.4A CN114650287B (en) | 2022-04-06 | 2022-04-06 | Breakpoint continuous method, device and system for data file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210354068.4A CN114650287B (en) | 2022-04-06 | 2022-04-06 | Breakpoint continuous method, device and system for data file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114650287A CN114650287A (en) | 2022-06-21 |
| CN114650287B true CN114650287B (en) | 2024-04-30 |
Family
ID=81996774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210354068.4A Active CN114650287B (en) | 2022-04-06 | 2022-04-06 | Breakpoint continuous method, device and system for data file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114650287B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009140813A1 (en) * | 2008-05-23 | 2009-11-26 | 中兴通讯股份有限公司 | File downloading method, system and terminal for realizing continuous transmission from the breakpoint |
| CN106528275A (en) * | 2015-09-10 | 2017-03-22 | 网易（杭州）网络有限公司 | Processing method of data tasks and task scheduler |
| CN110895483A (en) * | 2018-09-12 | 2020-03-20 | 北京奇虎科技有限公司 | Task recovery method and device |
| CN113726842A (en) * | 2021-07-09 | 2021-11-30 | 北京房江湖科技有限公司 | File uploading method and device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11226748B2 (en) * | 2016-07-05 | 2022-01-18 | Red Hat Israel, Ltd | Differentiating open and abandoned transactions in a shared storage environment |
-
2022
- 2022-04-06 CN CN202210354068.4A patent/CN114650287B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009140813A1 (en) * | 2008-05-23 | 2009-11-26 | 中兴通讯股份有限公司 | File downloading method, system and terminal for realizing continuous transmission from the breakpoint |
| CN106528275A (en) * | 2015-09-10 | 2017-03-22 | 网易（杭州）网络有限公司 | Processing method of data tasks and task scheduler |
| CN110895483A (en) * | 2018-09-12 | 2020-03-20 | 北京奇虎科技有限公司 | Task recovery method and device |
| CN113726842A (en) * | 2021-07-09 | 2021-11-30 | 北京房江湖科技有限公司 | File uploading method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114650287A (en) | 2022-06-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6749926B2 (en) | Method, device and system for synchronizing data | |
| KR100625595B1 (en) | Parallel Logging Method of Transaction Processing System | |
| CN112445626B (en) | Data processing method and device based on message middleware | |
| US10713378B2 (en) | Techniques for eventually consistent inheritance propagation for nested content item namespaces | |
| CN111225012A (en) | Transaction processing method, device and equipment | |
| US20120224482A1 (en) | Credit feedback system for parallel data flow control | |
| US11507277B2 (en) | Key value store using progress verification | |
| CN111404755B (en) | Network configuration method, device and storage medium | |
| US10838931B1 (en) | Use of stream-oriented log data structure for full-text search oriented inverted index metadata | |
| CN112115105A (en) | Service processing method, device and equipment | |
| CN113127564B (en) | Parameter synchronization method and device | |
| CN112114983B (en) | Communication method, device and equipment based on shared memory | |
| CN114650287B (en) | Breakpoint continuous method, device and system for data file | |
| CN111367948B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| EP3082050A1 (en) | Mass data fusion storage method and system | |
| CN116186082A (en) | Data summarizing method based on distribution, first server and electronic equipment | |
| CN108121580B (en) | Method and device for realizing application program notification service | |
| CN114553521A (en) | Remote memory access method, device, equipment and medium | |
| CN114900531B (en) | Data synchronization method, device and system | |
| CN111367694A (en) | Event processing method, server and computer storage medium | |
| CN113760860B (en) | Data reading method and device | |
| CN112860746B (en) | Cache reduction-based method, equipment and system | |
| US20230044681A1 (en) | Udsf record retrieval and deletion | |
| US20220222213A1 (en) | Correction, synchronization, and migration of databases | |
| CN111971667B (en) | Recoverable merge ordering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |