CN114461573A

CN114461573A - File migration method and device, storage medium and electronic equipment

Info

Publication number: CN114461573A
Application number: CN202210117411.3A
Authority: CN
Inventors: 陈露; 刘卓然
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2022-02-08
Filing date: 2022-02-08
Publication date: 2022-05-10

Abstract

The embodiment of the invention discloses a file migration method, a file migration device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a file block of a file to be migrated, and controlling a file block transmission thread to enter a multi-level scheduling queue, wherein the multi-level scheduling queue comprises at least two scheduling queues with different priority levels; determining the response ratio of the download factors of the file blocks, and adjusting the waiting position of the file block transmission thread in the multistage scheduling queue according to the response ratio of the download factors or the transmission execution time length of the file blocks; and determining a currently executed file block transmission thread according to the file block transmission threads in the multistage scheduling queue, and executing the file block transmission thread to transmit the corresponding file block to a receiving party. The migration duration of the whole file is shortened, the I/O utilization rate of the server is guaranteed, the total throughput and the thread switching overhead are balanced, and the maximization of the thread efficiency is guaranteed.

Description

File migration method and device, storage medium and electronic equipment

Technical Field

The embodiment of the invention relates to the technical field of data transmission, in particular to a file migration method, a file migration device, a storage medium and electronic equipment.

Background

With the rapid development of information technology and network technology and the coming of the digital era, the information management level of each enterprise and public institution is continuously improved, upgrading and transforming the system becomes common operation, and in the process of upgrading the system, file migration is required.

However, the existing file migration mode has the situations of low file transmission efficiency and low network resource utilization rate.

Disclosure of Invention

The embodiment of the invention provides a file migration method, a file migration device, a storage medium and electronic equipment, and aims to improve the migration efficiency of files.

In a first aspect, an embodiment of the present invention provides a file migration method, including:

acquiring a file block of a file to be migrated, and controlling a file block transmission thread to enter a multi-level scheduling queue, wherein the multi-level scheduling queue comprises at least two scheduling queues with different priority levels;

determining the response ratio of the download factors of the file blocks, and adjusting the waiting position of the file block transmission thread in the multistage scheduling queue according to the response ratio of the download factors or the transmission execution time length of the file blocks;

and determining a currently executed file block transmission thread according to the file block transmission thread in the multistage scheduling queue, and executing the file block transmission thread to transmit the corresponding file block to a receiving party.

In a second aspect, an embodiment of the present invention further provides a file migration apparatus, including:

the file block acquisition module is used for acquiring a file block of a file to be migrated;

the first thread scheduling module is used for controlling file block transmission threads to enter a multi-level scheduling queue, wherein the multi-level scheduling queue comprises at least two scheduling queues with different priority levels;

the second thread scheduling module is used for determining the download factor response ratio of each file block and adjusting the waiting position of the file block transmission thread in the multistage scheduling queue according to the download factor response ratio or the transmission execution time length of the file block;

and the file block transmission module is used for determining the currently executed file block transmission thread according to the file block transmission threads in the multistage scheduling queue, and executing the file block transmission thread so as to transmit the corresponding file block to a receiving party.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:

one or more processors;

a storage device to store one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the file migration method according to any one of the embodiments of the present disclosure.

In a fourth aspect, the embodiments of the present disclosure also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used for executing the file migration method according to any one of the embodiments of the present disclosure.

According to the technical scheme of the embodiment, the waiting positions of the file block transmission threads in the multi-stage scheduling queue are adjusted according to the download factor response ratio of the file blocks, so that a plurality of file blocks in the same file can be quickly responded in a short time, the migration time of the whole file is shortened, and the I/O utilization rate of the server is ensured. Meanwhile, in the transmission process of the file blocks, the waiting positions of the file block transmission threads in the multi-stage scheduling queue are adjusted through comparison of the transmission duration and the execution time slices, so that the total throughput and the thread switching overhead are balanced, and the maximization of the thread efficiency is ensured.

Drawings

Fig. 1 is a schematic flowchart of a file migration method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an application scenario provided by an embodiment of the present invention;

FIG. 3 is a flowchart illustrating a file migration method according to an embodiment of the present invention

FIG. 4 is a schematic structural diagram of a file migration apparatus according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device in an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1 is a schematic flow diagram of a file migration method according to an embodiment of the present invention, where this embodiment is applicable to a case of migrating a file block from a first system to a second system, and the method may be executed by a file migration apparatus according to an embodiment of the present invention, where the file migration apparatus may be implemented by software and/or hardware, and the file migration apparatus may be configured on an electronic computing device, and specifically includes the following steps:

s110, obtaining a file block of a file to be migrated, and controlling a file block transmission thread to enter a multi-level scheduling queue, wherein the multi-level scheduling queue comprises at least two scheduling queues with different priority levels.

S120, determining the download factor response ratio of each file block, and adjusting the waiting position of the file block transmission thread in the multistage scheduling queue according to the download factor response ratio or the transmission execution time length of the file block.

S130, determining a currently executed file block transmission thread according to the file block transmission threads in the multistage scheduling queue, and executing the file block transmission thread to transmit the corresponding file block to a receiving party.

In this embodiment, the file is migrated from the first system to the second system, without limiting the system types of the first system and the second system, and optionally, the first system may be an old system, and the second system may be a new system. In some embodiments, the first system and the second system may be configured in different hardware devices, for example, a terminal device such as a mobile phone, a computer device, a server, or the like, and the hardware device is configured with a storage component to implement storage of a file. In some embodiments, the first system and the second system may be configured in the same hardware device, and accordingly, the first system and the second system are respectively configured with a storage area, and the file migration from the first system to the second system is also a storage area migration from the storage area of the first system to the storage area of the second system.

The file migration apparatus for executing the file migration method provided in this embodiment may be configured in the hardware device where the first system is located, or may be configured in other electronic devices independent of the electronic devices where the first system and the second system are located, which is not limited herein. For example, referring to fig. 2, fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present invention, and the configuration file migration apparatus in the server in fig. 2 is configured to execute a file migration method provided in an embodiment to migrate a file in a first system (e.g., a system a) to a second system (e.g., a system B).

In this embodiment, a temporary storage is created to store a file acquired from the first system, where the file is a file to be migrated, and the maximum storage capacity of the temporary storage area is the sum of data volumes of the file to be migrated. And judging whether the data volume of each file is greater than a file threshold value or not for the files in the temporary storage area, and if so, blocking the files to obtain a plurality of file blocks. Specifically, the file may be partitioned based on a preset partition size, where each partition size may be set fixedly or may be adjusted according to a network condition. Different network conditions can correspond to different block sizes, the block sizes can be positively correlated with the network conditions, and the block sizes are larger as the network conditions are better. Optionally, the network condition may be determined according to a network speed, and different network conditions respectively correspond to network speeds in different ranges. Optionally, the network condition may be determined according to a network type, which may include, but is not limited to, WiFi, mobile network, and the like, for example. Optionally, the size of the partition may be determined according to the current network condition, or may be determined according to an average network condition of a preset time period, so as to ensure stability of the network state.

When the data amount of the file is smaller than the file threshold, the file is not subjected to the blocking process, and the file is transmitted as a whole as one file block.

And respectively setting file block identifications, namely block identifications, for each file block obtained by block processing. It should be noted that each file is respectively configured with a file identifier, a file block has an association relationship with the file to which the file belongs, and for example, the block identifier may include a file identifier and a block identifier of the file to which the file belongs. Illustratively, the file identifier may be a file name, and the block identifier may be a block number, such as 1, 2, 3, ABC, or the like, which may be determined according to the location of the file block in the file.

Calling a thread for each file block, wherein the thread pool illustratively comprises a plurality of threads, and an idle thread is respectively allocated to the newly added file blocks to execute the transmission of the file blocks, wherein the file block transmission thread transmits the file blocks in the temporary storage area to a receiver, namely a storage area of the second system.

In this embodiment, the scheduling of multiple file block transmission threads is implemented based on a multi-level scheduling queue. The multi-stage scheduling queues comprise at least two scheduling queues, wherein the scheduling priorities of different scheduling queues are different. Illustratively, the multi-level scheduling queue includes n scheduling queues Q1 to Qn, wherein the scheduling priorities of Q1 to Qn decrease in sequence. Further, the execution time slices of the scheduling queues in the multi-level scheduling queue are different, the execution time slices of the scheduling queues are inversely related to the priority, the lower the priority of the scheduling queue is, the longer the execution time slice is, and in some embodiments, the execution time slices of the scheduling queues are increased in multiples along with the reduction of the priority. For example, for Q1, Q2 … Qn, the scheduling priorities of Q1 to Qn decrease in sequence, the execution time slice of Q1 may be t0, the execution time slice of Q2 may be 2t0, and the execution time slice of Qn may be nt 0.

The file block transmission threads corresponding to the file blocks are scheduled through a multi-level scheduling queue, illustratively, each file block transmission thread can enter the multi-level scheduling queue for waiting, the executed file block transmission thread is determined according to the waiting position of each file block transmission thread in the multi-level scheduling queue, and other file block transmission threads are in a waiting state, so that the file block transmission threads are scheduled.

Optionally, controlling the file block transmission thread to enter the multistage scheduling queue includes: and controlling the file block transmission thread to enter a highest priority scheduling queue. The file block transmission thread is a newly added file block transmission thread, and the newly added file block transmission thread enters a highest priority scheduling queue.

In this embodiment, the multiple file block transmission threads in the multi-level scheduling queue perform real-time adjustment, and determine the file block transmission thread to be scheduled according to the waiting position of each file block transmission thread in the current multi-level scheduling queue. And adjusting the transmission thread of each file block in real time based on the response ratio of the download factor or the transmission execution time length of each file block.

The download factor response ratio of the file block is used to characterize the downloaded rate of the file in which the file block is located, where downloading is the successful transmission of the file to the recipient. Optionally, determining a download factor response ratio of each file block includes: for any file block, determining the file download ratio of the file to which the file block belongs; and determining the response ratio of the download factors of the file blocks based on the waiting time length, the expected execution time length and the file download ratio of the file blocks.

The file download ratio is determined based on a ratio of the number of the file blocks that have been completely transmitted in the file to which the file blocks belong to the total number of the file blocks, and may be, for example, the ratio of the number of the file blocks that have been completely transmitted in the file to which the file blocks belong to the total number of the file blocks in the file to which the file blocks belong. The total number of file blocks in the file to which the file block belongs can be determined according to the blocking processing result, the number of the file blocks which are transmitted in the file to which the file block belongs is obtained, and the current file download ratio of the file block can be determined.

On the basis of the above embodiment, the method further includes: setting a file transmission list, wherein the file transmission list comprises file block identifiers, file identifiers to which the file blocks belong and migration state identifiers; updating the migration state identifier in the file transfer list under the condition that any file block is transferred; and determining the total number of file blocks in the file to which the file block belongs according to the file block identifier and the file identifier to which the file block belongs in the file transmission list, and determining the number of the file blocks which are transmitted in the file to which the file block belongs based on the migration state identifier.

The file transmission list is created and used for recording the identification information and the migration state identification of each file block, so that the number of the file blocks of each file block which are completely transmitted is counted conveniently, the file download ratio of each file block to be migrated is determined, and the download factor response ratio of the file blocks is further determined. For example, the total number of file blocks of a file corresponding to the file identifier is determined based on the number of file block identifiers corresponding to the same file identifier, and the number of file blocks of which the migration status identifier is "downloaded" in a plurality of file blocks corresponding to the file identifier is determined as the number of file blocks that have completed transmission.

The determining that any file block completes migration may be by detecting a remaining data amount of the file block in the transmission process, and determining that the file block completes migration when the remaining data amount is 0. The file migration completion method can also be a transmission completion signal transmitted by the receiving party, and when the transmission completion signal transmitted by the receiving party is received, the file block migration completion corresponding to the file block identification in the transmission completion signal is determined. For a file block which does not start migration, the migration status is marked as 'not migrated', for a file block which starts migration but does not finish migration, the migration status is marked as 'not migrated', and after the file block is determined to finish migration, the migration status is marked as 'downloaded'. The transition state identifier may also be differentiated in the form of "0", "1" or "2", for example, 0 is used to indicate that the transition is not performed, 1 is used to indicate that the transition is performed, and 2 is used to indicate that the download is performed.

In some alternative embodiments, the download factor response ratio may be calculated by the following formula:

wherein, R is the response ratio of the download factors of the file blocks, W is the waiting time of the file block transmission thread in the multistage scheduling queue, T is the predicted execution time determined according to the data volume of the file, M is the total number of the file blocks of the file to which the file blocks belong, and k is the number of the file blocks which are completely transmitted in the file to which the file blocks belong.

According to the download factor response ratio determined in the above manner, when the number of file blocks which have finished transmission in the file to which the file block belongs increases, the position of the corresponding file block transmission thread in the multi-level scheduling queue is adjusted through the download factor response ratio, so that the migration priority of the file block transmission thread is adjusted in the multi-level scheduling queue.

Optionally, the file transfer list is read according to a predetermined time interval to update the download factor response ratio of each file block, where the predetermined time interval may be 1s, or 10s, and the like, which is not limited herein. Optionally, when the file transmission list is updated, the download factor response ratio of each non-migrated file block corresponding to the file identifier is updated according to the file block identifier to be updated and the corresponding file identifier.

In some embodiments, the waiting position of the file block transmission thread in the multi-stage scheduling queue may be adjusted based on the response ratio of the download factor or the transmission execution time length, wherein the file block transmission thread in the high-priority scheduling queue in the multi-stage scheduling queue is called preferentially, and when the high-priority scheduling queue is empty, that is, there is no waiting file block transmission thread, the file block transmission thread to be executed is determined from the next-priority scheduling queue.

For each file block transfer thread, a multi-level scheduling queue is entered, i.e., the scheduling queue initially entering the highest priority, which may be, for example, scheduling queue Q1, and priority adjustment is performed in the multi-level scheduling queue according to the execution status. Optionally, adjusting a waiting position of the file block transmission thread in the multistage scheduling queue according to the transmission execution duration of the file block includes: and for the file block transmission thread in any priority scheduling queue, under the condition that the execution duration is longer than the preset execution time slice of the scheduling queue, adjusting the file block transmission thread to the next priority scheduling queue.

In any priority scheduling queue, all file block transmission threads are sorted based on the download factor response ratio of the file blocks, and the waiting position in the current scheduling queue is determined according to the download factor response ratio of the newly-added file block transmission threads. It should be noted that, when the download factor response ratios of the multiple file block transmission threads are the same, random ordering may be performed. Recording the transmission duration of a file block under the condition that any one of the adjusted file block transmission threads performs file block transmission, scheduling a next file block transmission thread if the file block completes transmission in the preset execution time slice of the scheduling queue, adjusting the file block transmission thread to a scheduling queue of a next priority for waiting if the file block does not complete transmission in the preset execution time slice of the scheduling queue, namely network resource utilization rate is poor, for example, the next priority scheduling queue of the scheduling queue Q1 is a scheduling queue Q2, the next priority scheduling queue of the scheduling queue Q2 is a scheduling queue Q3, and so on. The file block transmission thread is scheduled preferentially by dynamically adjusting the file block transmission thread in the multistage scheduling module according to the execution duration of the file block, so that the utilization rate of network resources is improved.

In the scheduling process of the file block transmission thread, namely under the condition that the file block is migrated, the response ratio of the download factors of other file blocks is changed, namely the file blocks which are migrated and the other file blocks belong to the same file. And adjusting the waiting position of the corresponding file block transmission thread in the multistage scheduling queue based on the updated download factor response ratio so as to realize the dynamic adjustment of the file block transmission thread.

Optionally, adjusting the waiting position of the file block transmission thread in the multistage scheduling queue according to the download factor response ratio includes: and for the file block transmission threads in the secondary priority scheduling queue, under the condition that the response ratio of the current download factor is greater than the response ratio threshold of any high priority scheduling queue, adjusting the file block transmission threads into the high priority scheduling queue, wherein in any priority scheduling queue, the file block transmission threads are sequenced based on the response ratio of the download factor of the file block.

Each scheduling queue is provided with a response ratio threshold, the response ratio threshold is positively correlated with the priority of the scheduling queue, for example, the response ratio threshold of the scheduling queue Q1 is R1, the response ratio threshold of the scheduling queue Q2 is R2, and so on. For a secondary priority scheduling queue, for example, any one of the scheduling queues Q2-Qn, when the download factor response ratio of a file block corresponding to any file block transmission thread is updated, the updated download factor response ratio is compared with the response ratio threshold of a high priority scheduling queue, where the high priority scheduling queue may be any scheduling queue with a priority higher than that of the current scheduling queue, for example, the download factor response ratio of a file block corresponding to a file block transmission thread in the scheduling queue Q3 is updated, and the updated download factor response ratio scheduling queue Q2 and the response ratio threshold of the scheduling queue Q1 are compared respectively. And under the condition that the response ratio of the current download factor is greater than the response ratio threshold of any high-priority scheduling queue, adjusting the file block transmission thread into the high-priority scheduling queue. Illustratively, if the response ratio of the download factor of the file block corresponding to the file block transmission thread in the scheduling queue Q3 is greater than the response ratio threshold of the scheduling queue Q1, the file block transmission thread is adjusted to the scheduling queue Q1; and if the response ratio of the download factor of the file block corresponding to the file block transmission thread in the scheduling queue Q3 is greater than the response ratio threshold of the scheduling queue Q2 and less than the response ratio threshold of the scheduling queue Q1, adjusting the file block transmission thread to the scheduling queue Q2. And determining the waiting position in the adjusted scheduling queue according to the updated download factor response ratio.

In some embodiments, where the download factor response ratio of a file block is updated and is not greater than the response ratio threshold of any high priority scheduling queue, the wait position in the current scheduling queue may be adjusted according to the updated download factor response ratio.

Under the condition that the migration of the file blocks in the same file is completed, the response ratio of the download factors of other file blocks in the file is increased, accordingly, the scheduling priority in the multilevel scheduling queue is improved, the file blocks of the same file can respond quickly in the high-priority scheduling queue, and the same file can be migrated quickly in the effective time.

Optionally, when all file blocks in the same file are migrated, a file reassembly prompt signal is sent to the receiving party, so that the receiving party reassembles the received multiple file blocks into a complete file. Specifically, whether the file blocks in the same file are migrated is determined according to the file identifiers, the file block identifiers and the migration status identifiers in the file transmission list, for example, it is determined that all the file blocks in the file are migrated under the condition that a plurality of file block identifiers corresponding to the same file identifier are all configured with the "downloaded" migration status identifier. The receiver can reassemble the file blocks according to the file block identifiers, wherein the file block identifiers can represent the block sequence of the file blocks, namely the reassembly sequence.

When the migration of the previous file block transmission thread is completed or the migration is started, in the multi-level scheduling queue, the currently executed file block transmission thread is determined according to the waiting position of the file block transmission thread at the current time, and specifically, the currently executed file block transmission thread is determined according to the priority of the waiting position of the file block transmission thread at the current time. Optionally, determining a currently executed file block transmission thread according to the file block transmission thread in the multi-stage scheduling queue includes: and determining the file block transmission thread with the largest download factor response ratio in the non-empty highest priority scheduling queue as the currently executed file block transmission thread.

Illustratively, in the case that the highest priority scheduling queue Q1 is not empty, the file block transfer thread with the largest download factor response ratio in the scheduling queue Q1 is determined as the currently executed file block transfer thread, in the case that the highest priority scheduling queue Q1 is empty, the next priority scheduling queue Q2 is the non-empty highest priority scheduling queue, the file block transfer thread with the largest download factor response ratio in the scheduling queue Q2 is determined as the currently executed file block transfer thread, and so on.

On the basis of the above embodiment, under the condition that the high-priority scheduling queue is empty and the file block transmission thread in the secondary-priority scheduling queue is scheduled, in the execution process of any file block transmission thread in the secondary-priority scheduling queue, it is detected that a new file block transmission thread arrives in the high-priority scheduling queue, the file block transmission thread in the secondary-priority scheduling queue is suspended, and the new file block transmission thread in the high-priority scheduling queue is determined as the currently executed file block transmission thread. The new file block transmission line can be a newly added file block transmission thread. And (4) putting the file block transmission thread which is executed in a pause way back to the tail of the current scheduling queue, recording the transmission time length, and only allocating the unfinished time slices when the file block transmission thread which is executed in the pause way is called again.

And circularly executing the scheduling process until all the multi-stage scheduling queues are empty, indicating that the file migration is completed. Exemplarily, referring to fig. 3, fig. 3 is a schematic flowchart of file migration according to an embodiment of the present invention. The server in fig. 3 creates a storage area for the temporary files downloaded from system a, the storage area being at most the sum of the files to be migrated. And creating a file migration table, wherein fields in the migration table comprise a file identification ID, a file size, a file name, a successful downloading identification, a block sequence number and operation time, wherein the file identification ID, the file size and the file name are basic attributes of the original file, and the successful downloading identification (namely the migration state identification) describes whether the file is successfully downloaded from the archive server to the temporary storage area. Setting a file blocking threshold according to the network condition and the file size, if the size of the file to be downloaded is larger than the file threshold, dividing the file to be downloaded into file blocks according to the threshold size, adding a file blocking mark, and recording the serial number of the blocked file.

Putting a file block to be downloaded into a thread pool, and realizing thread scheduling based on a multistage feedback scheduling algorithm of a download factor response ratio, wherein the specific scheduling mode is as follows: determining the corresponding ratio of download factors of file blocks, (2) when a file block process enters a queue to be scheduled and waits, firstly entering Q1 with the highest priority for waiting, wherein the time slice is t0, if the execution of the t0 time slice in Q1 is not completed, the file block process enters Q2 with the next highest priority for waiting, the time slice is 2 × t0, and so on, the following queues are formed, and the priority is from high to low:

queue 1: q1, wherein the time slice is t0

Queue 2: q2, wherein the time slice is 2t0

Queue 3: q3, wherein the time slice is 3t0

......

And a queue N: QN, where the time slice is Nt0

(4) Firstly, scheduling a process in a queue with a high priority, and if the queue with the high priority has no schedulable process, scheduling a next-highest priority process, such as: the scheduling order of Q1, Q2, Q3 is from Q1 to Q3. For each process in the same queue, scheduling according to the download factor response ratio arrangement, and adding each process in the sub-optimal priority queue Q2-QN to the queue of the high priority Q1 again when the download factor response ratio R is higher than the response ratio threshold R1 (namely R > R1); when the process of the low priority queue runs and a new process of the high priority arrives, the running process needs to be immediately put back to the tail of the current queue. And then processes the high priority process. When the process is run again, only the last outstanding time slice is allocated.

And executing the processes until all the processes are executed, when the queue in the thread pool is empty, indicating that the files to be downloaded are downloaded completely, and if the queue is not empty, indicating that the files or the file blocks are still to be downloaded. 6. In the downloading process, whether the files or file blocks currently downloaded are abnormally downloaded is continuously detected, for the files or file blocks which are abnormally downloaded, current data downloading is suspended, breakpoint continuous transmission operation is carried out on the file blocks which are failed to download, the file blocks are released to a thread pool to be downloaded, and migration of the file blocks is executed again. 7. The file that is completed by a single download is checked for a chunking flag. And updating the file downloading success mark for the single file and continuing the next file downloading. 8. The block file detects whether the large file completes downloading of all file blocks, if so, the large file recombination is implemented, and the file downloading success mark is updated. And continuing to download and transmit the next file. 9. And detecting a file migration table, judging whether the downloaded file data exists according to the downloading success mark, adding the downloaded file to a file uploading thread pool according to the multi-thread transmission algorithm of the partitioned file, and sequentially uploading the successfully downloaded file in the temporary storage area to a target file system, namely a system B in the figure 3.

The multi-level feedback scheduling algorithm based on the download factor response ratio can better meet the CPU running time of all processes, different file blocks of the same file can be quickly responded within a short time difference, the file download uploading time interval is shortened, the I/O utilization rate of a server is guaranteed, and the file retransmission expense is reduced through the file breakpoint continuous transmission function.

On the basis of the above embodiment, the method further includes: and detecting whether the currently executed file block transmission thread is abnormally transmitted or not, if so, suspending the currently executed file block transmission thread, and carrying out reproduction scheduling on the file block transmission thread with failed transmission so as to carry out breakpoint continuous transmission on the file block with failed transmission.

In the process of scheduling each file block transmission thread, whether abnormal transmission exists is judged, for example, whether the transmitted data volume is the same as the data volume received by the receiver is compared, and the file block transmission thread is indicated to have abnormal transmission under the condition that the transmitted data volume is different from the data volume received by the receiver. For example, it may also be determined that a transmission flag bit of the receiving party meets a preset condition, where the transmission flag bit is set to a preset flag when the receiving party receives the file block, and when the transmission flag bit does not meet the preset flag, it indicates that the file block transmission thread has abnormal transmission.

For the file block transmission thread with abnormal transmission, the transmission is suspended and the transmission sites are recorded, the suspended file block transmission thread is used as a newly added file block transmission thread to reenter the thread pool, namely, the thread pool reenters the multistage scheduling queue, and under the condition of rescheduling, breakpoint continuous transmission is carried out based on the recorded transmission sites, so that invalid retransmission of the file blocks is avoided, and the overhead of file retransmission is reduced.

After the transmission thread of any file block finishes the migration of the file block, the thread is released and used as an idle thread to perform a thread pool, so that the thread pool can be called again conveniently.

According to the technical scheme of the embodiment, the waiting positions of the file block transmission threads in the multi-stage scheduling queue are adjusted according to the download factor response ratio of the file blocks, so that a plurality of file blocks in the same file can be quickly responded in a short time, the migration time of the whole file is shortened, and the I/O utilization rate of the server is ensured. Meanwhile, in the transmission process of the file blocks, the waiting positions of the file block transmission threads in the multistage scheduling queue are adjusted through comparison of the transmission duration and the execution time slices, so that the total throughput and the thread switching overhead are balanced, and the maximization of the thread efficiency is ensured.

Fig. 4 is a schematic structural diagram of a file migration apparatus provided in an embodiment of the present invention, where the apparatus includes:

a file block obtaining module 210, configured to obtain a file block of a file to be migrated;

the first thread scheduling module 220 is configured to control a file block transmission thread to enter a multi-level scheduling queue, where the multi-level scheduling queue includes at least two scheduling queues with different priority levels;

a second thread scheduling module 230, configured to determine a download factor response ratio of each file block, and adjust a waiting position of the file block transmission thread in the multi-stage scheduling queue according to the download factor response ratio or a transmission execution duration of the file block;

and the file block transmission module 240 is configured to determine a currently executed file block transmission thread according to the file block transmission threads in the multistage scheduling queue, and execute the file block transmission thread to transmit the corresponding file block to a receiving party.

Optionally, the second thread scheduling module 230 includes:

the file download ratio determining unit is used for determining the file download ratio of the file to which the file block belongs for any file block;

and the download factor response ratio determining unit is used for determining the download factor response ratio of the file block based on the waiting time length, the predicted execution time length and the file download ratio of the file block.

Optionally, the file download ratio is determined based on a ratio of the number of file blocks that have been transmitted in the file to which the file block belongs to the total number of file blocks;

the device also includes:

the file transmission list setting module is used for setting a file transmission list, and the file transmission list comprises file block identifiers, file identifiers to which the file blocks belong and migration state identifiers;

the file transfer list updating module is used for updating the migration state identifier in the file transfer list under the condition that any file block is transferred;

and the data acquisition module is used for determining the total number of file blocks in the file to which the file blocks belong according to the file block identifiers in the file transmission list and the file identifiers to which the file blocks belong, and determining the number of the file blocks which are transmitted in the file to which the file blocks belong based on the migration state identifiers.

Optionally, the first thread scheduling module 220 is configured to:

and controlling the file block transmission thread to enter a highest priority scheduling queue.

Optionally, the second thread scheduling module 230 includes:

the first scheduling unit is used for adjusting the file block transmission thread in any priority scheduling queue to the next priority scheduling queue under the condition that the execution duration is longer than the preset execution time slice of the scheduling queue;

alternatively, the first and second electrodes may be,

and the second scheduling unit is used for adjusting the file block transmission threads in the secondary priority scheduling queue to the high priority scheduling queue under the condition that the current download factor response ratio is greater than the response ratio threshold of any high priority scheduling queue, wherein in any priority scheduling queue, all the file block transmission threads are sequenced based on the download factor response ratio of the file blocks.

Optionally, the file block transmission module 240 is configured to:

determining the file block transmission thread with the largest download factor response ratio in the non-empty highest priority scheduling queue as the currently executed file block transmission thread;

alternatively, the first and second electrodes may be,

in the execution process of any file block transmission thread in the sub-priority scheduling queue, detecting that a new file block transmission thread arrives in the high-priority scheduling queue, pausing the file block transmission thread in the sub-priority scheduling queue, and determining the new file block transmission thread in the high-priority scheduling queue as the currently executed file block transmission thread.

Optionally, the apparatus further comprises:

the abnormal transmission detection module is used for detecting whether the currently executed file block transmission thread is abnormally transmitted or not;

and the breakpoint continuous transmission module is used for pausing the currently executed file block transmission thread if the currently executed file block transmission thread is abnormal transmission, and carrying out reproduction scheduling on the file block transmission thread with failed transmission so as to carry out breakpoint continuous transmission on the file block with failed transmission.

The file migration device provided by the embodiment of the invention can execute the file migration method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Referring now to fig. 5, a schematic diagram of an electronic device (e.g., the terminal device or the server of fig. 5) 400 suitable for implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 5, electronic device 400 may include a processing device (e.g., central processing unit, graphics processor, etc.) 401 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage device 408 into a Random Access Memory (RAM) 403. In the RAM403, various programs and data necessary for the operation of the electronic apparatus 400 are also stored. The processing device 401, the ROM402, and the RAM403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

Generally, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, tape, hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate wirelessly or by wire with other devices to exchange data. While fig. 5 illustrates an electronic device 400 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 409, or from the storage device 408, or from the ROM 402. The computer program performs the above-described functions defined in the methods of the embodiments of the present disclosure when executed by the processing device 401.

The electronic device provided by the embodiment of the disclosure and the file migration method provided by the embodiment belong to the same inventive concept, and technical details that are not described in detail in the embodiment can be referred to the embodiment, and the embodiment have the same beneficial effects.

The embodiment of the present disclosure provides a computer storage medium, on which a computer program is stored, which when executed by a processor implements the file migration method provided by the above-described embodiment.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit/module does not in some cases constitute a limitation of the unit itself.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for file migration, comprising:

2. The method of claim 1, wherein determining a download factor response ratio for each file block comprises:

for any file block, determining the file download ratio of the file to which the file block belongs;

and determining the response ratio of the download factor of the file block based on the waiting time length, the expected execution time length and the file download ratio of the file block.

3. The method according to claim 2, wherein the file download ratio is determined based on a ratio of the number of file blocks of the file to which the file blocks belong, which have been completely transmitted, to the total number of file blocks;

the method further comprises the following steps:

setting a file transmission list, wherein the file transmission list comprises file block identifiers, file identifiers to which the file blocks belong and migration state identifiers;

updating the migration state identifier in the file transfer list under the condition that any file block is transferred;

and determining the total number of file blocks in the file to which the file block belongs according to the file block identifier and the file identifier to which the file block belongs in the file transmission list, and determining the number of the file blocks which are transmitted in the file to which the file block belongs based on the migration state identifier.

4. The method of claim 1, wherein controlling the file block transfer thread to enter a multi-level scheduling queue comprises:

5. The method of claim 1, wherein adjusting the waiting position of the file block transmission thread in the multi-stage scheduling queue according to the download factor response ratio or the transmission execution time of the file block comprises:

for a file block transmission thread in any priority scheduling queue, under the condition that the execution duration is longer than the preset execution time slice of the scheduling queue, adjusting the file block transmission thread to the next priority scheduling queue;

alternatively, the first and second electrodes may be,

and for the file block transmission threads in the secondary priority scheduling queue, under the condition that the response ratio of the current download factor is greater than the response ratio threshold of any high priority scheduling queue, adjusting the file block transmission threads into the high priority scheduling queue, wherein in any priority scheduling queue, the file block transmission threads are sequenced based on the response ratio of the download factor of the file block.

6. The method according to claim 1, wherein the determining a currently executed file block transfer thread according to the file block transfer threads in the multi-stage scheduling queue comprises:

alternatively, the first and second electrodes may be,

7. The method of claim 1, further comprising:

and detecting whether the currently executed file block transmission thread is abnormally transmitted or not, if so, suspending the currently executed file block transmission thread, and carrying out reproduction scheduling on the file block transmission thread with failed transmission so as to carry out breakpoint continuous transmission on the file block with failed transmission.

8. A file migration apparatus, comprising:

the file block acquisition module is used for acquiring file blocks of the file to be migrated;

9. An electronic device, characterized in that the electronic device comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the file migration method of any of claims 1-7.

10. A storage medium containing computer-executable instructions for performing the file migration method of any one of claims 1-7 when executed by a computer processor.