CN111522792A - File migration method and device - Google Patents
File migration method and device Download PDFInfo
- Publication number
- CN111522792A CN111522792A CN202010311007.0A CN202010311007A CN111522792A CN 111522792 A CN111522792 A CN 111522792A CN 202010311007 A CN202010311007 A CN 202010311007A CN 111522792 A CN111522792 A CN 111522792A
- Authority
- CN
- China
- Prior art keywords
- file
- migration
- migrated
- subtask
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 249
- 238000013508 migration Methods 0.000 title claims abstract description 248
- 238000000034 method Methods 0.000 title claims abstract description 78
- 230000008569 process Effects 0.000 claims abstract description 40
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004806 packaging method and process Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 24
- 238000010586 diagram Methods 0.000 description 12
- 230000002159 abnormal effect Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 238000012856 packing Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000007420 reactivation Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/185—Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/03—Credit; Loans; Processing thereof
Abstract
The invention discloses a file migration method and a device, wherein the method comprises the following steps: reading a configuration file, wherein the configuration file comprises: a migration process of the file to be migrated; creating a plurality of migration subtasks corresponding to the files to be migrated according to the migration flow; according to a plurality of migration subtasks corresponding to a file to be migrated, splitting a thread pool into a plurality of sub-thread pools, so that each sub-thread pool corresponds to one migration subtask; and executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask. The invention can realize the file migration flow which can be customized and configured, create a plurality of migration subtasks according to the file migration flow, and allocate a separate sub thread pool for each migration subtask, so that each migration subtask of the file migration can be executed in parallel, thereby not only improving the file migration efficiency, but also being suitable for the file migration of different hardware platforms and having stronger compatibility.
Description
Technical Field
The present invention relates to the field of data backup, and in particular, to a file migration method and apparatus.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
After a new software system is developed or an old software system is upgraded, it is often necessary to migrate the data of one software system to another. The migrated data includes not only data stored in the relational database but also various files stored in the file system.
For example, when a bank loan transaction is initiated, due to the need to verify the relevant certification materials of the loan client, a client manager is often required to upload the certification documents of the loan client, such as certificates and real estate certificates, to the credit management system after photocopying, so that the data uploaded to the credit management system can be viewed in the subsequent loan approval process or after the loan transaction is completed. The partial data includes: loan documents and loan data, where loan documents refer to actual documents, such as a loan client's identification card copy or income-proving document, and the like, typically stored in a file system; loan data refers to the data of the relationship between the loan documents and the stream of loans, usually stored in a relational database. Because the loan documents need to occupy a large storage space, the document storage architecture adopted by the existing bank loan approval system is as follows: each branch system stores respective loan documents; the head office system stores loan data of each branch in a centralized manner.
For data stored in an old system database, various database technical means can be adopted to migrate the data to a new system; for the files stored in the file system, such as the loan files, the file formats are various and the contents are numerous, and the files are often stored in the file systems of different hardware platforms (for example, in each branch system storing the loan files), and for the file systems of different hardware platforms, migration programs of different versions are often required to be developed, which results in high program operation and maintenance costs. In addition, the conventional file migration method treats the whole file migration process as a migration task, because the file migration process involves many flow steps, the serial treatment mode can cause many resources to be idle, the whole file migration process consumes long time, if an error occurs in the whole file migration process, the reason of the error is difficult to be checked, and once the error occurs, the whole file migration process needs to be executed again.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a file migration method, which is used for solving the technical problems that in the prior art, the time consumption is long, the problems are difficult to be solved and different versions of programs need to be issued for different branch service systems when a database technology is adopted to migrate files in each branch service system of a bank, and the method comprises the following steps: reading a configuration file, wherein the configuration file comprises: a migration process of the file to be migrated; creating a plurality of migration subtasks corresponding to the files to be migrated according to the migration flow; according to a plurality of migration subtasks corresponding to a file to be migrated, splitting a thread pool into a plurality of sub-thread pools, so that each sub-thread pool corresponds to one migration subtask; and executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
The embodiment of the invention also provides a file migration device, which is used for solving the technical problems that in the prior art, the time consumption is long, the problems are difficult to be solved by adopting a database technology to migrate files in each branch business system of a bank, and different versions of programs need to be issued for different branch business systems, and comprises the following steps: the configuration module is used for reading a configuration file, wherein the configuration file comprises: a migration process of the file to be migrated; the task creating module is used for generating each migration subtask corresponding to the file to be migrated according to the migration flow; the thread allocation module is used for splitting the thread pool into a plurality of sub-thread pools according to a plurality of migration subtasks corresponding to the file to be migrated, so that each sub-thread pool corresponds to one migration subtask; and the task execution module is used for executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
The embodiment of the invention also provides computer equipment for solving the technical problems that in the prior art, the database technology is adopted to migrate files in each branch business system of a bank, the time consumption is long, the problems are difficult to be solved, and different versions of programs need to be issued for different branch business systems.
The embodiment of the invention also provides a computer readable storage medium, which is used for solving the technical problems that in the prior art, the database technology is adopted to migrate files in each branch business system of a bank, the time consumption is long, the problems are difficult to be solved, and different versions of programs need to be issued for different branch business systems.
In the embodiment of the invention, in the process of executing file migration, a configuration file of the file migration is firstly read, each migration subtask corresponding to the file to be migrated is created according to a migration flow of the file to be migrated contained in the configuration file, then a pre-created thread pool is divided into a plurality of sub-thread pools according to each migration subtask of the file to be migrated, each sub-thread pool corresponds to one migration subtask, and finally each migration subtask corresponding to the file to be migrated is executed by using the sub-thread pool corresponding to each migration subtask.
According to the embodiment of the invention, a customizable configured file migration flow can be realized, a plurality of migration subtasks are created according to the file migration flow, and a separate sub thread pool is allocated to each migration subtask, so that each migration subtask of file migration can be executed in parallel, the file migration efficiency is improved, the file migration method and the file migration system can adapt to file migration of different hardware platforms, and the compatibility is strong.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a flowchart of a file migration method provided in an embodiment of the present invention;
fig. 2 is a schematic diagram of a specific implementation flow architecture of file migration according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a file migration apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are used in an open-ended fashion, i.e., to mean including, but not limited to. Reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is for illustrative purposes to illustrate the implementation of the present application, and the sequence of steps is not limited and can be adjusted as needed.
Fig. 1 is a flowchart of a file migration method provided in an embodiment of the present invention, and as shown in fig. 1, the method may include the following steps:
s101, reading a configuration file, wherein the configuration file comprises: and (5) a migration process of the file to be migrated.
It should be noted that the migration process of the file to be migrated in S101 at least includes the following steps: the method comprises the steps of file searching, file packaging, file uploading, file analyzing, SQL file dynamic generation and SQL file execution. Optionally, the configuration file provided in the embodiment of the present invention may support dynamic configuration of the migration process, so as to adapt to differences in file migration processes executed by different hardware platforms.
It should be noted that, in order to reduce IO resource consumption caused by frequently reading the configuration file in the whole file migration process, the above S101 may read the configuration information included in the configuration file into the Redis cache.
Optionally, in order to adapt to file migration of different hardware platforms, the configuration file in S101 may further include: and migrating the parameter information of the file to be migrated based on different hardware platforms.
S102, according to the migration process, a plurality of migration subtasks corresponding to the file to be migrated are created.
It should be noted that, when creating a plurality of migration subtasks corresponding to a file to be migrated according to a migration flow of the file to be migrated, migration subtasks corresponding to each step may be created according to each step in the migration flow. For example, when the migration flow of the file to be migrated includes file search, file packaging, file uploading, file parsing, SQL file dynamic generation, and SQL file execution, six migration subtasks may be created.
S103, according to the plurality of migration subtasks corresponding to the file to be migrated, the thread pool is divided into a plurality of sub-thread pools, and each sub-thread pool corresponds to one migration subtask.
In the embodiment of the invention, a large thread pool is created in advance, and after each migration subtask corresponding to a file to be migrated is created according to a migration flow of the file to be migrated, the created large thread pool can be split into the sub thread pools corresponding to the migration subtasks, so that each sub thread pool corresponds to one migration subtask.
And S104, executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
It should be noted that after the pre-created thread pool is split into the sub-thread pools corresponding to the migration subtasks, the sub-thread pools corresponding to the migration subtasks may be used to execute the migration subtasks of the file to be migrated.
As can be seen from the above, in the file migration method provided in the embodiment of the present invention, in the process of executing file migration, the configuration file for file migration is first read, each migration subtask corresponding to the file to be migrated is created according to the migration flow of the file to be migrated included in the configuration file, then, according to each migration subtask of the file to be migrated, the thread pool created in advance is split into the multiple sub-thread pools, so that each sub-thread pool corresponds to one migration subtask, and finally, each migration subtask corresponding to the file to be migrated is executed by using the sub-thread pool corresponding to each migration subtask.
The file migration device provided by the embodiment of the invention can realize the file migration process capable of being configured in a customized manner, create a plurality of migration subtasks according to the file migration process, and allocate a separate sub thread pool to each migration subtask, so that each migration subtask of the file migration can be executed in parallel, the file migration efficiency is improved, the file migration device can adapt to the file migration of different hardware platforms, and the compatibility is strong.
In order to prevent the file migration efficiency from being affected by too large number of files to be migrated, in the embodiment of the present invention, for files to be migrated that include a large number, the files to be migrated are split into multiple migration subfiles to execute file migration, so that the migration processes of the migration subfiles can be executed in parallel. Therefore, in an embodiment, the step S104 may specifically include the following steps: acquiring a file to be migrated, wherein the file to be migrated comprises: a plurality of files to be migrated; splitting a file to be migrated into a plurality of subfiles to be migrated; and executing each migration subtask corresponding to different sub files to be migrated by using the sub thread pool corresponding to each migration subtask.
Further, in order to ensure that data synchronization or task state synchronization is achieved when a plurality of migration subtasks of a file to be migrated are processed in parallel, in the above embodiment, the file migration method provided in the embodiment of the present invention may further include the following steps: and synchronizing migration subtasks of the files to be migrated corresponding to the migration subfiles in a mode of service operation lock and file sharing.
In an embodiment, the file migration method provided in the embodiment of the present invention may further include the following steps: monitoring the execution state of each migration subtask; and recording log information of each migration subtask with failed execution. By the embodiment, the purpose of unified processing can be realized aiming at the abnormal conditions of different migration subtasks.
Because different migration subtasks are executed and the number of threads required is different, in an embodiment, the file migration method provided in the embodiment of the present invention may further include the following steps: and dynamically adjusting the size of the sub thread pool corresponding to each migration subtask. Optionally, when the size of the sub-thread pool corresponding to each migration sub-task is dynamically adjusted, the size of the corresponding sub-thread pool allocated to each migration sub-task may be determined according to the number of idle threads in the thread pool and/or the task execution priority of each migration sub-task.
Fig. 2 is a schematic diagram of a specific implementation flow architecture of file migration provided in an embodiment of the present invention, and as shown in fig. 2, a general module includes: the system comprises a configuration module, a task allocation module and an exception handling module, wherein the configuration module is used for realizing the general logic of a file migration process and loading a configuration file; the task dispatching module is used for generating subtasks; the exception handling module is used for exception retry and the like; the migration flow module is used for realizing the corresponding processing of all the subtasks, such as file screening, file searching, file packaging, file uploading, uploading result analysis, SQL file dynamic generation, SQL file execution and the like; the file module is used for storing related files involved in the file migration process, and includes but is not limited to: configuration files, general processing files (status files, log files, abnormal log files, etc.), processing result files, SQL files, etc.; the thread pool is a large thread pool consisting of a plurality of sub-thread pools, each sub-task corresponds to one sub-thread pool, the size of the sub-thread pools can be dynamically adjusted according to the existing vacant threads and the execution priority of the corresponding sub-tasks, and the execution of each sub-task can be executed only after the execution of the sub-task is acquired from the sub-thread pool. In a specific implementation process, besides the modules shown in fig. 2, some hardware resources, such as a database, a file storage server, and the like, may also be included. Taking loan documents as an example, the hardware resources may be databases, general file storage servers, individual branch file storage servers, and the file migration system, which have some IO interactions.
Therefore, the file migration method provided by the embodiment of the invention can realize the splitting of the subtasks of the file migration process, and by defining the triggering conditions and the synchronous state values of each subtask, after the triggering conditions and the synchronous state values of each subtask are determined, different developers can develop different subtasks without concerning the realization of the whole file migration process, so that the coupling among modules is ensured; meanwhile, the abnormal conditions which may occur in different subtasks are processed in a unified way, so that a unified abnormal log format specification can be defined, a universal abnormal retry mechanism can be realized, and the exceptions in all subtasks are processed by a unified abnormal processing module; because the complexity, the time complexity and the space complexity of different subtasks are different, in order to reasonably and efficiently utilize software and hardware resources to improve the efficiency of the whole file migration process, in the embodiment of the invention, after the subtasks of the file migration process are split, the task execution priority of each migration subtask can be configured in advance, so that the size of the corresponding subtask pool is dynamically allocated to each migration subtask according to the number of idle threads in the thread pool and the task execution priority of each migration subtask.
Based on the specific implementation flow architecture shown in fig. 2, taking the migration of loan documents as an example, the document migration method provided in the embodiment of the present invention may include the following two parts:
and (I) starting a migration program. Specifically, the starting of the file migration flow may be realized by calling a starting program (for example, developing a segment of shell program script, which is defined as start.
Optionally, in the embodiment of the present invention, the following five background threads are defined to execute each subtask based on the Java platform: the method comprises the following steps of loading configuration information thread loadSetting by file names, loading database record thread loadRecordsFromDB, task dispatch thread tasKDispatcher, thread pool management thread threedManager and exception handling thread exceptinonHandler.
Firstly, reading the configuration file from the configuration file by using thread loadSetting, reading all configuration items into a cache Redis, and destroying the thread when the reading of the configuration file is completed.
And secondly, connecting the thread loadRecordsFromDB to an old system database (such as a loan approval system database), searching a file record to be migrated from the database, writing the file record to be migrated into a state file, optionally splitting the file to be migrated to obtain a plurality of sub-files to be migrated, wherein each sub-file to be migrated corresponds to a unique file number.
Assuming that 100 ten thousand loan documents with document numbers of 0000000-1000000 are required to be migrated in total, 1000 sub-files to be migrated can be created for storing the 100 ten thousand loan documents, and the loan document information with the loan document numbers of 0000000-0000999 is stored in the sub-file to be migrated with the migration document number of 0001; and storing the loan file information with the loan file number of 0001000-0001999 into the sub-file to be migrated with the migration file number of 0002, and so on. It should be noted that the loan documents to be migrated may be configured by the operation and maintenance personnel of the different branch systems according to the business requirements.
Optionally, in the embodiment of the present invention, a file suffix of the file to be migrated is INIT, which is used to indicate that the file does not execute any migration subtask. When all file writes are complete, the thread will be destroyed.
And thirdly, dispatching tasks by utilizing the thread taskedispatcher, and generating different migration subtasks according to different suffix files by scanning files in the state file directory.
For example, if a file directory of rl0000320190630cn00000011.ready _ zip exists under the status file directory, and the content contained in the file meets the triggering condition of the file packing subtask, a message for generating the file packing subtask is sent to the thread pool module by using the thread taskshipcher (written into the MQ queue), and after the message is sent, the file suffix is modified into ziping to indicate that the file is executing the file packing subtask.
And fourthly, managing thread resources in the thread pool by using the thread threadManager, wherein the thread resources comprise the division of the sub-thread pool, the starting of the subtasks and the like. Polling the MQ queue, if the message exists, calling a subtask module corresponding to the message, determining whether to start the subtask according to a detection result according to whether an idle thread exists in a subtask pool corresponding to the monitoring subtask, and if the idle thread exists, directly starting the subtask; if no vacant thread exists, determining whether the capacity of a sub-thread pool corresponding to the subtask is expanded or not according to the number of the vacant threads and the priority of the subtask; if capacity expansion cannot be achieved, the current subtask can only enter a waiting queue, and the subtask is started again when waiting for the existence of the idle thread.
Utilizing thread exceptinonHandler to execute the processing of abnormal condition and reading abnormal log file to implement the execution of reactivation subtask. For example, when a FILE record named as "RL 0000320190630CN00000011-123.jpg-ZIP-FILE _ NOT _ FOUND" exists in the exception log information record FILE, it indicates that the FILE 123.jpg in the number RL0000320190630CN00000011 has an error when executing the subtask of loan FILE packaging, and the error is caused by FILE _ NOT _ FOUND, and a message is sent to the roadmanager by using the thread exceptohandler to re-execute the subtask and repair the exception; the exception information may be ignored and the operation and maintenance personnel may manually handle the exception.
And (II) executing the subtasks. The logical precedence relationship between the respective subtasks is described below, and it should be noted that the start of each subtask is independent, and there is no temporal precedence relationship. All the start of the subtasks are completed by the thread threadManager, and the successful or failed execution of the subtasks will modify the suffix value of the state file. Optionally, if the execution of the subtask fails, an exception log is additionally recorded.
File search subtasks: the image records in the state files indicate the relative paths of the files, and the subtask searches the corresponding files under the corresponding paths by reading the records in the state files; if the corresponding file cannot be found, an abnormal log is recorded, and the branch operation and maintenance personnel can be reminded to maintain the file again to the place under the corresponding path.
A file packaging sub-task: ZIP packaging is carried out on the files searched in the step I, and the packaged files are stored in a specified directory in the configuration file.
And thirdly, a file uploading subtask: and transmitting the ZIP files packaged in the step II to a new file server in an FTP connection mode.
Fourthly, uploading a result analysis sub-task (also called a file analysis sub-task): and the new file system performs warehousing processing on the uploaded ZIP packet and returns a processing result, wherein the processing result is stored in the appointed FTP directory, and the file format of the processing result can be agreed by a bilateral system. The subtask can analyze the processing result, execute subsequent processing on the files which are successfully put in storage, and execute exception processing on the files which are failed to put in storage.
And fifthly, dynamically generating the SQL subtask: and fourthly, generating an SQL file according to the successfully put-in files in the step IV, and synchronously migrating the file data after the file migration is finished.
Executing the SQL file subtask: connecting the database of the new file system, executing the SQL file generated in the fifth step, and realizing the migration of the file data.
And the customizable subtasks are as follows: the subtask does not belong to the conventional file migration function, different hardware platforms (e.g., each branch system) can be configured according to specific service requirements, for example, a certain branch file system needs to add "GD _" before the name of the image file, and then the subtask is dynamically added and embedded before the step (iv). The execution order of the other additional subtasks may be determined according to different business requirements.
Based on the same inventive concept, the embodiment of the present invention further provides a file migration apparatus, as described in the following embodiments. Because the principle of solving the problem of the embodiment of the device is similar to that of the file migration method, the implementation of the embodiment of the device can refer to the implementation of the method, and repeated details are not repeated.
Fig. 3 is a schematic diagram of a file migration apparatus provided in an embodiment of the present invention, and as shown in fig. 3, the apparatus may include: the configuration module 31 is configured to read a configuration file, where the configuration file includes: a migration process of the file to be migrated; the task creating module 32 is configured to generate each migration subtask corresponding to the file to be migrated according to the migration flow; the thread allocation module 33 is configured to split the thread pool into a plurality of sub-thread pools according to a plurality of migration subtasks corresponding to the file to be migrated, so that each sub-thread pool corresponds to one migration subtask; and the task execution module 34 is configured to execute each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
As can be seen from the above, in the file migration apparatus provided in the embodiment of the present invention, the configuration module 31 reads the configuration file of the file migration during the file migration; creating each migration subtask corresponding to the file to be migrated according to the migration flow of the file to be migrated included in the configuration file through the task creating module 32, and then creating each migration subtask corresponding to the file to be migrated according to each migration subtask of the file to be migrated; splitting a pre-established thread pool into a plurality of sub-thread pools through a thread allocation module 33, so that each sub-thread pool corresponds to one migration subtask; and executing each migration subtask corresponding to the file to be migrated by using the sub thread pool corresponding to each migration subtask through the task execution module 34.
The file migration device provided by the embodiment of the invention can realize the file migration process capable of being configured in a customized manner, create a plurality of migration subtasks according to the file migration process, and allocate a separate sub thread pool to each migration subtask, so that each migration subtask of the file migration can be executed in parallel, the file migration efficiency is improved, the file migration device can adapt to the file migration of different hardware platforms, and the compatibility is strong.
In an embodiment, the file migration apparatus provided in the embodiment of the present invention may further include: and the task synchronization module 35 is configured to synchronize migration subtasks of the files to be migrated corresponding to the respective migration subfiles in a service operation lock and file sharing manner.
In an embodiment, the file migration apparatus provided in the embodiment of the present invention may further include: a task state monitoring module 36, configured to monitor an execution state of each migration subtask; and a log information recording module 37, configured to record log information of each migration subtask whose execution fails.
In an embodiment, the file migration apparatus provided in the embodiment of the present invention may further include: and the resource dynamic adjustment module 38 is configured to dynamically adjust the size of the sub thread pool corresponding to each migration subtask.
Based on the same invention concept, the embodiment of the invention also provides computer equipment for solving the technical problems that in the prior art, the database technology is adopted to migrate files in each branch business system of a bank, the time consumption is long, the problems are difficult to be solved, and different versions of programs need to be issued for different branch business systems.
Based on the same invention concept, the embodiment of the invention also provides a computer readable storage medium, which is used for solving the technical problems that in the prior art, the database technology is adopted to migrate files in each branch business system of a bank, the time consumption is long, the problems are difficult to find, and different versions of programs need to be issued for different branch business systems.
In summary, embodiments of the present invention further provide a file migration method, an apparatus, a computer device, and a computer-readable storage medium, where the file migration is performed with sub-task splitting from two dimensions, i.e., a file migration flow and a file migration scale, and parallel processing of each migration sub-task is implemented by allocating an independent sub-thread pool to each migration sub-task, so that functions of parameterization configuration, multi-task parallel processing, unified exception handling mechanism, dynamic configuration flow, and dynamic resource adjustment of file migration can be implemented. The specific description is as follows:
firstly, parameterization configuration: the customized configuration of the file migration process is realized by reading the file migration configuration file, and the configuration file can be modified by operation and maintenance personnel of different file systems by combining hardware resources of different file systems, so that the file migration scheme can be compatible with different service requirements of different hardware platforms.
Multitask parallel processing: the file migration process relates to a plurality of steps of file positioning, file searching, file packaging, file transmission result analysis and the like. In the embodiment of the invention, the whole file migration process is divided into a plurality of subtasks, and different subtasks are distributed to process different subtasks, so that the subtasks can be processed in parallel without mutual influence. The task is split by taking the file migration scale as a dimension, all file sets needing to be migrated are split into a plurality of subsets, each subset has a separate task to execute, sub-task splitting of a file migration process is achieved from two dimensions, and file migration efficiency is further improved. Optionally, task synchronization between migration subtasks may be implemented based on technologies such as a business operation lock and a file sharing operation.
Unified exception handling mechanism: because the processing logics of different subtasks are different, the scenes that some subtasks fail to execute and the processing mechanisms after the subtasks fail to execute are different, in order to ensure the expandability of the system, in the embodiment of the invention, the abnormal conditions of each subtask are processed uniformly, and when a certain subtask fails to execute, the task execution failure log information is recorded (optionally, the recording format of the failure log adopts a preset uniform format). And the failure log information is analyzed by adopting the unified exception handling module, so that the decoupling of exception information recording and exception information handling is realized.
Fourthly, dynamically configuring the flow: different hardware platforms need to embed some special processing modules on the basis of realizing the general file migration function, for example, operations such as file renaming, file encryption transmission and the like. The configuration file provided by the embodiment of the invention supports the configuration of the file migration process and the migration parameters, and can realize the file migration customization function based on different hardware platforms.
Dynamically allocating thread resources: in the file migration process, according to each migration subtask corresponding to a file to be migrated, a pre-created thread pool is divided into a plurality of sub thread pools, so that each sub thread pool corresponds to one migration subtask, and parallel processing of each migration subtask is achieved. Optionally, the size of the sub-thread pool corresponding to each sub-task is determined according to the number of idle threads in the thread pool and the task execution priority of the sub-task, so that the purpose of dynamically adjusting resources can be achieved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (13)
1. A method for file migration, comprising:
reading a configuration file, wherein the configuration file comprises: a migration process of the file to be migrated;
creating a plurality of migration subtasks corresponding to the file to be migrated according to the migration flow;
splitting a thread pool into a plurality of sub-thread pools according to a plurality of migration subtasks corresponding to the file to be migrated, so that each sub-thread pool corresponds to one migration subtask;
and executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
2. The method of claim 1, wherein executing each migration subtask of the file to be migrated using the child thread pool corresponding to each migration subtask comprises:
acquiring a file to be migrated, wherein the file to be migrated comprises: a plurality of files to be migrated;
splitting the file to be migrated into a plurality of sub-files to be migrated;
and executing each migration subtask corresponding to different sub files to be migrated by using the sub thread pool corresponding to each migration subtask.
3. The method of claim 2, wherein the method further comprises:
and synchronizing migration subtasks of the files to be migrated corresponding to the migration subfiles in a mode of service operation lock and file sharing.
4. The method of claim 1, wherein the method further comprises:
monitoring the execution state of each migration subtask;
and recording log information of each migration subtask with failed execution.
5. The method of claim 1, wherein the method further comprises:
and dynamically adjusting the size of the sub thread pool corresponding to each migration subtask.
6. The method of claim 1, wherein the configuration file further comprises: and migrating the parameter information of the file to be migrated based on different hardware platforms.
7. The method according to any one of claims 1 to 6, wherein the migration flow of the file to be migrated at least comprises: the method comprises the steps of file searching, file packaging, file uploading, file analyzing, SQL file dynamic generation and SQL file execution.
8. A file migration apparatus, comprising:
the configuration module is used for reading a configuration file, wherein the configuration file comprises: a migration process of the file to be migrated;
the task creating module is used for generating each migration subtask corresponding to the file to be migrated according to the migration flow;
the thread allocation module is used for splitting a thread pool into a plurality of sub-thread pools according to a plurality of migration subtasks corresponding to the file to be migrated, so that each sub-thread pool corresponds to one migration subtask;
and the task execution module is used for executing each migration subtask of the file to be migrated by using the sub thread pool corresponding to each migration subtask.
9. The apparatus of claim 8, wherein the apparatus further comprises:
and the task synchronization module is used for synchronizing the migration subtasks of the migration subfiles corresponding to the files to be migrated in a business operation lock and file sharing mode.
10. The apparatus of claim 8, wherein the apparatus further comprises:
the task state monitoring module is used for monitoring the execution state of each migration subtask;
and the log information recording module is used for recording the log information of each migration subtask with failed execution.
11. The apparatus of claim 8, wherein the apparatus further comprises:
and the resource dynamic adjustment module is used for dynamically adjusting the size of the sub thread pool corresponding to each migration subtask.
12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the file migration method of any one of claims 1 to 7 when executing the computer program.
13. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the file migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010311007.0A CN111522792A (en) | 2020-04-20 | 2020-04-20 | File migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010311007.0A CN111522792A (en) | 2020-04-20 | 2020-04-20 | File migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111522792A true CN111522792A (en) | 2020-08-11 |
Family
ID=71902295
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010311007.0A Pending CN111522792A (en) | 2020-04-20 | 2020-04-20 | File migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111522792A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080016390A1 (en) * | 2006-07-13 | 2008-01-17 | David Maxwell Cannon | Apparatus, system, and method for concurrent storage pool migration and backup |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN109359085A (en) * | 2018-08-23 | 2019-02-19 | 平安科技（深圳）有限公司 | File migration method, apparatus, computer equipment and storage medium |
| CN110874344A (en) * | 2018-08-10 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data migration method and device and electronic equipment |
-
2020
- 2020-04-20 CN CN202010311007.0A patent/CN111522792A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080016390A1 (en) * | 2006-07-13 | 2008-01-17 | David Maxwell Cannon | Apparatus, system, and method for concurrent storage pool migration and backup |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN110874344A (en) * | 2018-08-10 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data migration method and device and electronic equipment |
| CN109359085A (en) * | 2018-08-23 | 2019-02-19 | 平安科技（深圳）有限公司 | File migration method, apparatus, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘晴和等: "基于业务分割的并行式大数据迁移策略研究", 《计算机与现代化》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11593149B2 (en) | Unified resource management for containers and virtual machines | |
| CN110941446B (en) | Version release method and device based on multi-environment offline task | |
| EP3889774A1 (en) | Heterogeneous computing-based task processing method and software-hardware framework system | |
| US8104043B2 (en) | System and method for dynamic cooperative distributed execution of computer tasks without a centralized controller | |
| US8181159B2 (en) | Test automation using virtual machines | |
| WO2014140971A1 (en) | Deploying parallel data integration applications to distributed computing environments | |
| KR20140070583A (en) | System and method for supporting automatically deploying/undeploying application components in a transactional middleware machine environment | |
| CN111176818B (en) | Distributed prediction method, device, system, electronic equipment and storage medium | |
| US20170206208A1 (en) | System and method for merging a mainframe data file to a database table for use by a mainframe rehosting platform | |
| US8015374B2 (en) | Method, system and computer program for backing-up data with centralized control | |
| CN112596762A (en) | Rolling upgrading method and device | |
| CN112311605B (en) | Cloud platform and method for providing machine learning service | |
| US20120072709A1 (en) | Unstacking Software Components for Migration to Virtualized Environments | |
| CN110494849B (en) | System and method for determining success of cross-platform application migration | |
| US20180189095A1 (en) | Physical to virtual scheduling system and method | |
| US20130117749A1 (en) | Provisioning and Managing an Application Platform | |
| CN111522792A (en) | File migration method and device | |
| CN115587141A (en) | Database synchronization method and device | |
| WO2022199136A1 (en) | Application modification method, and system, cluster, medium and program product | |
| Wannipurage et al. | A Framework to capture and reproduce the Absolute State of Jupyter Notebooks | |
| US8584117B2 (en) | Method to make SMP/E based products self describing | |
| CN114327770A (en) | Container cluster management system and method | |
| KR101841847B1 (en) | Method and apparatus for managing provisioning virtual disk | |
| KR102621742B1 (en) | An automation service system using a no-code based automation service platform and method thereof, and computer program | |
| US11941444B2 (en) | Facilitating scheduling of storage system pods on nodes of a cluster based on discovery of local block devices associated with the nodes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200811 |