CN114860694B - Asynchronous collaborative data migration method and device for wind power plant monitoring system - Google Patents
Asynchronous collaborative data migration method and device for wind power plant monitoring system Download PDFInfo
- Publication number
- CN114860694B CN114860694B CN202210780970.2A CN202210780970A CN114860694B CN 114860694 B CN114860694 B CN 114860694B CN 202210780970 A CN202210780970 A CN 202210780970A CN 114860694 B CN114860694 B CN 114860694B
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- database
- target
- job
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an asynchronous collaborative data migration method and device for a wind power plant monitoring system, wherein the method comprises the steps of preprocessing of a target host and a source host, data migration and integrity verification after the data migration of a database of the target host; the data migration comprises the steps of generating migration tasks of a target quantity according to a received migration strategy, executing the migration tasks in parallel, and migrating the data of the corresponding service types stored in the database of the source host to the database of the target host. According to the invention, through environment detection, data extraction and data conversion, data conversion result verification, target database backup, target database table clearing, data migration and integrity verification after target database migration, the whole data migration process is reasonably planned by adopting task arrangement and task scheduling, the workload of operators is greatly reduced, and the migration efficiency is improved under the condition that an old monitoring system and an intelligent system can normally operate at the same time.
Description
Technical Field
The invention relates to the technical field of data migration, in particular to an asynchronous collaborative data migration method and device for a wind power plant monitoring system.
Background
With the continuous development of intelligent technology, the wind field monitoring system is transformed from a traditional digital system to an intelligent system, after the development of a new intelligent system is completed, the original monitoring system with different designs needs to be switched to the new system, and in order to ensure the integrity of user data experience, the first task of system switching is to integrate the data resources of the original monitoring system. Data resource consolidation includes two steps: data arrangement and data conversion, wherein the arrangement is to arrange original system data into data which can be identified by a system conversion program, the data arrangement is very difficult, the related data amount is large, and the data arrangement cannot be completed through manual inspection; the data conversion is to convert the sorted data into a data format required by a new system according to a certain conversion rule, and the new and old system migration is to make a feasible plan on the basis of correct data conversion so as to ensure smooth and stable transition of service transaction to the new system.
The whole migration process can be faced with the work of network environment detection, program dependence environment deployment, data arrangement and conversion, data conversion result verification, data migration, data integrity verification and the like, and if the work is completed step by step manually, the workload is large, the time is wasted, and the error probability is increased. Therefore, how to provide an efficient data migration method for a wind power plant monitoring system is a technical problem which needs to be solved urgently.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide an asynchronous collaborative data migration method and device for a wind power plant monitoring system, and aims to solve the technical problem that the normal operation of a source monitoring system and a new monitoring system cannot be guaranteed when the data migration workload of the existing wind power plant monitoring system is large and the data migration cannot be guaranteed.
In order to achieve the above object, the present invention provides an asynchronous collaborative data migration method for a wind farm monitoring system, which comprises the following steps:
s1: preprocessing a target host and a source host;
s2: data migration; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing the migration tasks in parallel, and migrating the data of the corresponding service types stored in the database of the source host to the database of the target host;
s3: integrity verification after data migration of the database of the target host.
Optionally, the preprocessing of the target host and the source host includes: environment detection and dependency package installation, data extraction and data conversion, data conversion result verification and database preprocessing of a target host.
Optionally, the environment detection includes environment detection for the target host and the source host, and the installation of the dependency package includes installation of a migration program dependency package under the target host.
Optionally, after the installation of the dependent package, performing a Docker service detection and a Mysql service detection on the target host.
Optionally, the data extraction includes a basic data table and a service data table for the source host, and the data conversion includes a vertical-horizontal table conversion for the basic data table and a mapping conversion for the service data table.
Optionally, the crossbar table conversion includes: extracting a data dictionary in a database of a source host computer into a memory, and converting the data dictionary into a transverse table according to the mapping relation between the parameter serial number and the maximum value, the minimum value and the average value; the mapping conversion comprises: and loading a service data table of the database of the target host computer into the memory, and automatically mapping the rows with the transverse table according to the service rule.
Optionally, the data conversion result verification includes verification of a conversion result of the obtained transverse table and verification of a conversion result of the converted service data table.
Optionally, the database preprocessing of the target host includes: database backup of the target host and database inventory of the target host.
Optionally, when migrating the data of the corresponding service type stored in the database of the source host to the database of the target host, the method further includes:
acquiring data migration progress information and sending the data migration progress information to a terminal;
the data migration progress information comprises data table migration success information, the number of data tables of a database of the source host and the number of data tables of a database of the target host.
In addition, in order to achieve the above object, the present invention further provides an asynchronous collaborative data migration apparatus for a wind farm monitoring system, where the apparatus includes:
the preprocessing module is used for executing preprocessing on the target host and the source host;
the data migration module is used for executing data migration on the target host and the source host; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing the migration tasks in parallel, and migrating the data of the corresponding service types stored in the database of the source host to the database of the target host;
and the integrity verification module is used for executing integrity verification on the data after the database of the target host is migrated.
The method comprises the steps of preprocessing a target host and a source host, data migration and integrity verification after the data migration of a database of the target host; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing the migration tasks in parallel, and migrating the data of the corresponding service types stored in the database of the source host to the database of the target host. According to the invention, through environment detection, data extraction and data conversion, data conversion result verification, target database backup, target database table clearing, data migration and integrity verification after target database migration, the whole data migration process is reasonably planned by adopting task arrangement and task scheduling, the workload of operators is greatly reduced, and the migration efficiency is improved under the condition that an old monitoring system and an intelligent system can normally operate at the same time.
Drawings
FIG. 1 is a schematic flow chart of an asynchronous collaborative data migration method of a wind farm monitoring system according to the present invention;
FIG. 2 is a schematic diagram of a data migration task definition according to the present invention;
fig. 3 is a schematic diagram of env.ini profile definition according to the present invention;
FIG. 4 is a schematic diagram of JDK and PIP environment configuration of the present invention;
FIG. 5 is a flowchart illustrating detection of a Docker service and a Mysql service according to the present invention;
FIG. 6 is a diagram illustrating a verification rule profile definition according to the present invention;
FIG. 7 is a schematic diagram of an asynchronous cooperative data migration apparatus of a wind farm monitoring system according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
At present, in the related technical field, the data migration workload of a wind power plant monitoring system is large, and normal operation of a source monitoring system and a new monitoring system cannot be guaranteed when data migration is carried out.
In order to solve the problem, various embodiments of the asynchronous collaborative data migration method of the wind farm monitoring system are provided. According to the asynchronous collaborative data migration method for the wind power plant monitoring system, the whole data migration process is reasonably planned by task arrangement and task scheduling through environment detection, data extraction and data conversion, data conversion result verification, target database backup, target database inventory, data migration and integrity verification after target database migration, the workload of operators is greatly reduced, and the migration efficiency is improved under the condition that an old monitoring system and an intelligent system can normally operate at the same time.
The embodiment of the invention provides an asynchronous collaborative data migration method for a wind power plant monitoring system, and with reference to fig. 1, fig. 1 is a schematic flow diagram of an embodiment of the asynchronous collaborative data migration method for the wind power plant monitoring system.
In this embodiment, the asynchronous collaborative data migration method for the wind farm monitoring system includes the following steps:
s1: preprocessing a target host and a source host;
s2: data migration; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing each migration task in parallel, and migrating the data of the corresponding service type stored in the database of the source host to the database of the target host;
s3: integrity verification after data migration of a database of a target host.
It should be noted that, in this embodiment, the preprocessing of the target host and the source host includes steps of environment detection and dependency installation, data extraction and data conversion, data conversion result verification, target database backup, target database inventory, and the like.
Specifically, referring to fig. 2, the asynchronous collaborative data migration method of the wind farm monitoring system includes 7 steps to implement the asynchronous collaborative data migration process of the wind farm monitoring system from the source host to the target host.
step-1: environment detection and dependency installation, comprising the following sub-steps: job-1-1: detecting the environment of a target host and a source host; job-1-2: installing a dependency package required by a migration program in an LINUX environment of an intelligent system server; job-1-3: mysql service detection.
In the embodiment, the host environment detection comprises the steps of setting the host IP, alias and time difference value to be checked to be 30s in a configuration file mode, calling check _ evn () through a script file, feeding back a time _ diff () function to a user host whether to be communicated or not and whether the host time is consistent or not, and feeding back the user residual space size and whether the residual space size meets the data backup requirement or not through executing a domain _ space () function; automatically executing JAVA environment detection and PIP environment detection through a script file, and automatically installing required SQLAlchemy, pandas and Pymysql dependent files according to different LINUX operating systems such as Centos or kylin operating systems; and executing a check _ Mysql () function through the script file to feed back whether the Mysql service on the host to be operated by the user is started or not and whether the user name and the password of the database and the port are correct or not to prepare for the subsequent steps of migration.
step-2: data extraction and conversion, comprising the following sub-steps: job-2-1: converting a base table into a vertical table and a horizontal table; and job-2-2 mapping conversion of other service tables.
In this embodiment, the old system data dictionary is sorted, the old system data quality is analyzed, the new system data dictionary is sorted, the new and old system data difference is analyzed, and the mapping relationship between the new and old system data is established according to the business rules. And scheduling the task, calling the job-2-1 task, extracting the original monitoring system data dictionary from the memory, automatically converting the original monitoring system data dictionary into a transverse table A according to the mapping relation between the parameter serial number and the maximum value, the minimum value, the average value and the like, loading a new monitoring system basic table B into the memory, automatically mapping the new monitoring system basic table B to the transverse table A according to the service rule, and calling the job-2-2 task to map and convert other service tables.
step-3: the data conversion result verification comprises the following sub-steps: job-3-1: verifying whether the horizontal and vertical table conversion is correct or not according to the configuration file rule; job-3-2: and detecting whether the conversion results of other tables are correct or not according to the configuration file service rule.
In this embodiment, after the conversion execution is finished, the conversion result needs to be verified, and the job scheduling calls the job-3-1 and job-3-2 to execute the verification script for verification, so that the execution is not stopped by the subsequent task.
step-4: the backup of the target database comprises the following substeps: job-4-1: and backing up the target database.
In this embodiment, after the verification, job-4-1 is called to perform backup on the intelligent system database according to a backup strategy, where the backup strategy is divided into two parts, one is to perform full backup of the database in a specified time period for the online system, and the other is to perform backup of a database table structure and a basic configuration table for the test system.
step-5: the target database inventory comprises the following substeps: job-5-1: and performing table clearing operation on the designated table of the target database.
In the embodiment, after the database file backup is completed, the task scheduling performs table clearing operation on the intelligent system database according to the joba-5-1 and the database clearing strategy.
step-6: data migration, comprising the following sub-steps: job-6-1: a migration task A; job-6-2: a migration task B; job-6-3: a migration task C; job-6-4: and migrating the task D.
In this embodiment, task scheduling calls joba-6-1 to migrate a database after a library cleaning task is successfully executed, 4 jobs are configured in a configuration file according to a migration policy, and 20 concurrencies are set, so that data tables of different service types of an old monitoring system are synchronized into a new intelligent system, synchronization progress of joba is output in real time through a terminal like a user in a synchronization process, whether each table is successfully migrated is recorded in an image log file, the total number of records of table migration of an original monitoring system and the number of records of a table corresponding to the new intelligent system after migration is completed are recorded.
step-7: the integrity verification after the target database is migrated comprises the following sub-steps: job-7-1: and carrying out data integrity verification after the target database is migrated.
In this embodiment, after all tables are successfully migrated, task scheduling triggers joba-7-1 to detect integrity scripts to extract records that are not successfully migrated from log files, and counts the migration success rate, and if 100% of the entire migration tasks are completed, and if less than 100%, manual analysis is performed according to the records that are not successfully extracted.
In order to explain the application more clearly, a specific example of the asynchronous collaborative data migration method of the wind farm monitoring system is provided.
(1) The whole migration process definition Task represents a data migration Task, step represents steps, the steps are executed in series, job represents specific things to do, and the Job is executed in parallel in the Step, and FIG. 2 shows the whole definition.
(2) And performing task arrangement according to the definition shown in FIG. 2, sequentially executing the following steps in a thread pool mode, executing jobs in the steps concurrently, and defining a configuration file in step-1 environment detection and dependency installation.
env.ini sets xxxx.xxxx.xxxx.xxxx.xxxx represents an IP address and node represents a host alias as shown in fig. 3; root _ password is the original monitoring system server user name and password; the time _ sync _ diff is set to 30s, and the two hosts are considered to be consistent when the time difference between the two hosts is less than or equal to 30 s. And the starting thread 1 calls the env.sh file to execute the detection of whether the host connectivity, the host time difference and the disk residual space capacity meet the space requirement of the backup database.
As shown in fig. 4, the start thread 2 executes env.sh script in parallel to configure JDK and PIP environments, and the migration program needs to install a dependent package.
Specifically, whether OpenJdk exists or not is judged, and if yes, unloading is executed; if not, java-version judges whether the jdk is installed.
After that, if jdk is not installed, etc/profile (i.e. environment variable configuration file) is backed up, jdk environment variables are configured in etc/profile, and etc/profile is executed by using source command. If the jdk is installed, the pip-version judges whether the pip is installed, and if so, the pip installation and migration program dependent package file is executed; if not, after the pip installation is executed, the pip installation migration program dependent package file is executed.
As shown in fig. 5, the start thread 3 performs the Docker service and Mysql service detection in parallel.
Specifically, whether the Docker service is started or not is judged firstly, if not, the Docker service is started, and then, whether the Mysql service is started or not is judged, and if not, the Mysql service is started, so that detection of the Docker service and the Mysql service is completed.
(3) According to the definition shown in fig. 2, after the environment detection and the dependency package installation are completed, step-2 data extraction and conversion are executed, the original monitoring system data dictionary is extracted and automatically converted into a transverse table A in the memory according to the mapping relation between the parameter serial number and the maximum value, the minimum value, the average value and the like, and a new monitoring system basic table B is loaded to the memory to automatically map the transverse table A in rows according to the business rules.
(4) According to the data conversion result verification of the step-3 of task scheduling execution after the definition mapping shown in fig. 2 is completed, the rule of conversion of the vertical table and the horizontal table is shown in the configuration file table. Ini as shown in fig. 6, columns represent the number of columns generated by converting the vertical table into the horizontal table, a thread is started to inquire whether the number of columns of the horizontal table and columns after the conversion of the old monitoring system are the same, and whether the number of intermediate tables generated by other service types is the same as the intermediate _ table in the configuration file.
(5) And according to the definition shown in the figure 2, after the checking is finished, the task scheduling starting thread sequentially executes the backup of the step-4 database and the database cleaning operation of the step-5 database.
(6) According to the definition of the task scheduling starting thread after the database backup and database cleaning operation is completed, step-6 data migration operation is executed, 4 jobs are defined in the data migration step, a job-6-1 migration task A, job-6-2 migration task B, job-6-3 migration task C, job-6-4 migration task D starts four threads to concurrently execute the migration process.
(7) And after the migration is finished, the task scheduling executes step-7 data integrity check to check the total number of the data tables of different service types which are respectively migrated.
The embodiment provides an asynchronous collaborative data migration method for a wind power plant monitoring system, which reasonably plans the whole data migration process by adopting task arrangement and task scheduling through environment detection, data extraction and data conversion, data conversion result verification, target database backup, target database inventory, data migration and integrity verification after target database migration, reduces the workload of operators to a great extent, and improves the migration efficiency under the condition of ensuring that an old monitoring system and an intelligent system can normally operate simultaneously.
Referring to fig. 7, fig. 7 is a structural block diagram of an embodiment of an asynchronous cooperative data migration device of a wind farm monitoring system according to the present invention.
As shown in fig. 7, the asynchronous collaborative data migration apparatus for a wind farm monitoring system according to the embodiment of the present invention includes:
a preprocessing module 10, configured to perform preprocessing on a target host and a source host;
a data migration module 20, configured to perform data migration on the target host and the source host; the data migration comprises the steps of generating migration tasks of a target quantity according to a received migration strategy, executing the migration tasks in parallel, and migrating the data of the corresponding service types stored in the database of the source host to the database of the target host;
and the integrity verification module 30 is configured to perform integrity verification on the data after the database of the target host is migrated.
Other embodiments or specific implementation manners of the asynchronous collaborative data migration device of the wind farm monitoring system according to the present invention may refer to the above method embodiments, and are not described herein again.
In addition, an embodiment of the present invention further provides a storage medium, where the storage medium stores a wind farm monitoring system asynchronous collaborative data migration method program, and when the wind farm monitoring system asynchronous collaborative data migration method program is executed by a processor, the steps of the wind farm monitoring system asynchronous collaborative data migration method described above are implemented. Therefore, a detailed description thereof will be omitted. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application. It is determined that the program instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or distributed across multiple sites and interconnected by a communication network, as examples.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by a computer program, which may be stored in a computer readable storage medium and includes the processes of the embodiments of the methods described above when the program is executed. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
It should be noted that the above-described embodiments of the apparatus are merely schematic, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement without inventive effort. Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by software plus necessary general hardware, and may also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions may be various, such as analog circuits, digital circuits, or dedicated circuits. However, the implementation of a software program is a more preferable embodiment for the present invention. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, where the computer software product is stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
Claims (4)
1. A wind power plant monitoring system asynchronous collaborative data migration method is characterized by comprising the following steps:
s1: preprocessing a target host and a source host, wherein the preprocessing comprises environment detection and dependency package installation, data extraction and data conversion, data conversion result verification and database preprocessing of the target host; wherein:
the environment detection comprises network environment detection of a target host and a source host, and the service environment detection comprises whether a target host Docker service and a Mysql service are started or not; the dependent package installation comprises installing a migration program dependent package under a target host;
the data extraction and conversion comprises: extracting a data dictionary of the original monitoring system from the memory, automatically converting the data dictionary into a transverse table A according to the mapping relation between the parameter serial number and the maximum value, the minimum value and the average value, and loading a basic table B of a new monitoring system into the memory to automatically map the transverse table A with columns according to the business rules;
the data conversion result verification comprises: verifying the obtained conversion result of the transverse table and verifying the conversion result of the converted service data table;
s2: data migration; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing each migration task in parallel, and migrating the data of the corresponding service type stored in the database of the source host to the database of the target host;
s3: integrity verification after data migration of a database of the target host;
the method adopts a task scheduling and task scheduling mode to execute data migration, and comprises the following steps:
step-1: environment detection and dependency installation, comprising the following sub-steps: job-1-1: detecting the environment of a target host and a source host; job-1-2: installing a dependency package required by a migration program in an LINUX environment of an intelligent system server; job-1-3: detecting Mysql service;
step-2: data extraction and conversion, comprising the following sub-steps: job-2-1: converting a base table into a vertical table and a horizontal table; mapping and converting the jobs-2-2 other service tables;
step-3: the data conversion result verification comprises the following sub-steps: job-3-1: verifying whether the horizontal and vertical table conversion is correct or not according to the configuration file rule; job-3-2: detecting whether the conversion results of other tables are correct or not according to the service rules of the configuration files;
step-4: the backup of the target database comprises the following substeps: job-4-1: backing up a target database;
step-5: the target database tabulation method comprises the following substeps: job-5-1: performing table clearing operation on the designated table of the target database;
step-6: data migration, comprising the following sub-steps: job-6-1: a migration task A; job-6-2: a migration task B; job-6-3: a migration task C; job-6-4: a migration task D;
step-7: the integrity verification after the target database is migrated comprises the following sub-steps: job-7-1: and carrying out data integrity verification after the target database is migrated.
2. The wind farm monitoring system asynchronous collaborative data migration method according to claim 1, wherein the database preprocessing of the target host includes: database backup of the target host and database inventory of the target host.
3. The asynchronous collaborative data migration method for the wind farm monitoring system according to claim 1, wherein when migrating data of a corresponding traffic type stored in the database of the source host to the database of the target host, the method further comprises:
acquiring data migration progress information and sending the data migration progress information to a terminal; the data migration progress information comprises data table migration success information, the number of data tables of a database of the source host and the number of data tables of a database of the target host.
4. An asynchronous collaborative data migration device for a wind farm monitoring system, the device comprising:
the system comprises a preprocessing module, a data extraction module, a data conversion module and a data conversion module, wherein the preprocessing module is used for executing preprocessing on a target host and a source host, and the preprocessing comprises environment detection and dependency package installation, data extraction and data conversion, data conversion result verification and database preprocessing of the target host; wherein:
the environment detection is used for detecting the network environment of the target host and the source host, and the service environment detection comprises whether the Docker service and the Mysql service of the target host are started or not; the dependent package installation comprises installing a migration program dependent package under a target host;
the data extraction and conversion is used for extracting the mapping relation between the original monitoring system data dictionary and the internal memory according to the parameter serial number, the maximum value, the minimum value and the average value, automatically converting the mapping relation into a transverse table A, and loading a new monitoring system basic table B to the internal memory to automatically perform column mapping with the transverse table A according to the business rule;
the data conversion result verification is used for verifying whether the horizontal and vertical table conversion is correct and verifying the conversion result of the converted service data table according to the configuration file rule;
the data migration module is used for executing data migration on the target host and the source host; the data migration comprises the steps of generating migration tasks of a target number according to a received migration strategy, executing each migration task in parallel, and migrating the data of the corresponding service type stored in the database of the source host to the database of the target host;
the integrity verification module is used for performing integrity verification on the data after the database of the target host is migrated;
the task scheduling and dispatching module is used for executing the following steps by adopting a task scheduling and dispatching mode when data migration is executed:
step-1: environment detection and dependency installation, comprising the following sub-steps: job-1-1: detecting the environment of a target host and a source host; job-1-2: installing a dependency package required by a migration program in an LINUX environment of an intelligent system server; job-1-3: detecting Mysql service;
step-2: data extraction and conversion, comprising the following sub-steps: job-2-1: converting a base table into a vertical table and a horizontal table; mapping and converting the jobs-2-2 other service tables;
step-3: the data conversion result verification comprises the following sub-steps: job-3-1: verifying whether the horizontal and vertical table conversion is correct or not according to the configuration file rule; job-3-2: detecting whether the conversion results of other tables are correct or not according to the configuration file service rule;
step-4: the backup of the target database comprises the following substeps: job-4-1: backing up a target database;
step-5: the target database inventory comprises the following substeps: job-5-1: performing table clearing operation on the designated table of the target database;
step-6: data migration, comprising the following substeps: job-6-1: a migration task A; job-6-2: a migration task B; job-6-3: a migration task C; job-6-4: a migration task D;
step-7: the integrity verification after the target database is migrated comprises the following sub-steps: job-7-1: and carrying out data integrity verification after the target database is migrated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210780970.2A CN114860694B (en) | 2022-07-05 | 2022-07-05 | Asynchronous collaborative data migration method and device for wind power plant monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210780970.2A CN114860694B (en) | 2022-07-05 | 2022-07-05 | Asynchronous collaborative data migration method and device for wind power plant monitoring system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114860694A CN114860694A (en) | 2022-08-05 |
| CN114860694B true CN114860694B (en) | 2022-11-01 |
Family
ID=82626813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210780970.2A Active CN114860694B (en) | 2022-07-05 | 2022-07-05 | Asynchronous collaborative data migration method and device for wind power plant monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114860694B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115129661B (en) * | 2022-08-30 | 2022-11-22 | 东方电气风电股份有限公司 | Data migration method and system after power-off restart of wind field monitoring system server |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109002474A (en) * | 2018-06-15 | 2018-12-14 | 北京信安世纪科技股份有限公司 | A kind of data migration method and device |
| CN109067823A (en) * | 2018-06-11 | 2018-12-21 | 中国平安人寿保险股份有限公司 | A kind of data migration method, device, computer equipment and storage medium |
| CN113792025A (en) * | 2021-08-11 | 2021-12-14 | 浪潮软件集团有限公司 | MMT data migration method based on domestic CPU environment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1763745A (en) * | 2005-09-06 | 2006-04-26 | 阎海强 | Structure oriented data migration method and system |
| US20070100805A1 (en) * | 2005-09-14 | 2007-05-03 | Jorey Ramer | Mobile content cross-inventory yield optimization |
| US11030166B2 (en) * | 2018-11-22 | 2021-06-08 | Accenture Global Solutions Limited | Smart data transition to cloud |
| CN109918442A (en) * | 2019-01-24 | 2019-06-21 | 中国联合网络通信集团有限公司 | A kind of processing method and processing device of data |
| CN111325626B (en) * | 2020-02-21 | 2023-07-25 | 中国工商银行股份有限公司 | Account checking data pushing method and device |
| CN111752928A (en) * | 2020-06-30 | 2020-10-09 | 武汉虹信技术服务有限责任公司 | Database object migration method and system |
-
2022
- 2022-07-05 CN CN202210780970.2A patent/CN114860694B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067823A (en) * | 2018-06-11 | 2018-12-21 | 中国平安人寿保险股份有限公司 | A kind of data migration method, device, computer equipment and storage medium |
| CN109002474A (en) * | 2018-06-15 | 2018-12-14 | 北京信安世纪科技股份有限公司 | A kind of data migration method and device |
| CN113792025A (en) * | 2021-08-11 | 2021-12-14 | 浪潮软件集团有限公司 | MMT data migration method based on domestic CPU environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114860694A (en) | 2022-08-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765026B (en) | Automatic test method, device, storage medium and equipment | |
| US9477581B2 (en) | Integrated system and method for validating the functionality and performance of software applications | |
| CN102567203A (en) | Method and system for testing performance of distributed file system (DFS) | |
| CN111125444A (en) | Big data task scheduling management method, device, equipment and storage medium | |
| US20150100830A1 (en) | Method and system for selecting and executing test scripts | |
| CN108153675A (en) | A kind of Android application automated testing methods towards mobile cloud computing | |
| CN109063005B (en) | Data migration method and system, storage medium and electronic device | |
| CN112463631A (en) | Chip driver testing method, device and equipment and readable storage medium | |
| CN114860694B (en) | Asynchronous collaborative data migration method and device for wind power plant monitoring system | |
| CN112650676A (en) | Software testing method, device, equipment and storage medium | |
| CN115102877B (en) | Virtual network card network detection method, device, equipment and medium | |
| CN116166525A (en) | Method and device for generating test script | |
| CN112000567A (en) | Regulation and control software test service method based on cloud platform | |
| CN111787045A (en) | Migration method and device from private cloud to public cloud | |
| CN114610334A (en) | Product visualization deployment method, device, system and storage medium | |
| CN110750445A (en) | Method, system and equipment for testing high-availability function of YARN component | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN114286366A (en) | Wireless grid network upgrading method and device, main node and storage medium | |
| CN114050989A (en) | Distributed test execution method based on cloud computing technology | |
| CN114356643B (en) | Automatic task discovery failure and recovery method in remote sensing satellite processing system | |
| CN110908768B (en) | Operation and maintenance method, device and equipment for virtualization platform and readable storage medium | |
| CN117573327B (en) | Method, equipment and storage medium for intelligent scheduling and trend monitoring | |
| CN114817393B (en) | Data extraction and cleaning method and device and storage medium | |
| CN111581014B (en) | Method and device for restarting automatic control host and related equipment | |
| CN100518100C (en) | Device and method of automatic test for pressure resistance capability of communication system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |