CN117591497B - Nuclear power historical data cross-system migration method - Google Patents
Nuclear power historical data cross-system migration method Download PDFInfo
- Publication number
- CN117591497B CN117591497B CN202410070800.4A CN202410070800A CN117591497B CN 117591497 B CN117591497 B CN 117591497B CN 202410070800 A CN202410070800 A CN 202410070800A CN 117591497 B CN117591497 B CN 117591497B
- Authority
- CN
- China
- Prior art keywords
- metadata
- data
- measuring point
- writing
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013509 system migration Methods 0.000 title claims abstract description 22
- 238000005259 measurement Methods 0.000 claims abstract description 71
- 238000013508 migration Methods 0.000 claims abstract description 58
- 230000005012 migration Effects 0.000 claims abstract description 58
- 238000004891 communication Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims description 12
- 238000012800 visualization Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000013480 data collection Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 abstract description 2
- 238000013500 data storage Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 238000007667 floating Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/865—Monitoring of software
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A nuclear power historical data cross-system migration method comprises the following steps: s1, determining a source system data source, a target system data source, a driving file and a data source configuration file to be migrated in response to a data migration request, and establishing communication among the source system data source, a local system and the target system data source; s2, acquiring metadata of a measuring point to be migrated; s3, comparing the measuring point metadata with the measuring point metadata standard file, if the measuring point metadata and the measuring point metadata standard file are different, updating the measuring point metadata standard file and confirming the range of the measuring point to be migrated; s4, acquiring main data of the measuring points based on the range of the measuring points to be migrated and the migration time period, storing the main data into a local system, and generating acquisition metadata; s5, writing main data into a target system data source from a local system based on the range of the measurement point to be migrated and the migration time period, and generating writing metadata, and S6, performing integrity and traceability judgment of historical data cross-system migration based on writing and collecting metadata.
Description
Technical Field
The invention relates to the technical field of nuclear power, in particular to a nuclear power historical data cross-system migration method.
Background
Under the current condition of rapid development of big data environment and information technology, nuclear power historical data becomes a core asset of a nuclear power system, and data storage and migration become problems of concern for informatization construction of a nuclear power plant. In the updating and updating of the implementation system, the existing data is required to be migrated to a new system, but the data migration is rapidly, correctly and completely realized under the service constraint condition due to the different database structures of the new and old systems, so that the correctness, the integrity and the traceability of the nuclear power data migration process are ensured, and the support of a data migration theoretical method is lacking in the industry.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method for migrating nuclear power history data across systems to solve the above-mentioned problems.
The invention provides a nuclear power historical data cross-system migration method which comprises the following steps:
S1, determining a source system data source to be migrated, a target system data source, a driving file and a source system data source configuration file which correspond to the source system data source and the target system data source respectively, and establishing communication between the source system data source and a local system and between the local system and the target system data source based on the driving file and the source system data source configuration file; the data migration request at least comprises a source system data source, a target system data source and a migration time period;
s2, obtaining measurement point metadata to be migrated in a source system data source, wherein the measurement point metadata is used for representing the name, relation information and parameter information of the measurement point to be migrated, the relation information at least comprises factories, units and codes corresponding to the measurement point, and the parameter information at least comprises the data type corresponding to the measurement point;
S3, writing the measuring point metadata into the local system and comparing the measuring point metadata with a measuring point metadata standard file prestored in the local system, wherein the measuring point metadata standard comprises a measuring point id, a measuring point name, a data type, a power plant and unit information; if the name of the measuring point, the relation information and the parameter information are different, updating the measuring point metadata standard file based on the measuring point metadata, and reconfirming the range of the measuring point to be migrated in the data migration request based on the updated measuring point metadata standard file;
S4, collecting main data from the source system data source based on the range of the measuring point to be migrated and the migration time period, and storing the main data into a local system; meanwhile, counting a main data acquisition flow log to generate acquisition metadata, wherein the acquisition metadata is used for representing acquisition progress information of the main data;
s5, acquiring corresponding main data from a local system based on the range of the measuring point to be migrated and the migration time period, writing the main data into a data source of a target system, and simultaneously, counting a main data writing process log to generate writing metadata, wherein the writing metadata is used for representing writing progress information of the main data;
s6, based on the written metadata and the collected metadata, integrity and traceability judgment of the historical data cross-system migration are carried out to confirm the problem point.
Preferably, in the step S4, collecting main data from the source system data source based on the range of the to-be-migrated measuring points and the migration time period includes the following steps: based on the range of the measuring point to be migrated and the migration time period, splitting a primary acquisition task into a plurality of acquisition subtasks, wherein each acquisition subtask is used for acquiring main data of each day in a specified time range of a specified measuring point by taking a day as a unit, and the plurality of acquisition subtasks are performed concurrently.
Further, the acquisition metadata includes the total number of successful subtasks, the total number of acquisition lost prodigal tasks, the total acquisition progress, and the total acquisition failure date, where the total acquisition progress = total number of successful subtasks/total number of tasks.
Preferably, the main data are stored under a folder of a specified path of the local system server in a slicing mode, wherein the storage and access path of the folder corresponding to each measuring point is set as a power plant/unit/measuring point according to the metadata standard file of the measuring point, each file stores the main data of one day of the measuring point, and each file is named by a timestamp corresponding to the main data under the specified folder.
Further, storing the main data in a folder of a specified path of the local system server includes the following steps:
(A) Analyzing each main data to obtain corresponding time stamp and value to form a file, wherein the file stores the corresponding value and names the file based on the time stamp;
(B) And (3) acquiring a corresponding storage path of the measuring point in the server through the metadata standard file of the measuring point, and storing the file formed in the step (A) into a designated storage path.
Preferably, the step of obtaining corresponding main data from the local system based on the range of the measurement point to be migrated and the migration time period and writing the main data into the data source of the target system includes the following steps:
(C) Analyzing the range of the to-be-migrated measuring points to obtain all to-be-written measuring points, and obtaining the storage path of each to-be-written measuring point in a server based on the metadata standard file of the measuring points;
(D) Analyzing the migration time period to obtain all time stamps corresponding to the migration time period, inquiring the storage path of each measuring point to be written according to all time stamps under the condition of (C), and sequentially writing the files corresponding to each time stamp into a target system data source.
Preferably, the write-once task is split into a plurality of write-in subtasks based on the range of the measurement point to be migrated and the migration time period, each write-in subtask is used for writing the main data of each day in the specified time range of the specified measurement point into the target data source from the local system in a unit of a day, and the plurality of write-in subtasks are performed concurrently.
Further, the writing metadata includes total number of writing successful subtasks, total number of writing unsuccessful subtasks, total progress of writing subtasks, and writing failed date, wherein total progress of writing = total number of writing unsuccessful tasks/total number of tasks.
Preferably, the step S6 of judging the integrity and traceability of the current historical data cross-system migration based on the written metadata and the collected metadata includes the following steps:
S61, acquiring a collecting total progress and a writing total progress, and if the two values are 100%, considering that the historical data is completely migrated; if not, turning to step S62 to carry out tracing judgment;
s62, positioning of the abnormal measuring point and the abnormal date is carried out based on the acquisition failure date and the writing failure date.
Preferably, a visualization module is further provided, and the visualization module is respectively in communication connection with the database where the acquisition metadata and the writing metadata are located, so as to be used for visualization display of the total number of successful subtasks, the total number of acquisition lost prodigal tasks, the total acquisition progress/acquisition failure date, the total number of writing successful subtasks, the total number of writing sub-failure subtasks, the total writing sub-progress and the writing failure date.
Preferably, in the step S1, the data source configuration file of the source system includes metadata collection authority, metadata writing authority, main data collection authority and main data writing authority settings of the source system data source, and in the step S2, obtaining metadata of a measurement point to be migrated in the source system data source includes the following steps: reading metadata acquisition permission and metadata writing permission of a source system data source, and acquiring measurement point metadata to be migrated from the source system data source if the metadata acquisition permission and the metadata writing permission are judged to be on; and if the metadata acquisition permission and the metadata writing permission are not set, generating measurement point metadata for the source system data source in the local system.
Compared with the prior art, the invention has the following beneficial effects:
According to the nuclear power historical data cross-system migration method, firstly, the local system is used as a bridge to realize historical data migration between the source system data source and the target system data source, on one hand, 100% success cannot be guaranteed in the migration process, if failure caused by network fluctuation is caused in the middle, the next time data can be directly written into the target library from the local side without reading the data from the source system data source again, and meanwhile, the problem that the data cannot be migrated due to the fact that the source system data source cannot be directly communicated with the target system data source in part of nuclear power plants can be effectively solved.
Secondly, in the data migration process, based on the data interaction log records between the source system data source and the local system and between the local system and the target system data source, the acquisition metadata and the writing metadata used for representing the interaction process are respectively obtained at the same time, and after the historical data migration is completed, the acquisition metadata and the writing metadata are directly analyzed, so that the problem link can be conveniently and rapidly positioned and traced, and the accuracy, the completeness and the traceability management of the data migration process are ensured.
Third, in the method for transferring nuclear power historical data across systems, a measurement point metadata standard file and a main data storage standard are preset in a local system, main data are stored in a server, a storage path (power plant/unit/measurement point) of each measurement point in the server is determined according to the measurement point metadata standard file, the main data are stored in a designated folder according to day slices, each file is named according to a corresponding timestamp, when the nuclear power historical data are transferred, comparison between measurement point metadata generated based on a source system data source and the measurement point metadata standard file preset in the local system is firstly carried out, the fact that the measurement point metadata sent in different source system data sources all comprise designated relation information and parameter information is ensured, then data collection and storage are carried out according to the preset main data storage standard, and when the main data are transferred from the local system to a target data source, file searching and downloading of a corresponding time period are only needed under the designated storage path according to the timestamp information, and data verification in the file is not needed, so that quick positioning of a large amount of data can be achieved. Simultaneously, a large amount of main data is stored in the server, and occupation of a system memory can be reduced as much as possible when nuclear power historical data migration is performed.
In addition, the method also supports visual inquiry of the measuring point information, and is convenient for a user to grasp the measuring point acquisition condition, the migration progress condition and the like.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic flow chart of an embodiment of a method for cross-system migration of nuclear power historical data;
Fig. 2 is a schematic diagram of a storage structure of main data in a local system.
Detailed Description
The following detailed description of preferred embodiments of the application is made in connection with the accompanying drawings, which form a part hereof, and together with the description of the embodiments of the application, are used to explain the principles of the application and are not intended to limit the scope of the application.
Referring to fig. 1, a cross-system migration method for nuclear power historical data in the present embodiment includes the following steps:
S1, determining a source system data source to be migrated, a target system data source, driving files and data source configuration files corresponding to the source system data source and the target system data source respectively, and establishing communication between the source system data source and a local system and between the local system and the target system data source based on the driving files and the data source configuration files;
In this embodiment, a data migration request includes at least a source system data source, a target system data source, a migration time period, and a write interval, and simultaneously, a corresponding driving file and a data source configuration file are preset for each source system data source and target system data source, and when the data migration request is received, the corresponding driving file and data source configuration file are confirmed based on the data source in the request, where the driving file includes a write driver and an acquisition driver, and communication between the source system data source and a local system is established based on the acquisition driver, and communication between the local system and the target system data source is established based on the write driver; the data source configuration file is used for representing a functional range corresponding to the data source, wherein the data source configuration file of the source system comprises metadata acquisition permission, metadata writing permission, main data acquisition permission and main data writing permission of the source system data source, and the data source configuration file of the target system comprises main data acquisition permission and main data writing permission.
S2, obtaining measurement point metadata to be migrated in a source system data source, wherein the measurement point metadata are used for representing measurement point names, relation information and parameter information of the measurement points to be migrated, the relation information comprises factories, units and codes corresponding to the measurement points, and the parameter information comprises data types corresponding to the measurement points;
The metadata of the measurement points are generated from the source system data source, and some source system data sources do not provide data services including metadata and main data for the outside because of data security consideration or do not have the function of a data interface, so in this embodiment, when the metadata of the measurement points to be migrated in the source system data source is acquired, S2 firstly determines whether the metadata of the measurement points to be migrated can be acquired from the source system data source according to the source system data source configuration file, where if metadata acquisition permission and metadata writing permission of the source system data source are set to be available, the metadata of the measurement points are directly acquired from the source system data source, and if the corresponding permission is set to be unavailable, the metadata of the measurement points to be migrated is imported from the outside. In this embodiment, the information of the corresponding to-be-migrated measurement point is imported by the measurement point metadata importing module, and the measurement point metadata importing module supports the functions of downloading the measurement point metadata template, filling in data content according to the template, uploading a file and selecting whether to update synchronously during import.
And S3, comparing the measuring point metadata with a measuring point metadata standard file preset by the local system, if the difference among the measuring point names, the relation information and the parameter information exists, updating the measuring point metadata standard file based on the measuring point metadata, and confirming the measuring point to be migrated in the data migration request based on the updated measuring point metadata standard file.
Because the data storage modes and the management modes of the data sources of the different source systems are different, and the measurement point metadata are provided according to the internal formats of the data sources, the data content of the measurement point metadata provided by the data sources of the different source systems is different, and in order to unify the time sequence data formats, the bottom data storage modes (pi, iotdb or other) are shielded, so that the same data information can be obtained from the data sources of the different source systems to cooperate with the realization of the subsequent main data collection and the local storage work, and the measurement point metadata are compared with the preset measurement point metadata standard file after being obtained, in the embodiment, the format of the measurement point metadata standard file is designed as shown in the following table, wherein FQ represents the name of a power plant; 01 represents a No. 1 unit; the type of the flow representation value is floating point type, so as to ensure that the measurement point id, the measurement point name, the type, the power plant and the unit information of each measurement point in each source system data source can be obtained through the comparison of the flow representation value and the floating point type,
When the information of each measuring point is compared after the measuring point metadata is received, firstly judging whether each measuring point in the measuring point metadata exists in the measuring point metadata standard file, and if not, creating a new building for the measuring point and registering in the metadata standard; if all the measuring points exist, judging whether the metadata of the current collected data source changes or not and whether the metadata needs to be updated to the local? If any one of the measurement point name, the relation information and the parameter information is different, the update is judged to be needed.
After the updated measurement point metadata standard file is confirmed through the comparison, the step S4 is shifted to collect main data from the source system data source based on the measurement point determined in the measurement point metadata standard file, but considering that the user may also need to purposefully adjust part of the measurement points after creating the data migration request, including adding new measurement points to be collected, deleting some measurement points whose data has been collected, and the like, in this embodiment, the step of adding new measurement points or deleting measurement points in the updated measurement point metadata standard file according to the user requirement is further included. Specifically, the nuclear power historical data cross-system migration method based on metadata is further provided with a front-end display page, the measuring points in the measuring point metadata standard file can display a measuring point list outwards through the front-end display page, and a user can add new measuring points or delete appointed measuring points in the measuring point list according to requirements.
S4, acquiring main data from the source system data source based on the to-be-migrated measuring point confirmed in the S3 and the data migration request, and then writing the main data into a local system and storing the main data; meanwhile, generating acquisition metadata based on a log file of a main data acquisition flow, wherein the acquisition metadata comprises the total number of acquisition success tasks, the total number of acquisition failure tasks, the total acquisition progress and the acquisition failure date;
S5, based on the range of the measuring point to be migrated and the migration time period, corresponding measuring point metadata and main data are obtained from a local system and written into a data source of a target system, meanwhile, the main data are counted and written into a process log to generate writing metadata, and the writing metadata are used for representing writing progress information of the main data;
Because the nuclear power historical data is migrated, the data volume to be migrated is mass data with more than one hundred million levels, and when writing and reading data are performed through a database, the following problems exist: firstly, when the nuclear power historical data is read and written in a large batch through a database, the database is limited by concurrent processing capacity limitation, and the database can not respond to each thread in time, so that the situation that the data is easy to miss writing occurs; secondly, when nuclear power historical data is analyzed to form data assets, the nuclear power data which is an analysis object is generally carried out in a year unit, and when massive data above one hundred million levels are inquired across years, a large amount of time is required to be spent on finding and transmitting all specified data in a database due to overlarge data quantity (above one hundred million levels), and the response time is too long and cannot meet the requirements; thirdly, as the data is written and read through the database, the memory occupation of the system is high, and the normal operation of the program in the system can be influenced; fourth, when data is exported from a database, the data amount is too large, and the export time is slow.
As a preferential scheme, in step S4, as shown in fig. 2, a local system presets a main data storage standard, that is, main data is stored in a file form under a folder of a specified path of a local system server in a fragmentation manner, where a storage and access path of a folder corresponding to each measurement point is set as a power plant/unit/measurement point according to the metadata standard file of the measurement point, each file stores main data of one day of the measurement point and each file is named under the specified folder with a timestamp corresponding to the main data;
Then, after collecting main data from the source system data source based on the range of the measurement point to be migrated and the migration time period, storing the main data into a local system includes the following steps:
(A) Analyzing the main data to obtain corresponding time stamps and values to form a file, wherein the file stores the corresponding values and names the file based on the time stamps;
(B) And (3) acquiring a corresponding storage path of the measuring point in the server through the metadata standard file of the measuring point, and storing the file formed in the step (A) into a designated storage path. In this embodiment, the measurement point metadata standard file is stored through the mysql database, and the measurement point metadata is considered to be bound with a large number of relational queries and operations, so that the relational database is used for storage, thereby facilitating query operation statistics.
Correspondingly, the step S5 of acquiring corresponding measuring point metadata and main data from the local system based on the measuring point range to be migrated and the migration time period and writing the corresponding measuring point metadata and main data into a target system data source comprises the following steps of:
(C) Analyzing the range of the to-be-migrated measuring points to obtain all to-be-written measuring points, and obtaining the storage path of each to-be-written measuring point in a server based on the metadata standard file of the measuring points;
(D) Analyzing the migration time period to obtain all time stamps corresponding to the migration time period, inquiring the storage path of each measuring point to be written according to all time stamps under the condition of (C), and sequentially writing the files corresponding to each time stamp into a target system data source.
When the method is set, for the specified measuring point, main data acquired in a specified time period are stored in a specified folder in a server, on one hand, the system memory is not occupied, meanwhile, when a large amount of data is inquired, the data content of the specified file is not required to be inquired and read, and only the corresponding date is read in the specified storage folder, packaged and downloaded; specifically, for example, when data of the measuring point a from 1 st 2 nd 12 nd 31 st 2022 is required to be written into a target system data source from a local system, the time of 1 st 2 nd 12 nd 31 nd four years in 2019 is respectively split into 1460 time stamps, the format of each time stamp is the year, month and day, such as 20190101, then a storage path (power plant/unit/measuring point) of the measuring point a on a server is acquired according to a measuring point metadata standard file of the measuring point a, file names corresponding to the 1460 time stamps are inquired and acquired under the file folder, then the corresponding file names are found according to each time stamp, and the file is called to be written into the target system data source until all files corresponding to all time stamps are written.
Further, in this embodiment, in S4, collecting main data from the source system data source based on the range of the measurement point to be migrated and the migration time period includes the following steps: dividing a primary acquisition task into a plurality of acquisition subtasks based on the range of the measuring points to be migrated and the migration time period, wherein each acquisition subtask is used for acquiring main data of each day in a specified time range of a specified measuring point by taking a day as a unit; wherein, each acquisition task can be performed concurrently; s5, obtaining corresponding main data from a local system based on the range of the measuring point to be migrated and the migration time period and writing the main data into a target system data source, wherein the method comprises the following steps of: dividing a write-once task into a plurality of write-in subtasks based on the range of the measuring points to be migrated and the migration time period, wherein each write-in subtask is used for writing the daily main data in the specified time range of the specified measuring point into a target data source from a local system by taking a day as a unit; the above-mentioned writing subtasks can also be carried out concurrently, for each file to be written, find the corresponding main data file in the server of the local system according to the concrete date + position, then call the writing drive, write the file into the goal database;
According to the nuclear power historical data cross-system migration method, when data are collected and written, corresponding collecting metadata and writing metadata are generated synchronously based on log information of a collecting and writing process respectively, the two sets of metadata are information sets of key attributes and characteristics in the collecting and writing process respectively, clear classification and description are provided for storing main data in a data migration process, and the method is used for problem link positioning and tracing management after the data migration is completed.
In this embodiment, as a preferred solution, the collection metadata includes a total number of collection success subtasks, a total number of collection lost prodigal tasks, a total collection progress, and a total collection failure date, where the total collection progress=total number of collection success subtasks/total collection tasks. The writing metadata includes total number of successful subtasks, total number of unsuccessful subtasks, total progress of the writing, and date of the writing failure, wherein total progress of the writing = total number of successful subtasks/total number of tasks. S6, based on the written metadata and the collected metadata, performing accuracy, completeness and traceability management on the historical data migration, wherein the steps are as follows:
S61, acquiring a collecting total progress and a writing total progress, and if the two values are 100%, considering that the historical data is completely migrated; if not, turning to step S62 to carry out tracing judgment;
s62, positioning of the abnormal measuring point and the abnormal date is carried out based on the acquisition failure date and the writing failure date.
In step S61, the total progress of collection and total progress of writing is confirmed based on calculating the ratio of the total number of successful subtasks to the total task amount, if the total progress of collection and total progress of writing are both 100%, it is indicated that all the subtasks of collection and the subtasks of writing are completed, corresponding configurably data are also completely migrated, but if one of the two values is less than 100%, it is indicated that one or more subtasks of writing/subtasks of collection are problematic, corresponding to each other, the collection failure date and the writing failure date are queried, and the time and the measuring point recorded in the collection failure date and the writing failure date are taken as the problem point of the current historical data migration.
Furthermore, in this embodiment, a visualization module is further provided, and the information is displayed through the visualization module, which includes displaying the total number of successful collection subtasks, the total number of lost collection prodigal tasks, the total collection progress/collection failure date, the total number of successful writing subtasks, the total number of unsuccessful writing subtasks, the total writing progress and the date of unsuccessful writing, so that a user can grasp the collection condition, the migration progress condition, and the like of the measurement point.
According to the nuclear power historical data cross-system migration method, management of nuclear power historical time sequence data migration engineering can be achieved through setting the acquisition metadata and the writing metadata, and accuracy, completeness and traceability of a data migration process are guaranteed.
While the invention has been described with respect to the preferred embodiments, the scope of the invention is not limited thereto, and any changes or substitutions that would be apparent to those skilled in the art are intended to be included within the scope of the invention.
Claims (6)
1. A nuclear power historical data cross-system migration method is characterized by comprising the following steps of: the method comprises the following steps:
S1, determining a source system data source to be migrated, a target system data source, driving files and data source configuration files corresponding to the source system data source and the target system data source respectively, and establishing communication between the source system data source and a local system and between the local system and the target system data source based on the driving files and the source system data source configuration files; the data migration request at least comprises a source system data source, a target system data source and a migration time period;
s2, obtaining measurement point metadata to be migrated in a source system data source, wherein the measurement point metadata is used for representing the name, relation information and parameter information of the measurement point to be migrated, the relation information at least comprises factories, units and codes corresponding to the measurement point, and the parameter information at least comprises the data type corresponding to the measurement point;
S3, writing the measuring point metadata into the local system and comparing the measuring point metadata with a measuring point metadata standard file prestored in the local system, wherein the measuring point metadata standard comprises a measuring point id, a measuring point name, a data type, a power plant and unit information; if the name of the measuring point, the relation information and the parameter information are different, updating the measuring point metadata standard file based on the measuring point metadata, and reconfirming the range of the measuring point to be migrated in the data migration request based on the updated measuring point metadata standard file;
S4, collecting main data from the source system data source based on the range of the measuring point to be migrated and the migration time period, and storing the main data into a local system; meanwhile, counting a main data acquisition flow log to generate acquisition metadata, wherein the acquisition metadata is used for representing acquisition progress information of the main data;
In the step S4, collecting main data from the source system data source based on the range of the measurement points to be migrated and the migration time period includes the following steps: based on the range of the measuring point to be migrated and the migration time period, splitting a primary acquisition task into a plurality of acquisition subtasks, wherein each acquisition subtask is used for acquiring main data of each day in a specified time range of a specified measuring point by taking a day as a unit, and the plurality of acquisition subtasks are performed simultaneously;
The acquisition metadata comprises the total number of successful acquisition subtasks, the total number of acquisition lost prodigal tasks, the total acquisition progress and the total acquisition failure date, wherein the total acquisition progress = the total number of successful acquisition subtasks/the total number of acquisition tasks;
The main data are stored under a folder of a specified path of a local system server in a slicing mode in a file mode, wherein the storage and access path of the folder corresponding to each measuring point is set as a power plant/unit/measuring point according to the metadata standard file of the measuring point, each file stores the main data of one day of the measuring point, and each file is named under the specified folder by a timestamp corresponding to the main data;
storing the main data into a folder of a specified path of a local system server comprises the following steps:
(A) Analyzing each main data to obtain corresponding time stamp and value to form a file, wherein the file stores the corresponding value and names the file based on the time stamp;
(B) Acquiring a corresponding storage path of the measuring point in the server through a metadata standard file of the measuring point, and storing the file formed in the step (A) into a designated storage path;
s5, acquiring corresponding main data from a local system based on the range of the measuring point to be migrated and the migration time period, writing the main data into a data source of a target system, and simultaneously, counting a main data writing process log to generate writing metadata, wherein the writing metadata is used for representing writing progress information of the main data;
In S5, obtaining corresponding main data from the local system based on the range of the measurement point to be migrated and the migration time period, and writing the main data into the data source of the target system, including the following steps:
(C) Analyzing the range of the to-be-migrated measuring points to obtain all to-be-written measuring points, and obtaining the storage path of each to-be-written measuring point in a server based on the metadata standard file of the measuring points;
(D) Analyzing a migration time period to obtain all time stamps corresponding to the migration time period, inquiring under the condition that a storage path is determined according to all time stamps for each measuring point to be written, obtaining files corresponding to each time stamp, and sequentially writing the files into a target system data source;
s6, based on the written metadata and the collected metadata, integrity and traceability judgment of the historical data cross-system migration are carried out to confirm the problem point.
2. The nuclear power historical data cross-system migration method according to claim 1, wherein the method comprises the following steps: and S5, splitting the write-once task into a plurality of write-in subtasks based on the range of the measuring point to be migrated and the migration time period, wherein each write-in subtask is used for writing the daily main data in the specified time range of the specified measuring point into a target data source from a local system by taking a day as a unit, and the plurality of write-in subtasks are performed concurrently.
3. The nuclear power historical data cross-system migration method according to claim 1, wherein the method comprises the following steps: the writing metadata includes total number of successful subtasks, total number of unsuccessful subtasks, total progress of the writing, and date of the writing failure, wherein total progress of the writing = total number of successful subtasks/total number of tasks.
4. The nuclear power historical data cross-system migration method according to claim 1, wherein the method comprises the following steps: s6, based on the written metadata and the collected metadata, the integrity and traceability judgment of the current historical data cross-system migration comprises the following steps:
S61, acquiring a collecting total progress and a writing total progress, and if the two values are 100%, considering that the historical data is completely migrated; if not, turning to step S62 to carry out tracing judgment;
s62, positioning of the abnormal measuring point and the abnormal date is carried out based on the acquisition failure date and the writing failure date.
5. The nuclear power historical data cross-system migration method according to claim 1, further comprising a visualization module, wherein the visualization module is in communication connection with the collection metadata and the database where the writing metadata are located respectively and is used for visualization display of the total number of successful subtasks, the total number of lost prodigal tasks, the total collection progress/collection failure date, the total number of successful subtasks, the total number of failed subtasks, the total progress of the writing sub and the date of failed writing.
6. The method for cross-system migration of nuclear power historical data according to claim 1, wherein in S1, the source system data source configuration file includes metadata collection authority, metadata writing authority, main data collection authority and main data writing authority settings of a source system data source, and in S2, acquiring metadata of a measurement point to be migrated in the source system data source includes the following steps: reading metadata acquisition permission and metadata writing permission of a source system data source, and acquiring measurement point metadata to be migrated from the source system data source if the metadata acquisition permission and the metadata writing permission are judged to be on; and if the metadata acquisition permission and the metadata writing permission are not set, generating measurement point metadata for the source system data source in the local system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410070800.4A CN117591497B (en) | 2024-01-18 | 2024-01-18 | Nuclear power historical data cross-system migration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410070800.4A CN117591497B (en) | 2024-01-18 | 2024-01-18 | Nuclear power historical data cross-system migration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117591497A CN117591497A (en) | 2024-02-23 |
| CN117591497B true CN117591497B (en) | 2024-05-03 |
Family
ID=89913678
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410070800.4A Active CN117591497B (en) | 2024-01-18 | 2024-01-18 | Nuclear power historical data cross-system migration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117591497B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118092811A (en) * | 2024-04-18 | 2024-05-28 | 中核武汉核电运行技术股份有限公司 | Safety configuration method for collecting and writing multiple data sources |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096684A (en) * | 2009-12-11 | 2011-06-15 | 华大天元(北京)电力科技有限公司 | Grid real-time data integrating and sharing platform |
| CN103914573A (en) * | 2014-04-29 | 2014-07-09 | 国家电网公司 | Measuring point migration method and device |
| CN104156630A (en) * | 2014-09-05 | 2014-11-19 | 西南科技大学 | Three-dimensional nuclide diffusion computing method |
| CN108804606A (en) * | 2018-05-29 | 2018-11-13 | 上海欣能信息科技发展有限公司 | A kind of electric power measures class Data Migration to the method and system of HBase |
| CN110688255A (en) * | 2019-04-23 | 2020-01-14 | 许昌许继软件技术有限公司 | Power distribution master station system, historical data transfer method and system |
| CN112100227A (en) * | 2020-09-22 | 2020-12-18 | 国网辽宁省电力有限公司电力科学研究院 | Big data processing method based on multilevel heterogeneous data storage |
| WO2021082104A1 (en) * | 2019-10-29 | 2021-05-06 | 平安科技（深圳）有限公司 | Method and apparatus for data migration, computer device, and storage medium |
| CN113868225A (en) * | 2021-09-09 | 2021-12-31 | 国网辽宁省电力有限公司辽阳供电公司 | Method for measuring historical data migration based on power system equipment |
| CN114064563A (en) * | 2020-07-30 | 2022-02-18 | 深圳市杉岩数据技术有限公司 | Data migration method and server based on object storage |
| CN115221143A (en) * | 2022-04-26 | 2022-10-21 | 中国电子科技集团公司第十五研究所 | Cross-type migration operator-based multi-source big data processing method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110010518A1 (en) * | 2005-12-19 | 2011-01-13 | Srinivas Kavuri | Systems and Methods for Migrating Components in a Hierarchical Storage Network |
| JP5082310B2 (en) * | 2006-07-10 | 2012-11-28 | 日本電気株式会社 | Data migration apparatus and program |
-
2024
- 2024-01-18 CN CN202410070800.4A patent/CN117591497B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096684A (en) * | 2009-12-11 | 2011-06-15 | 华大天元(北京)电力科技有限公司 | Grid real-time data integrating and sharing platform |
| CN103914573A (en) * | 2014-04-29 | 2014-07-09 | 国家电网公司 | Measuring point migration method and device |
| CN104156630A (en) * | 2014-09-05 | 2014-11-19 | 西南科技大学 | Three-dimensional nuclide diffusion computing method |
| CN108804606A (en) * | 2018-05-29 | 2018-11-13 | 上海欣能信息科技发展有限公司 | A kind of electric power measures class Data Migration to the method and system of HBase |
| CN110688255A (en) * | 2019-04-23 | 2020-01-14 | 许昌许继软件技术有限公司 | Power distribution master station system, historical data transfer method and system |
| WO2021082104A1 (en) * | 2019-10-29 | 2021-05-06 | 平安科技（深圳）有限公司 | Method and apparatus for data migration, computer device, and storage medium |
| CN114064563A (en) * | 2020-07-30 | 2022-02-18 | 深圳市杉岩数据技术有限公司 | Data migration method and server based on object storage |
| CN112100227A (en) * | 2020-09-22 | 2020-12-18 | 国网辽宁省电力有限公司电力科学研究院 | Big data processing method based on multilevel heterogeneous data storage |
| CN113868225A (en) * | 2021-09-09 | 2021-12-31 | 国网辽宁省电力有限公司辽阳供电公司 | Method for measuring historical data migration based on power system equipment |
| CN115221143A (en) * | 2022-04-26 | 2022-10-21 | 中国电子科技集团公司第十五研究所 | Cross-type migration operator-based multi-source big data processing method |
Non-Patent Citations (1)
| Title |
|---|
| "核电厂关键与敏感设备智慧管理平台建设研究";刘旭嘉等;《中国核电》;20221231;刘旭嘉 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117591497A (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117591497B (en) | Nuclear power historical data cross-system migration method | |
| CN103793479A (en) | Log management method and log management system | |
| US7290018B2 (en) | Change-driven replication of data | |
| US20080091742A1 (en) | System and method for detecting and updating geographical information dataset versions | |
| CN113111129A (en) | Data synchronization method, device, equipment and storage medium | |
| CN112016287B (en) | Data management method, platform, storage medium and electronic equipment | |
| CN116842055A (en) | System and method for integrated processing of internet of things data batch flow | |
| US20110295761A1 (en) | Business form management system, method and program | |
| CN116303310B (en) | Data sharing method and system of industrial Internet | |
| CN111858767A (en) | Synchronous data processing method, device, equipment and storage medium | |
| CN111459943A (en) | Data processing method, device, system, equipment and storage medium | |
| CN117591495B (en) | Data storage method and system for nuclear power historical data migration | |
| CN116186116A (en) | Asset problem analysis method based on equal protection assessment | |
| US20080155352A1 (en) | Method and system for carrying out an operation based on a log record of a computer program | |
| CN112395476A (en) | Engineering data management method | |
| CN115237875B (en) | Log data processing method, device, equipment and storage medium | |
| CN111507066B (en) | Method and system for uploading and loading files and analyzing and warehousing | |
| CN117591577B (en) | Nuclear power historical data comparison method and system based on file storage | |
| CN117608536B (en) | Gap data online template customization and supplementary recording system and method thereof | |
| CN117609414B (en) | Method for automatically extracting space metadata and automatically associating space metadata with space data model | |
| CN115309751A (en) | Data resource center construction method and system | |
| CN114996052A (en) | Database backup device and method | |
| CN118069667A (en) | Heterogeneous database synchronization method based on log | |
| CN115934468A (en) | Log acquisition method and device and computer readable storage medium | |
| CN116108035A (en) | Data recovery method and device, processor and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |