CN118092811B - Safety configuration method for collecting and writing multiple data sources - Google Patents
Safety configuration method for collecting and writing multiple data sources Download PDFInfo
- Publication number
- CN118092811B CN118092811B CN202410465038.XA CN202410465038A CN118092811B CN 118092811 B CN118092811 B CN 118092811B CN 202410465038 A CN202410465038 A CN 202410465038A CN 118092811 B CN118092811 B CN 118092811B
- Authority
- CN
- China
- Prior art keywords
- data source
- data
- api interface
- source
- driving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000013508 migration Methods 0.000 claims abstract description 19
- 230000005012 migration Effects 0.000 claims abstract description 19
- 238000004891 communication Methods 0.000 claims description 6
- 238000004806 packaging method and process Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 6
- 238000007792 addition Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4488—Object-oriented
- G06F9/449—Object-oriented method invocation or resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4494—Execution paradigms, e.g. implementations of programming paradigms data driven
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Water Supply & Treatment (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Primary Health Care (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a safety configuration method for collecting and writing multiple data sources, which comprises the steps of determining the contents of the data sources and tasks according to a data migration request; uploading the corresponding drive realization package and configuration file according to the corresponding relation among the drive realization package, the configuration file and the data source, and setting an API interface module of the drive frame, wherein the first API interface is used for executing initialization of the drive frame; the second API interface is used for acquiring metadata corresponding to the data source based on the data source identifier; the third API interface is used for acquiring main data corresponding to the data source based on the data source identification and the metadata; the fourth API interface is used for writing the main data into the target system data source based on the data source identification; and calling the driving instance and the corresponding API interface module to execute task content. When a data source is newly added, the package and the configuration file are realized through the corresponding drive, so that the subsequent data acquisition and/or writing operation of the data source can be realized, the universality is strong, and the operation is convenient.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a safety configuration method for collecting and writing multiple data sources.
Background
Under the current condition of rapid development of big data environment and information technology, nuclear power historical data becomes a core asset of a nuclear power system, and data storage and migration become problems of concern for informatization construction of a nuclear power plant.
In the domestic updating of nuclear power software, the existing data is required to be migrated from a source system to a database of a new system, at present, when nuclear power data migration is performed, data acquisition and writing operation are usually performed from a data access system of the source system in a hard coding mode, because the data access systems of the source systems are different in different nuclear power stations and the access systems of different types of systems are also different, including databases, HTTP interfaces, object storage services, data files and the like, in addition, one system may also comprise a plurality of access modes, and when the access mode of the source system changes or the acquisition requirement changes, the code is required to be modified to perform the task of re-executing the data, so that the level requirement on technicians is very high and the operation flow is complex; meanwhile, when data acquisition is performed in a hard coding mode, the data has direct operation authority (such as authority for checking, modifying, adding and deleting the database), and if an operator is in error, the risk of false deleting or false modifying of the data in the source database is easily caused due to the fact that the authority is too high, so that the safety of nuclear power data in the source database is seriously affected.
Disclosure of Invention
In view of this, it is necessary to provide a secure configuration method for collecting and writing data from multiple data sources, which can be used for data migration between different access systems, and has strong versatility and high security.
The invention provides a safety configuration method for collecting and writing multiple data sources, which is based on an extensible driving frame and a local system to collect and write nuclear power data among multiple types of data sources, wherein the driving frame and the local system respectively preset multiple driving realization packages and multiple configuration files corresponding to the number of the data sources, and the safety configuration method for collecting and writing the multiple data sources comprises the following steps:
Acquiring and analyzing a data migration request, and determining a data source and task content according to the data migration request, wherein the data comprises a source system data source and/or a target system data source, and the task content comprises main data acquisition of the source system data source; or migrating the main data of the source system data source to the target system data source; or main data of a target system data source is written;
Uploading corresponding driving realization packages and configuration files in a local system according to the corresponding relation among the driving realization packages, the configuration files and the data sources, and setting an API interface module of the driving framework based on the task content, the driving realization packages and the configuration files, wherein the API interface module comprises a first API interface, a third API interface and a fourth API interface;
the first API interface is configured to perform initialization of the drive framework based on a drive instance and a configuration file, and includes:
S21, loading the drive realization package into a memory, and instantiating to form a drive instance;
s22, acquiring the driving instance in a local system, and respectively establishing communication with various access modes in the data source based on the configuration file to form corresponding connection objects, wherein different connection objects form a connection object set of the data source;
s23, assigning the connection object set to the driving framework and packaging the connection object set into a data source operation object, and generating a unique data source identification for the data source operation object;
the second API interface is used for acquiring metadata corresponding to the data source based on the data source identifier;
The third API interface is used for acquiring main data corresponding to the data source based on the data source identification and the metadata;
The fourth API interface is used for writing the main data into the target system data source based on the data source identifier;
and calling the driving instance and the corresponding API interface module to execute the task content.
Preferably, the data source comprises a source system data source, and the task content is to collect main data of the source system data source; then
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
s311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312;
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
s32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
s321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322;
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; and if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data.
Further, S312, which calls the second API interface through the driving framework to perform a metadata obtaining operation to obtain corresponding metadata, includes the following steps:
(1) Acquiring the data source operation object in a source system data source through the data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Inquiring and acquiring metadata from each connection object according to the access mode corresponding to the connection object, and summarizing to form a metadata set;
(3) And returning the metadata set.
Further, S322 performs a main data obtaining operation to obtain main data by calling the third API interface of the data source operation object through the driving framework includes the following operations:
(1) Acquiring the data operation object in a source system data source through the data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Inquiring and acquiring main data from the corresponding connection object according to the metadata set, and summarizing to generate a main data set;
(3) And returning the main data set.
Further, the step S312 obtains a second API interface switching parameter in the configuration file and checks whether the second API interface is implemented according to the parameter value, where a value of the second API interface switching parameter is True indicating that the interface is implemented and False indicating that the interface is not implemented;
and/or, the step S322 obtains a third API interface switching parameter in the configuration file and checks whether the third API interface is implemented according to the parameter value, where a value of the third API interface switching parameter is True indicates that the third API interface is implemented, and False indicates that the third API interface is not implemented.
Preferably, the data source comprises a target system data source, and the task content is to write main data of the target system data source; then
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API module for setting the driving framework comprises a first API interface and a fourth API interface;
s3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31' checking whether the main data need to be written;
s32' calls a fourth API interface through a driving framework to execute main data writing operation, and main data is written into a target data source;
S33' generates a log of write success or failure according to the returned result.
Further, S32' executes a main data write operation by calling a fourth API interface through the driving framework, and writing the main data to the target data source includes the following steps:
(1) Acquiring a data source operation object in a local system according to a data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Checking whether storage positions corresponding to the connection objects in the target system exist or not, and if so, writing corresponding main data into the designated storage positions according to the respective corresponding access modes; if the connection objects do not exist, creating storage positions for the connection objects, and writing the storage positions into the corresponding storage positions;
(3) The results are returned, including the write success or failure status.
Preferably, the data sources include a source system data source and a target system data source, and the task content is to migrate main data of the source system data source to the target system data source, and then the task content includes:
(A) And (3) main data acquisition:
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
S311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312;
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface of the data source operation object through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
s32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
s321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322;
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data;
(B) Main data writing, comprising:
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API module for setting the driving framework comprises a first API interface and a fourth API interface;
s3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31' checking whether the main data need to be written; in this embodiment, it is first checked whether the main data needs to be written, if there is a log record that has been successfully written, then it is not necessary to repeat the writing;
s32' executes a main data writing operation by calling a fourth API interface of the data source through the driving framework, and writes main data into the target data source;
S33' generates a log of write success or failure according to the returned result.
Preferably, the configuration file is an encrypted file, the configuration file is decrypted before the initialization, the configuration file after the decryption is parsed to obtain configuration parameters, and the connection with different connection objects in the data source is established through the configuration parameters.
Preferably, the data source identification is generated based on a map data structure.
Compared with the prior art, the invention has the following beneficial effects:
(1) When data migration is carried out, different access systems in different data sources are combined, collection or writing of the data sources is realized through corresponding drive realization packages and configuration files, plug-in components are more flexible, hard coding is not needed, universality is high, a user can uniformly manage the drive realization packages and the configuration files, addition, deletion, modification and searching of the drive are supported, configuration of the data sources can be completed by combining the drive, and collected or written data sources are provided for migration engineering.
(2) When realizing data migration by driving realization package, packaging different access system combinations on a service level to form four groups of API interfaces, so as to set up, (a) by packaging a data source, configuration information such as IP address, URL address, user name, password and the like is prevented from being exposed to an external system, and safety control of the data access system is realized; (b) Through encapsulation, access control of a data level is realized, and granularity is finer than that of an operation level (addition, deletion and correction); (c) Due to standardized encapsulation, it is possible to multiplex (by modifying the configuration file, for data sources comprising the same combination of access means, it is possible to adapt different data sources based on the same driver implementation package).
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a block diagram of a method for secure configuration of data source acquisition and writing in accordance with the present invention;
FIG. 2 is a schematic workflow diagram of a method for secure configuration of data source acquisition and writing provided by the present invention;
FIG. 3 is a schematic diagram of an initialization flow based on a first API interface;
FIG. 4 is a schematic diagram of a metadata collection flow based on a second API interface;
FIG. 5 is a schematic diagram of a primary data collection flow based on a third API interface;
FIG. 6 is a schematic diagram of a main data write process based on a fourth API interface.
Detailed Description
The following detailed description of preferred embodiments of the application is made in connection with the accompanying drawings, which form a part hereof, and together with the description of the embodiments of the application, are used to explain the principles of the application and are not intended to limit the scope of the application.
When nuclear power data is migrated, the system comprises a source system, a local system and a target system, wherein the source system and the target system respectively represent a data source, the data source can be collected or written, namely, when data is required to be migrated from the data source, the system where the data source is located is used as the source system, when the data is required to be written into the system, the system where the data source is located is used as the target system, and the local system is used as a transfer between the source system and the target system.
Because of the specificity of the nuclear power data, each data source corresponds to a respective access system, and each access system (which may include one or more combinations of databases, HTTP interfaces, object storage services, data files, etc.) correspondingly includes a combination of multiple access modes (i.e., modes for storing and reading data), for example, when the database type in one data source may be one or more of IoTDB, mySQL, oracle, redis, different access modes exist for different types of databases, and meanwhile, if the types of the data files, the object storage services, the HTTP interfaces, etc. in the data source are different, the access modes are also different.
When data is migrated, source systems in different nuclear power stations are not identical, target systems to be written are not identical, so that data access mode combinations of different data sources are also uncertain, and in order to realize convenient and safe data migration between the source systems and the target systems, connection and change are needed to be carried out again one by one in a single coding mode according to different access modes of different data sources when migration is not needed.
As shown in fig. 1 and 2, the present invention provides a method for secure configuration of multiple data source collection and writing, which is based on an extensible driving framework and a local system to collect and write nuclear power data between multiple types of data sources, wherein the driving framework and the local system respectively preset multiple driving implementation packages and multiple configuration files corresponding to the number of the data sources, and the method for secure configuration of multiple data source collection and writing comprises the following steps:
S1, acquiring and analyzing a data migration request, and determining a data source and task content according to the data migration request, wherein the data comprises a source system data source and/or a target system data source, and the task content comprises main data acquisition of the source system data source; or migrating the main data of the source system data source to the target system data source; or main data of a target system data source is written;
S2, uploading a corresponding driving realization package and a corresponding configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the data source, and setting an API interface module of the driving framework based on the task content, the driving realization package and the configuration file, wherein the API interface module comprises a first API interface, a third API interface and a fourth API interface;
the first API interface is configured to perform initialization based on a driving instance and a configuration file, and includes:
S21, loading the drive realization package into a memory, and instantiating to form a drive instance;
s22, acquiring the driving instance in a local system, and respectively establishing communication with various access modes in the data source based on the configuration file to form corresponding connection objects, wherein different connection objects form a connection object set of the data source;
s23, assigning the connection object set to the driving framework and packaging the connection object set into a data source operation object, and generating a unique data source identification for the data source operation object;
the second API interface is used for acquiring metadata corresponding to the data source based on the data source identifier;
The third API interface is used for acquiring main data corresponding to the data source based on the data source identification and the metadata;
The fourth API interface is to write the primary data to the target system data source based on the data source identification.
S3, calling the driving instance and the API interface module to execute the task content.
The invention discloses a safety configuration method for collecting and writing multiple data sources, wherein one data source corresponds to a drive realization packet and a configuration file, the drive realization packet is a drive file (corresponding to a storage system of the data source and comprising connection, reading and writing and the like), the drive file realizes a defined API interface module, a first API interface is used for informing a local system how to establish communication with the data source, and a second API interface is used for informing the local system what data (metadata) is in the source system to be migrated; the third API interface enables the local system to collect the data according to the metadata (namely the data description); the fourth API interface is used for the local system to write the collected data into a data source in the target system.
When a data migration request is received, firstly analyzing a source system and/or a target system, respectively confirming a corresponding driving realization packet and a corresponding configuration file according to the one-to-one correspondence between a data source and a driving realization packet and the configuration file, loading the corresponding driving realization packet into a memory for instantiation, simultaneously, when the configuration file is used for initializing a first API interface of a driving instance, respectively establishing communication with a plurality of access modes in the corresponding data source, each communication is established with one access mode, forming a connection object, different connection objects of the plurality of access modes form a connection object set, the connection object set is returned to an operation object of the driving frame and then packaged into a data source, generating a unique data source identification, subsequently acquiring the data source operation object in the local system according to the data source identification according to the different task requirements, and calling the data source operation object through the driving frame to realize the completion of corresponding task content, and when a data source is newly added, simultaneously, adding the corresponding driving realization packet in the driving frame, and simultaneously, correspondingly adding the configuration file in the local system to realize the subsequent data source and/or the subsequent data acquisition operation; when the data source is changed or changed, an operator only needs to correspondingly adjust the driving realization package and the configuration file, and then the data of different types of data sources can be collected and written through the shared API interface, so that the universality is strong and the operation is convenient.
In a preferred embodiment, the data source identifier is in a map (an unordered key-value-based data structure), as shown in table 1, and the unique identifier ds_1 represents a connection object L1 of an access manner, and the local system performs subsequent API interface operations through the data source operation object.
TABLE 1
| Key | Value |
| Data source unique identifier DS_1 | Connection object L1 |
| Data source unique identifier DS_2 | Connection object L2 |
| Data source unique identifier DS_3 | Connection object L3 |
| …… | …… |
As mentioned above, during nuclear power data migration, the task content includes three kinds of following: (1) performing main data acquisition of a source system data source; (2) Migrating primary data of a source system data source to a target system data source; (3) The main data of the data source of the target system is written, the task content is different, the configuration of the corresponding API interface module, the specific implementation steps of calling the driving instance and executing the task content by the API interface module are also different, and the safety configuration method for collecting and writing the data source shown in the application is respectively described below in connection with different task contents.
In the first embodiment, when the task content analyzed in S1 is main data acquisition of a source system data source and the data source is the source system data source, the security configuration method for acquiring and writing multiple data sources shown in the invention further includes the following steps:
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API interface module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
In a preferred embodiment, the configuration file is an encrypted file, the configuration file further includes a decryption configuration file before the initialization, and the configuration file after the decryption is parsed to obtain configuration parameters, so as to establish connection of different access modes in the data source through the configuration parameters, so as shown in fig. 3, the step of setting the first API interface specifically includes the following steps:
(1) Decrypting the configuration file;
(2) Analyzing the configuration file to obtain configuration parameters;
(3) Establishing different access mode connections in the data source through configuration parameters, and obtaining each connection object in the data source;
(4) And returning the connection object set of the data source.
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
S311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312; if the corresponding metadata file already exists in the local system, metadata does not need to be acquired from the source system.
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface of the data source operation object through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
In this step, a second API interface switching parameter in the configuration file is obtained and whether the second API interface is implemented is checked according to the parameter value, where a value of the second API interface switching parameter is True indicates that the interface is implemented, and False indicates that the interface is not implemented.
In a preferred embodiment, in this step, the step of calling the second API interface of the call data source operation object through the driving framework to perform a metadata obtaining operation to obtain corresponding metadata includes the following steps:
(1) Acquiring the data source operation object in a source system data source through the data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Inquiring and acquiring metadata from each connection object according to the access mode corresponding to the connection object, and performing metadata processing to form a metadata set;
Taking IoTDB databases and MySQL databases included in a source system data source as an example, as shown in fig. 4, for the data in the IoTDB databases, by calling IoTDB a time series query, the MySQL table query is called for the nuclear power data of the MySQL database, and the nuclear power data is converted into a metadata set after being processed (for example, map < String, inter > structure metadata set is generated, key is a measurement point name, and value is a data type).
(3) And returning the metadata set.
S32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
S321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322; in this step, it is first checked whether the local system has a master data file, i.e. a log record has been successfully acquired, and if so, repeated acquisition is not required.
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; and if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data. In this step, a third API interface switching parameter in the configuration file is obtained, and whether the third API interface is implemented is checked according to the parameter value, where a value of the third API interface switching parameter is True indicates that the interface is implemented, and False indicates that the interface is not implemented.
Further, invoking the third API interface of the data source operation object through the driving framework to perform the main data obtaining operation to obtain the main data includes the following operations:
(1) Acquiring the data operation object in a source system data source through the data source identifier; acquiring all the connection objects according to the data operation objects;
(2) Inquiring and acquiring main data from the corresponding connection object according to the metadata set, and then processing the main data to generate a main data set;
As shown in fig. 5, based on the metadata set, the time sequence query and the MySQL table query are respectively invoked IoTDB in this step, which corresponds to the data set that respectively generates the quality, timestamp, value set, list < Object > of the List < Map < String, object > > structure. Wherein IoTDB, as shown in Table 2, an example of an element Map < String, object > in the List represents a data record of a measurement point:
TABLE 2
In MySQL database, as shown in table 3, an example of an element Map < String, object > in List represents a data record of a table:
TABLE 3 Table 3
(3) And returning the main data set.
In the second embodiment, when the task content analyzed in S1 is to write main data of the target system data source and the data source is the target system data source, the security configuration method for collecting and writing the data source according to the present invention further includes the following steps:
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API interface module for setting the driving framework comprises a first API interface and a fourth API interface; the first API interface is set as described above, and will not be described herein.
S3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31' checking whether the main data need to be written; in this embodiment, it is first checked whether the main data needs to be written, if there is a log record that has been successfully written, then it is not necessary to repeat the writing;
s32' executes a main data writing operation by calling a fourth API interface of the data source through the driving framework, and writes main data into the target data source;
As a preferred scheme, S32' executes a main data write operation by calling a fourth API interface of the data source through the driving framework, and writing the main data to the target data source includes the steps of:
(1) Acquiring a data source operation object in a local system according to the data source identification,
(2) Respectively acquiring each connection object included in the data source operation object;
Checking whether storage positions corresponding to all connection objects in a target system exist or not, and if so, writing corresponding main data into the designated storage positions according to respective access modes; if the connection object does not exist, the corresponding storage position is written after the storage position is created for each connection object.
As shown in fig. 6, in this step, firstly, whether a IoTDB time sequence and MySQL table exist or not is determined, if so, then, a IoTDB write statement and an MySQL Insert write statement are respectively called, and data with a List < Map < String, object > structure is written into a target system (e.g., ioTDB returns to Boolean as a state); if not, a IoTDB time series or MySQL table is created and written.
(3) The results are returned, including the write success or failure status.
S33' generates a log of write success or failure according to the returned result.
In a third embodiment, when the task content analyzed in S1 is to migrate main data of a source system data source to a target system data source, the data source includes the source system data source and the target system data source. Because of the need of data migration, two data sources exist, namely a source system data source and a target data source, the invention discloses a safety configuration method for collecting and writing in multiple data sources, firstly, after a main data acquisition process is executed on a source system data source according to the method shown in the first embodiment, then, a main data writing process is executed on a target system data source according to the method shown in the second embodiment, so that data migration can be completed, and the method specifically comprises the following steps:
(A) And (3) main data acquisition:
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
S311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312;
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface of the data source operation object through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
s32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
S321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322;
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; and if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data.
After the main data acquisition step is completed, deleting the corresponding data source and unloading the corresponding drive realization packet, and then turning to the main data writing in step (B), wherein the main data writing in step (B) comprises the following steps:
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API module for setting the driving framework comprises a first API interface and a fourth API interface;
s3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31' checking whether the main data need to be written; in this embodiment, it is first checked whether the main data needs to be written, if there is a log record that has been successfully written, then it is not necessary to repeat the writing;
s32' executes a main data writing operation by calling a fourth API interface of the data source through the driving framework, and writes main data into the target data source;
S33' generates a log of write success or failure according to the returned result.
While the invention has been described with respect to the preferred embodiments, the scope of the invention is not limited thereto, and any changes or substitutions that would be apparent to those skilled in the art are intended to be included within the scope of the invention.
Claims (8)
1. A safety configuration method for collecting and writing in multiple data sources is characterized in that: the acquisition and writing of nuclear power data is performed between multiple types of data sources based on an extensible drive framework and a local system, the driving framework and the local system respectively preset a plurality of driving realization packages and a plurality of configuration files corresponding to the number of the data sources, and the safety configuration method for collecting and writing the multiple data sources comprises the following steps:
Acquiring and analyzing a data migration request, and determining a data source and task content according to the data migration request, wherein the data source comprises a source system data source and/or a target system data source, and the task content comprises main data acquisition of the source system data source; or migrating the main data of the source system data source to the target system data source; or main data of a target system data source is written;
Uploading corresponding driving realization packages and configuration files in a local system according to the corresponding relation among the driving realization packages, the configuration files and the data sources, and setting an API interface module of the driving framework based on the task content, the driving realization packages and the configuration files, wherein the API interface module comprises a first API interface, a second API interface, a third API interface and a fourth API interface;
the first API interface is configured to perform initialization of the drive framework based on a drive instance and a configuration file, and includes:
S21, loading the drive realization package into a memory, and instantiating to form a drive instance;
s22, acquiring the driving instance in a local system, and respectively establishing communication with various access modes in the data source based on the configuration file to form corresponding connection objects, wherein different connection objects form a connection object set of the data source;
s23, assigning the connection object set to the driving framework and packaging the connection object set into a data source operation object, and generating a unique data source identification for the data source operation object;
the second API interface is used for acquiring metadata corresponding to the data source based on the data source identifier;
The third API interface is used for acquiring main data corresponding to the data source based on the data source identification and the metadata;
The fourth API interface is used for writing the main data into the target system data source based on the data source identifier;
Calling the driving instance and the corresponding API interface module to execute the task content;
the data source comprises a source system data source, and the task content is to collect main data of the source system data source
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
s311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312;
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
s32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
s321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322;
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data;
and/or the data source is a target system data source, and the task content is the main data writing of the target system data source; then
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API module for setting the driving framework comprises a first API interface and a fourth API interface;
s3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31' checking whether the main data need to be written;
s32' calls a fourth API interface through a driving framework to execute main data writing operation, and main data is written into a target data source;
S33' generates a log of write success or failure according to the returned result.
2. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: s312, calling the second API interface through the driving framework to execute metadata acquisition operation to acquire corresponding metadata comprises the following steps:
(1) Acquiring the data source operation object in a source system data source through the data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Inquiring and acquiring metadata from each connection object according to the access mode corresponding to the connection object, and summarizing to form a metadata set;
(3) And returning the metadata set.
3. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: s322, calling a third API interface of the data source operation object through the driving framework to execute a main data acquisition operation to acquire main data comprises the following operations:
(1) Acquiring the data operation object in a source system data source through the data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Inquiring and acquiring main data from the corresponding connection object according to the metadata set, and summarizing to generate a main data set;
(3) And returning the main data set.
4. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: the step S312 is to acquire a second API interface switch parameter in the configuration file and check whether the second API interface is realized according to the parameter value, wherein the fact that the second API interface switch parameter value is True indicates that the interface is realized, and False indicates that the interface is not realized;
and/or, the step S322 obtains a third API interface switching parameter in the configuration file and checks whether the third API interface is implemented according to the parameter value, where a value of the third API interface switching parameter is True indicates that the third API interface is implemented, and False indicates that the third API interface is not implemented.
5. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: s32' calls a fourth API interface through a drive framework to execute a main data writing operation, and writing main data into a target data source comprises the following steps:
(1) Acquiring a data source operation object in a local system according to a data source identifier, and acquiring all the connection objects according to the data operation object;
(2) Checking whether storage positions corresponding to the connection objects in the target system exist or not, and if so, writing corresponding main data into the designated storage positions according to the respective corresponding access modes; if the connection objects do not exist, creating storage positions for the connection objects, and writing the storage positions into the corresponding storage positions;
(3) The results are returned, including the write success or failure status.
6. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: the data sources comprise a source system data source and a target system data source, and the task content is that main data of the source system data source is migrated to the target system data source, then
(A) And (3) main data acquisition:
S2, uploading the corresponding driving realization package and configuration file in a local system respectively according to the corresponding relation among the driving realization package, the configuration file and the source system data source, wherein the API module for setting the driving framework comprises a first API interface, a second API interface and a third API interface;
S3, the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
S31, acquiring metadata, comprising the following steps:
S311, checking whether metadata corresponding to the source system data source in the local system exists, if so, turning to step S32, and if not, turning to step S312;
S312, checking whether the second API interface is realized, if not, directly importing a metadata file corresponding to the source system data source into a local system; if the second API interface is realized, calling the second API interface of the data source operation object through the driving framework to execute metadata acquisition operation so as to acquire corresponding metadata;
s32, acquiring main data from the source system data source according to the metadata, wherein the method comprises the following steps of:
s321, checking whether a main data file exists in the local system, if so, ending the flow, and if not, turning to step S322;
S322, checking whether the third API interface is implemented, if not, directly importing a main data file into a local system; if so, calling the third API interface of the data source operation object through a driving framework to execute main data acquisition operation so as to acquire main data;
(B) Main data writing, comprising:
S2' respectively uploading the corresponding driving realization package and configuration file in a local system according to the corresponding relation among the driving realization package, the configuration file and the target system data source, wherein the API module for setting the driving framework comprises a first API interface and a fourth API interface;
s3', the local system calls the driving instance and the API interface module to execute the task content, and the method comprises the following steps:
s31' checking whether the main data need to be written; in this embodiment, it is first checked whether the main data needs to be written, if there is a log record that has been successfully written, then it is not necessary to repeat the writing
S32' executes a main data writing operation by calling a fourth API interface of the data source through the driving framework, and writes main data into the target data source;
S33' generates a log of write success or failure according to the returned result.
7. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: the configuration file is an encrypted file, the configuration file is decrypted before initialization, the decrypted configuration file is analyzed to obtain configuration parameters, and connection with different connection objects in a data source is established through the configuration parameters.
8. A method of security configuration for data source acquisition and writing as claimed in claim 1, wherein: the data source identification is generated based on a map data structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410465038.XA CN118092811B (en) | 2024-04-18 | 2024-04-18 | Safety configuration method for collecting and writing multiple data sources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410465038.XA CN118092811B (en) | 2024-04-18 | 2024-04-18 | Safety configuration method for collecting and writing multiple data sources |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118092811A CN118092811A (en) | 2024-05-28 |
| CN118092811B true CN118092811B (en) | 2024-08-16 |
Family
ID=91159957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410465038.XA Active CN118092811B (en) | 2024-04-18 | 2024-04-18 | Safety configuration method for collecting and writing multiple data sources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118092811B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268275A (en) * | 2021-07-19 | 2021-08-17 | 成都菁蓉联创科技有限公司 | Hardware equipment driving system based on microkernel and driving method thereof |
| CN117591497A (en) * | 2024-01-18 | 2024-02-23 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955712A (en) * | 2019-11-26 | 2020-04-03 | 北京天元创新科技有限公司 | Development API processing method and device based on multiple data sources |
| EP3974960B1 (en) * | 2020-09-29 | 2024-05-01 | Tata Consultancy Services Limited | Method and system for automated testing of web service apis |
| US20230064313A1 (en) * | 2021-08-30 | 2023-03-02 | Quanta Cloud Technology Inc. | Method and system for consistent api application to heterogeneous bare metal servers |
| CN116527572A (en) * | 2023-04-19 | 2023-08-01 | 浙江理工大学 | Multipath data transmission system based on 5G |
-
2024
- 2024-04-18 CN CN202410465038.XA patent/CN118092811B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268275A (en) * | 2021-07-19 | 2021-08-17 | 成都菁蓉联创科技有限公司 | Hardware equipment driving system based on microkernel and driving method thereof |
| CN117591497A (en) * | 2024-01-18 | 2024-02-23 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118092811A (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110968509B (en) | Method and system for batch customizing of variables | |
| CN106095678B (en) | The automation result inspection method operated for data bank service under windows platform | |
| CN112035344A (en) | Multi-scenario test method, device, equipment and computer readable storage medium | |
| CN111737227A (en) | Data modification method and system | |
| CN116719799A (en) | Environment-friendly data management method, device, computer equipment and storage medium | |
| CN109657167B (en) | Data acquisition method, device, server and storage medium | |
| CN117407369B (en) | Operation log recording method and device and electronic equipment | |
| CN116560801B (en) | Cross-container counter system credit migration method and equipment | |
| CN118092811B (en) | Safety configuration method for collecting and writing multiple data sources | |
| CN113032263A (en) | Case test processing method and device, server and readable storage medium | |
| CN105988785A (en) | RPC (remote procedure call) service development method and device | |
| CN108763127B (en) | Method for realizing Modbus adapter capable of realizing interconversion of source data and target data | |
| CN112069158A (en) | Data restoration method, device, equipment and storage medium | |
| CN111694724A (en) | Testing method and device of distributed table system, electronic equipment and storage medium | |
| CN113468446A (en) | Method, system and equipment for supporting identification of third-party two-dimensional code data | |
| CN111639936A (en) | Transaction information acquisition method and device, electronic equipment and readable storage medium | |
| CN111538651A (en) | Interface testing method, device, server and storage medium | |
| CN114900531B (en) | Data synchronization method, device and system | |
| CN112860780B (en) | Data export method and device and terminal equipment | |
| CN111767330B (en) | Data interaction method and device, electronic equipment and storage medium | |
| CN112347095B (en) | Data table processing method, device and server | |
| CN116720818B (en) | Warehouse processing method, device and computer equipment based on block chain | |
| CN110659259B (en) | Database migration method, server and computer storage medium | |
| WO2023230943A1 (en) | System and method of data management | |
| CN117873446A (en) | Worker link tracking method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |