CN113220659A - Data migration method, system, electronic device and storage medium - Google Patents
Data migration method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN113220659A CN113220659A CN202110398368.8A CN202110398368A CN113220659A CN 113220659 A CN113220659 A CN 113220659A CN 202110398368 A CN202110398368 A CN 202110398368A CN 113220659 A CN113220659 A CN 113220659A
- Authority
- CN
- China
- Prior art keywords
- data
- uuid
- csv file
- field
- connecting edge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013508 migration Methods 0.000 title claims abstract description 34
- 230000005012 migration Effects 0.000 title claims abstract description 34
- 238000013507 mapping Methods 0.000 claims abstract description 55
- 238000006243 chemical reaction Methods 0.000 claims abstract description 5
- 238000012216 screening Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012545 processing Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a method, a system, an electronic device and a storage medium for data migration, wherein the method comprises the following steps: acquiring data table structure information in a relational database, and performing corresponding conversion on the data table structure information according to a mapping relation table; then, the data is screened, and a new global UUID is setn(ii) a According to id of data in data tableoFrom a library of mapping relations MDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSet to the data, if not, will global UUIDnSet to the data and store to MDBPerforming the following steps; and finally, storing the data after the id setting is finished in batches as a csv file of the node, reading the csv file of the node after all the data are stored as the csv file, and performing M according to the associated fields of the dataDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file, thereby realizing the purpose of migrating the relational database data to a database.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method, a system, an electronic device, and a storage medium for data migration.
Background
With the continuous development of network application forms, more and more relational data tables need to be created to realize richer and richer network applications, the data tables are not only linked but also independent, and when an application program needs to simultaneously acquire related data in different data tables through combined query, a huge performance bottleneck can be faced. The advent of non-relational databases such as JanusGraph (extensible distributed graph database) has solved the problem of simultaneous storage of multiple data source data. However, in the existing tools, there is no tool or method for directly importing data from a relational database to a database such as janussgraph.
In the related art, data migration is realized based on message middleware, however, the data volume of each transmission of the data migration is too small, which is not beneficial to data generation when a graph database is initially established, and is also not beneficial to establishing a link relation between data. In addition, in some related arts, data migration is performed between similar databases, and the purpose of migrating data in an existing relational database to a graph database is not achieved.
At present, no effective solution is provided for the problem that a method for directly importing data from a relational database to a graph database is lacked in the related art.
Disclosure of Invention
Embodiments of the present application provide a method, a system, an electronic device, and a storage medium for data migration, so as to solve at least the problem in the related art that a method for importing data from a relational database directly to a graph database is lacking.
In a first aspect, an embodiment of the present application provides a method for data migration, where the method includes:
acquiring data table structure information in a relational database, and performing corresponding conversion on the data table structure information according to a mapping relation table;
screening data in the data table structure information, and setting a new global UUIDN for the screened data;
according to the id of the data in the data tableoMapping the relational library M from idDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the global UUID to replace the datanIf not, the global UUID is usednSetting the data to the mapping relation library MDBPerforming the following steps;
storing the data after the id setting is finished into a node csv file in batches, reading the node csv file after all the data are stored into the csv file, and according to the associated fields of the data, storing the data into MDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
In some embodiments, the correspondingly converting the data table structure information according to the mapping relation table includes:
converting information in the relational database into corresponding graph database information through the mapping relation table, wherein the information in the relational database comprises: field name, mapping relation of attribute name, name of associated field in data table, field name of primary key in data table, id in data tableoThe name of the field.
In some embodiments, the filtering the data in the data table structure information includes:
clearing format error values and special characters in the data;
screening data which can be stored in a graph database according to screening conditions, wherein the screening conditions comprise whether the field type of the data accords with the field type of the data in the graph database.
In some of these embodiments, the association field according to the dataAt MDBSearching for the UUID of the associated data corresponding to the associated field, and establishing the connection edge comprises the following steps:
according to id in the csv file of the nodeoThe value of the field is extracted from the mapping relation library MDBQuerying to obtain a first UUID;
obtaining the id of the associated data corresponding to the associated fieldoAccording to said idoFrom the mapping relation library MDBObtaining a second UUID of the record;
and generating a connecting edge according to the first UUID and the second UUID, and storing the connecting edge into a connecting edge csv file.
In some embodiments, the setting of the new global UUID for the filtered datanThe method comprises the following steps:
the global UUIDnGenerating different global UUIDs according to time generation and time variationn。
In a second aspect, an embodiment of the present application provides a system for data migration, where the system includes:
a data processing module for obtaining the data table structure information in the relational database and correspondingly converting the data table structure information according to the mapping relation table,
screening the data in the data table structure information, and setting a new global UUID for the screened datan；
An ID setting module for setting ID of data in the data tableoMapping the relational library M from idDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the global UUID to replace the datanIf not, the global UUID is usednSetting the data to the mapping relation library MDBPerforming the following steps;
the edge connection establishing module is used for storing the data after the id setting is finished into a node csv file in batches, reading the node csv file after all the data are stored into the csv file, and performing M according to the associated fields of the dataDBWhere the search isAnd establishing a connecting edge by the UUID of the associated data corresponding to the associated field, and storing the connecting edge as a connecting edge csv file.
In some embodiments, the data processing module is further configured to convert information in the relational database into corresponding graph database information through the mapping relationship table, where the information in the relational database includes: field name, mapping relation of attribute name, name of associated field in data table, field name of primary key in data table, id in data tableoThe name of the field.
In some embodiments, the edge connection establishing module is further configured to establish an edge connection between the node csv file and the node csv file according to id in the node csv fileoThe value of the field is extracted from the mapping relation library MDBQuerying to obtain a first UUID;
obtaining the id of the associated data corresponding to the associated fieldoAccording to said idoFrom the mapping relation library MDBObtaining a second UUID of the record;
and generating a connecting edge according to the first UUID and the second UUID, and storing the connecting edge into a connecting edge csv file.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method for data migration as described in the first aspect above is implemented.
In a fourth aspect, the present application provides a storage medium, on which a computer program is stored, where the program is executed by a processor to implement the method for data migration as described in the first aspect.
Compared with the related art, the data migration method provided by the embodiment of the application acquires the structure information of the data table in the relational database, and correspondingly converts the structure information of the data table according to the mapping relation table; then, screening the data in the data table structure information, and setting a new global UUID for the screened datan(ii) a For the screened data, according to the id of the data in the data tableoMapping a relational library from idMDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the data to replace the global UUIDnIf not, the generated global UUIDnIs set to the data and stored in a mapping relation library MDBPerforming the following steps; and finally, storing the data after the id setting is finished in batches as a csv file of the node, reading the csv file of the node after all the data are stored as the csv file, and performing M according to the associated fields of the dataDBThe UUID of the associated data corresponding to the associated field is searched, the connecting edge is established, and the connecting edge is stored as a connecting edge csv file, so that the problem that a method for directly importing data from a relational database to a database is lacked in the related technology is solved. Compared with the prior art, the method has the advantages that the graph database is not imported, the data to be migrated are not correspondingly processed aiming at the target database, the globally unique id is not formed in the data, and the problem of data repetition in the graph database cannot be solved. The application provides a data migration method for distributing unified UUIDs for global data and carrying out data formatting, a data file suitable for a graph database is generated according to data and a structure in an existing relational database, particularly the data file of a Janus graph database, the purpose of migrating the data of the relational database to the graph database is achieved, unique UUIDs in the graph database are given to each piece of data through unique ids of the database, the UUIDs are used for retrieval and modification in the graph database, and data repetition in the graph data is prevented.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow diagram of a method of data migration according to an embodiment of the present application;
FIG. 2 is a block diagram of a system for data migration according to an embodiment of the present application;
FIG. 3 is a data processing flow diagram of a method of data migration according to an embodiment of the present application;
fig. 4 is an internal structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
The present application provides a method of data migration, and fig. 1 is a flowchart of a method of data migration according to an embodiment of the present application, and as shown in fig. 1, the flowchart includes the following steps:
step S101, obtaining the structure information of a data table in a relational database, and carrying out corresponding conversion on the structure information of the data table according to a mapping relation table; optionally, in this embodiment, the structural information SCHEMA of the data table in the relational database is read, and the mapping relation table M is used to determine the mapping relationtConverting information in the relational database into corresponding graph database information, wherein the information in the relational database comprises: the mapping relation between the field name of the JanusGraph and the attribute name in the JanusGraph, the name of the associated field in the data table, the field name of the primary key of the data table and the unique id in the data table are required to be storedoThe name of the field.
Step S102, screening the data in the data table structure information, and setting a new global UUID for the screened datan；
According to the embodiment, all data of each table are read from the relational database according to the obtained data table structure information SCHEMA, and the obtained data is formatted, cleaned and screened according to the content in SCHEMA. Preferably, the screening the data in the data table structure information in this embodiment includes: clearing format error values and special characters in the data; screening data which can be stored in a graph database according to screening conditions, wherein the screening conditions comprise whether the field type of the data accords with the field type of the data in the graph database, if so, retaining, otherwise, if not, processing and converting the field type of the data into the field type which accords with the data in the graph database.
After finishing the sorting, cleaning and screening of the data, setting a new global UUID for the datanThe UUIDnA globally uniformly assigned and unique data ID, and a set global UUIDnIs generated according to time, and different global UUIDs are generated according to the change of timen. Thus, UUIDnThe ID is not repeated, and the stored data can be ensured not to be repeated.
Step S103, according to the id of the data in the data tableoMapping the relational library M from idDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting data to replace the global UUIDnIf not, the global UUID is usednSetting the data to the mapping relation library MDBPerforming the following steps;
in this embodiment, the id mapping relation library MDBId stored with data in relational databaseoAs a key, and a key-value pair having a global UUID and a data type as values, e.g., [ (globally unique UUID, data type)]. The data type in the key value pair refers to that the data is recorded in a data table where the original relational database is located. MDBThe database is to prevent the same id from different data tablesoAppears in the graph database, so id is setoMapping to UUID and storing in MDBIn (1). From which id is queriedoThe steps of the corresponding relation with the UUID are as follows:
s1: according to the unique id of the data in the data tableoAs a key from MDBQuerying whether there is a unique global UUIDoCorresponding to it;
s2: if there is a unique global UUIDoThen the global UUID is appliedoSet to the data
S3: if there is no unique global UUIDoThen a global UUID will be generatednIs set to the data and stored in a mapping relation library MDBIn (1).
In this embodiment, a one-to-one mapping relation query library M is established between the original id and the key value pair (i.e., (UUID, field type) in the data table through the above stepsDBQuery the library based on MDBThe mapping relation query library can ensure that the unique UUID of the database is not changed when the data is updated, so that the consistency of the data is ensured.
Step S104, storing the data after the id setting is finished as a node csv file in batches, reading the node csv file after all the data are stored as the csv file, and performing M according to the associated fields of the dataDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
Optionally, data records in the csv file of the node are read, and according to id in the read csv fileoThe value of the field is mapped from the relational library MDBQuerying to obtain a first UUID; then, the id of the associated data corresponding to the associated field is obtainedoAccording to idoFrom a library of mapping relations MDBObtaining a second UUID of the record; and finally, generating a connecting edge by the inquired first UUID and the recorded second UUID, and storing the connecting edge as a connecting edge csv file.
The present embodiment depends on the id of the associated fieldoAnd MDBAnd the relational database is mapped, so that a link relation is established between the relational database and the graph database, and data migration is facilitated.
Through the steps S101 to S104, in this embodiment, first, structural information of data in a required data table is acquired from a relational database, then, the data field name is mapped to a type allowed in janussgraph, then, the data is further cleaned according to a type requirement in the database, and then, a globally unique UUID is allocated to the cleaned data. When all data are marked with globally unique UUIDs, the UUIDs of the associated data are further acquired according to the associated field records in the data, so that a link relation is established between the graph database JanusGraph and the relational database, the problem that a method for directly importing data from the relational database to the graph database is lacked in the related technology is solved, the purpose of transferring data of the relational database to the graph database is achieved, the unique UUIDs in the graph database are given to each piece of data through the unique ids of the database, the UUIDs are used in the graph database for retrieval and modification, and data repetition in the graph data is prevented.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
The present embodiment further provides a system for data migration, where the system is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated for what has been described. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 2 is a block diagram of a system for data migration according to an embodiment of the present application, and as shown in fig. 2, the system includes a data processing module 21, an ID setting module 22, and a connection edge establishing module 23:
a data processing module 21, configured to obtain the structure information of the data table in the relational database, perform corresponding conversion on the structure information of the data table according to the mapping relation table, screen the data in the structure information of the data table, and set a new global UUID for the screened datan(ii) a An ID setting module 22 for setting the ID according to the data in the data tableoMapping the relational library M from idDBWhether the inquiry is storedAt a unique global UUIDoIf so, the global UUID is addedoSetting the data to replace the global UUIDnIf not, the global UUID is usednIs set to the data and stored in a mapping relation library MDBPerforming the following steps; the edge connection establishing module 23 is configured to store the data with the id set as the node csv file in batch, read the node csv file after all the data are stored as the csv file, and perform M according to the associated field of the dataDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
Through the system, the data processing module 21 acquires the data in the relational database, arranges, cleans and screens the data, and sets a new global UUID generated according to time for the datanEnsuring that the stored data is not repeated; the ID setting module 22 establishes a one-to-one mapping relation between the original ID and the key value pair (UUID, field type) in the data table, and queries the library MDBQuery the library based on MDBThe mapping relation query library can ensure that the unique UUID of the database is not changed when the data is updated, so that the consistency of the data is ensured; the edge connection establishing module 23 establishes the id according to the associated fieldoAnd MDBAnd the relational database is mapped, so that a link relation is established between the relational database and the graph database, and data migration is facilitated. The whole system solves the problem that a method for directly importing data from a relational database to a database is absent in the related technology, achieves the purpose of migrating the data of the relational database to the database, gives a unique UUID in the database for each piece of data through the unique id of the database, and uses the UUID to retrieve and modify in the database, thereby preventing the data in the graph data from being repeated.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
The present invention will be described in detail with reference to the following application scenarios.
The invention aims to provide a data migration method for distributing unified UUIDs to global data and performing data formatting processing, and fig. 3 is a data processing flow chart of the data migration method according to the embodiment of the application, as shown in fig. 3. The flow steps of the technical scheme of the data migration method in the embodiment include:
s1, reading the structure information SCHEMA of the data table in the relational database;
s2, according to the mapping relation table Mt, the structure information of the data table is converted correspondingly;
s3, reading all data of each table from the relational database according to the obtained SCHEMA;
s4, formatting, cleaning and screening the acquired data according to the content in SCHEMA;
s5, assigning a new globally unique UUID to all cleaned datan；
S6, according to the id of the data in each data tableoMapping the relational library M from idDBIn-process lookup whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the data to replace the global UUIDnIf not, the generated global UUID is usednIs set to the data and stored in a mapping relation library MDBPerforming the following steps;
s7, storing the data with the set id in batches as a csv file;
s8, after all data are stored as csv file, reading node csv file, according to data correlation field, at MDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.
In addition, in combination with the method for data migration in the foregoing embodiments, the embodiments of the present application may provide a storage medium to implement. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements the method of data migration of any of the above embodiments.
In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data migration. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
In one embodiment, fig. 4 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 4, there is provided an electronic device, which may be a server, and its internal structure diagram may be as shown in fig. 4. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used for providing calculation and control capability, the network interface is used for communicating with an external terminal through network connection, the internal memory is used for providing an environment for an operating system and the running of a computer program, the computer program is executed by the processor to realize a data migration method, and the database is used for storing data.
Those skilled in the art will appreciate that the configuration shown in fig. 4 is a block diagram of only a portion of the configuration associated with the present application, and does not constitute a limitation on the electronic device to which the present application is applied, and a particular electronic device may include more or less components than those shown in the drawings, or combine certain components, or have a different arrangement of components.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data migration, the method comprising:
acquiring data table structure information in a relational database, and performing corresponding conversion on the data table structure information according to a mapping relation table;
screening the data in the data table structure information, and setting a new global UUID for the screened datan；
According to the id of the data in the data tableoMapping the relational library M from idDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the global UUID to replace the datanIf not, the global UUID is usednSetting the data to the mapping relation library MDBPerforming the following steps;
storing the data after the id setting is finished into a node csv file in batches, reading the node csv file after all the data are stored into the csv file, and according to the associated fields of the data, storing the data into MDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
2. The method according to claim 1, wherein said correspondingly converting the data table structure information according to the mapping relation table comprises:
converting information in the relational database into corresponding graph database information through the mapping relation table, wherein the information in the relational database comprises: field name, mapping relation of attribute name, name of associated field in data table, field name of primary key in data table, id in data tableoThe name of the field.
3. The method of claim 1, wherein the screening the data in the data table structure information comprises:
clearing format error values and special characters in the data;
screening data which can be stored in a graph database according to screening conditions, wherein the screening conditions comprise whether the field type of the data accords with the field type of the data in the graph database.
4. The method of claim 1, wherein said associating is performed at M according to said data's associated fieldDBSearching for the UUID of the associated data corresponding to the associated field, and establishing the connection edge comprises the following steps:
according to id in the csv file of the nodeoThe value of the field is extracted from the mapping relation library MDBQuerying to obtain a first UUID;
obtaining the id of the associated data corresponding to the associated fieldoAccording to said idoFrom the mapping relation library MDBObtaining a second UUID of the record;
and generating a connecting edge according to the first UUID and the second UUID, and storing the connecting edge into a connecting edge csv file.
5. The method of claim 1, wherein setting a new global UUID for the filtered datanThe method comprises the following steps:
the global UUIDnGenerated differently according to time, with variation of timeGlobal UUIDn。
6. A system for data migration, the system comprising:
a data processing module for obtaining the data table structure information in the relational database and correspondingly converting the data table structure information according to the mapping relation table,
screening the data in the data table structure information, and setting a new global UUID for the screened datan；
An ID setting module for setting ID of data in the data tableoMapping the relational library M from idDBQuerying whether there is a unique global UUIDoIf so, the global UUID is addedoSetting the global UUID to replace the datanIf not, the global UUID is usednSetting the data to the mapping relation library MDBPerforming the following steps;
the edge connection establishing module is used for storing the data after the id setting is finished into a node csv file in batches, reading the node csv file after all the data are stored into the csv file, and performing M according to the associated fields of the dataDBAnd searching for the UUID of the associated data corresponding to the associated field, establishing a connecting edge, and storing the connecting edge as a connecting edge csv file.
7. The system of claim 6,
the data processing module is further configured to convert information in the relational database into corresponding graph database information through the mapping relationship table, where the information in the relational database includes: field name, mapping relation of attribute name, name of associated field in data table, field name of primary key in data table, id in data tableoThe name of the field.
8. The system of claim 6,
the above-mentionedThe edge connection establishing module is also used for establishing an edge connection according to id in the csv file of the nodeoThe value of the field is extracted from the mapping relation library MDBQuerying to obtain a first UUID;
obtaining the id of the associated data corresponding to the associated fieldoAccording to said idoFrom the mapping relation library MDBObtaining a second UUID of the record;
and generating a connecting edge according to the first UUID and the second UUID, and storing the connecting edge into a connecting edge csv file.
9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is configured to execute the computer program to perform the method of data migration of any of claims 1 to 5.
10. A storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the method of data migration of any of claims 1 to 5 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110398368.8A CN113220659B (en) | 2021-04-08 | 2021-04-08 | Data migration method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110398368.8A CN113220659B (en) | 2021-04-08 | 2021-04-08 | Data migration method, system, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113220659A true CN113220659A (en) | 2021-08-06 |
| CN113220659B CN113220659B (en) | 2023-06-09 |
Family
ID=77087257
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110398368.8A Active CN113220659B (en) | 2021-04-08 | 2021-04-08 | Data migration method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113220659B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113761043A (en) * | 2021-08-17 | 2021-12-07 | 紫金诚征信有限公司 | Data extraction method and device, computer equipment and storage medium |
| CN114936199A (en) * | 2022-07-21 | 2022-08-23 | 平安银行股份有限公司 | Data processing method for system reconstruction, computer equipment and storage medium |
| CN115269745A (en) * | 2022-07-27 | 2022-11-01 | 国网江苏省电力有限公司电力科学研究院 | Relational data-to-graph data mapping method, device and storage medium |
| CN115658978A (en) * | 2022-11-14 | 2023-01-31 | 杭州欧若数网科技有限公司 | Graph database system multi-source data importing method and device |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174098A1 (en) * | 2001-05-04 | 2002-11-21 | Lasmsoft Corporation | Method and system for providing a dynamic and real-time exchange between heterogeneous database systems |
| US20030112742A1 (en) * | 2001-12-19 | 2003-06-19 | Piper Richard Matthew | System and method for multiple-threaded access to a database |
| US20160092596A1 (en) * | 2014-09-26 | 2016-03-31 | Samsung Sds Co., Ltd. | Database migration method and apparatus |
| CN106227800A (en) * | 2016-07-21 | 2016-12-14 | 中国科学院软件研究所 | The storage method of the big data of a kind of highlights correlations and management system |
| WO2017186774A1 (en) * | 2016-04-26 | 2017-11-02 | Pilab S.A. | Systems and methods for querying databases |
| CN108280159A (en) * | 2018-01-16 | 2018-07-13 | 云南大学 | A method of converting chart database to relational database |
| CN109408535A (en) * | 2018-09-28 | 2019-03-01 | 中国平安财产保险股份有限公司 | Big data quantity matching process, device, computer equipment and storage medium |
| CN109753537A (en) * | 2019-01-25 | 2019-05-14 | 中国人民大学 | A kind of interactive data moving method from relation data to diagram data |
| CN110209826A (en) * | 2018-02-06 | 2019-09-06 | 武汉观图信息科技有限公司 | A kind of financial map construction and analysis method towards bank risk control |
| CN111488406A (en) * | 2020-04-16 | 2020-08-04 | 南京安链数据科技有限公司 | Graph database management method |
| US20200320098A1 (en) * | 2019-04-03 | 2020-10-08 | Sap Se | High Throughput Cross Database Table Synchronization and Transactional Replication in Federated Databases |
| CN111930958A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Graph database construction method, computing device and readable storage medium |
| CN112015818A (en) * | 2020-08-31 | 2020-12-01 | 杭州欧若数网科技有限公司 | Distributed graph database UUID generation method, device, equipment and medium |
| US20210056104A1 (en) * | 2019-08-24 | 2021-02-25 | International Business Machines Corporation | Supporting Synergistic and Retrofittable Graph Queries Inside a Relational Database |
-
2021
- 2021-04-08 CN CN202110398368.8A patent/CN113220659B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174098A1 (en) * | 2001-05-04 | 2002-11-21 | Lasmsoft Corporation | Method and system for providing a dynamic and real-time exchange between heterogeneous database systems |
| US20030112742A1 (en) * | 2001-12-19 | 2003-06-19 | Piper Richard Matthew | System and method for multiple-threaded access to a database |
| US20160092596A1 (en) * | 2014-09-26 | 2016-03-31 | Samsung Sds Co., Ltd. | Database migration method and apparatus |
| WO2017186774A1 (en) * | 2016-04-26 | 2017-11-02 | Pilab S.A. | Systems and methods for querying databases |
| CN106227800A (en) * | 2016-07-21 | 2016-12-14 | 中国科学院软件研究所 | The storage method of the big data of a kind of highlights correlations and management system |
| CN108280159A (en) * | 2018-01-16 | 2018-07-13 | 云南大学 | A method of converting chart database to relational database |
| CN110209826A (en) * | 2018-02-06 | 2019-09-06 | 武汉观图信息科技有限公司 | A kind of financial map construction and analysis method towards bank risk control |
| CN109408535A (en) * | 2018-09-28 | 2019-03-01 | 中国平安财产保险股份有限公司 | Big data quantity matching process, device, computer equipment and storage medium |
| CN109753537A (en) * | 2019-01-25 | 2019-05-14 | 中国人民大学 | A kind of interactive data moving method from relation data to diagram data |
| US20200320098A1 (en) * | 2019-04-03 | 2020-10-08 | Sap Se | High Throughput Cross Database Table Synchronization and Transactional Replication in Federated Databases |
| US20210056104A1 (en) * | 2019-08-24 | 2021-02-25 | International Business Machines Corporation | Supporting Synergistic and Retrofittable Graph Queries Inside a Relational Database |
| CN111488406A (en) * | 2020-04-16 | 2020-08-04 | 南京安链数据科技有限公司 | Graph database management method |
| CN111930958A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Graph database construction method, computing device and readable storage medium |
| CN112015818A (en) * | 2020-08-31 | 2020-12-01 | 杭州欧若数网科技有限公司 | Distributed graph database UUID generation method, device, equipment and medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113761043A (en) * | 2021-08-17 | 2021-12-07 | 紫金诚征信有限公司 | Data extraction method and device, computer equipment and storage medium |
| CN114936199A (en) * | 2022-07-21 | 2022-08-23 | 平安银行股份有限公司 | Data processing method for system reconstruction, computer equipment and storage medium |
| CN115269745A (en) * | 2022-07-27 | 2022-11-01 | 国网江苏省电力有限公司电力科学研究院 | Relational data-to-graph data mapping method, device and storage medium |
| CN115269745B (en) * | 2022-07-27 | 2023-11-14 | 国网江苏省电力有限公司电力科学研究院 | Method, equipment and storage medium for mapping relational data to graph data |
| CN115658978A (en) * | 2022-11-14 | 2023-01-31 | 杭州欧若数网科技有限公司 | Graph database system multi-source data importing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113220659B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113220659A (en) | Data migration method, system, electronic device and storage medium | |
| US10467245B2 (en) | System and methods for mapping and searching objects in multidimensional space | |
| US7447839B2 (en) | System for a distributed column chunk data store | |
| EP3236365A1 (en) | Data query method and device | |
| US9323791B2 (en) | Apparatus and method for expanding a shared-nothing system | |
| CN112015820A (en) | Method, system, electronic device and storage medium for implementing distributed graph database | |
| US20070061543A1 (en) | Method for a distributed column chunk data store | |
| US11693843B2 (en) | Data processing method and system of a distributed graph database | |
| CN112363979A (en) | Distributed index method and system based on graph database | |
| CN111797134A (en) | Data query method and device of distributed database and storage medium | |
| CN113094387A (en) | Data query method and device, electronic equipment and machine-readable storage medium | |
| WO2019165763A1 (en) | Method for use in querying data | |
| CN111651424A (en) | Data processing method and device, data node and storage medium | |
| CN113515518A (en) | Data storage method and device, computer equipment and storage medium | |
| US8005844B2 (en) | On-line organization of data sets | |
| CN111008198A (en) | Service data acquisition method and device, storage medium and electronic equipment | |
| US10042957B2 (en) | Devices and methods for implementing dynamic collaborative workflow systems | |
| CN115328950A (en) | Secondary index-based hbase query method, terminal device and storage medium | |
| US10114864B1 (en) | List element query support and processing | |
| CN113849482A (en) | Data migration method and device and electronic equipment | |
| CN113672640A (en) | Data query method and device, computer equipment and storage medium | |
| KR20210110123A (en) | Apparatus for nonrelationlal database using relational database structure and data standardization method thereof | |
| CN113111138A (en) | Data processing method, device, computing equipment and medium | |
| US9442948B2 (en) | Resource-specific control blocks for database cache | |
| CN112231377A (en) | Data mapping method, system, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 310000 7th floor, building B, No. 482, Qianmo Road, Xixing street, Binjiang District, Hangzhou City, Zhejiang Province Patentee after: Huoshi Creation Technology Co.,Ltd. Address before: 310000 7th floor, building B, No. 482, Qianmo Road, Xixing street, Binjiang District, Hangzhou City, Zhejiang Province Patentee before: HANGZHOU FIRESTONE TECHNOLOGY Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder |