CN110209732A - Method of data synchronization and device of the relevant database to Hadoop database - Google Patents
Method of data synchronization and device of the relevant database to Hadoop database Download PDFInfo
- Publication number
- CN110209732A CN110209732A CN201910341127.2A CN201910341127A CN110209732A CN 110209732 A CN110209732 A CN 110209732A CN 201910341127 A CN201910341127 A CN 201910341127A CN 110209732 A CN110209732 A CN 110209732A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- synchronization
- configuration
- variable element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000001360 synchronised effect Effects 0.000 claims abstract description 50
- 238000012986 modification Methods 0.000 claims abstract description 33
- 230000004048 modification Effects 0.000 claims abstract description 33
- 230000008569 process Effects 0.000 claims description 11
- 241000208340 Araliaceae Species 0.000 claims description 10
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 claims description 10
- 235000003140 Panax quinquefolius Nutrition 0.000 claims description 10
- 235000008434 ginseng Nutrition 0.000 claims description 10
- 230000010076 replication Effects 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 6
- 238000013461 design Methods 0.000 description 8
- 230000008859 change Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application involves a kind of relevant databases to the method for data synchronization of Hadoop database, device, computer equipment and storage medium, method includes: according to for the pre-stored invariant parameter in business side to be synchronized and its configuration data, it identifies the invariant parameter in each table of the relevant database of business side, and the invariant parameter identified is configured；For the variable element in each table of relevant database, corresponding configuration data modification interface is provided, so that user configures variable element；And operated according to the configuration to variable element in configuration data modification interface, determine the configuration data of variable element；Determine the relevant database of business side to the data synchronization level of Hadoop database；Determine corresponding synchronization scheme, and synchronous according to the data that synchronization scheme executes relevant database to Hadoop database.The application can reduce workload, improve synchronous efficiency.
Description
Technical field
The present invention relates to big data processing technology fields, more particularly to a kind of relevant database to Hadoop database
Method of data synchronization, device, computer equipment and storage medium.
Background technique
In the synchronizing process of relevant database to Hadoop database (i.e. HadoopDatabase, abbreviation Hbase),
No matter which kind of tool is used, the Synchronization Design of doping service logic is generally done both for single table.Doping business is done for single table
The Synchronization Design of logic refers to: designing a synchronization scheme for each table.Because different tables of data, the content of storage is not
Equally, the information such as field of design are also all different, for example, when order table has order ID, order title, the order amount of money, delivery
Between etc. fields, user's table have the field informations such as user's name, User ID, user mobile phone number.For example, in relevant database
There are 100 tables, needs to do 100 synchronization schemes.This data method of synchronization heavy workload, and do not have versatility.
Summary of the invention
The embodiment of the present application provides a kind of relevant database to the method for data synchronization, device, meter of Hadoop database
Machine equipment and storage medium are calculated, general synchronization scheme can be provided, workload is greatly reduced.
The embodiment of the present application provides a kind of method of data synchronization of relevant database to Hadoop database, the method
Include:
According to the pre-stored invariant parameter in business side to be synchronized and its configuration data is directed to, the business side is identified
Relevant database each table in invariant parameter, and the invariant parameter identified is configured；
For the variable element in each table of the relevant database, corresponding configuration data modification circle is provided
Face, so that user configures the variable element；And according to variable element described in configuration data modification interface
Configuration operation, determine the configuration data of the variable element；
Determine the relevant database of the business side to the data synchronization level of Hadoop database；
Matched according to the configuration data of the invariant parameter of the relevant database of the business side, the variable element
Data and the data synchronization level are set, determine corresponding synchronization scheme, and relational data is executed according to the synchronization scheme
The data of library to Hadoop database are synchronous.
In some embodiments, the method also includes: by the constant ginseng of the relevant database of the business side
Several configuration datas and the data synchronization level are stored into default middle database, for obtaining in data synchronization process next time
The configuration data and data synchronization level of the invariant parameter.
In some embodiments, the default configuration ginseng of the variable element is provided in configuration data modification interface
Number；It is corresponding, it is described that corresponding configuration data modification interface is provided, so that user configures the variable element, comprising:
Corresponding configuration data modification interface is provided, so that silent configuration data of the user to the variable element is modified.
In some embodiments, the data of relevant database to the Hadoop database of the determination business side are same
Walk grade, comprising: according to the type of the business side, the business side business scenario and the method for synchronization at least one of,
Determine the data synchronization level.
In some embodiments, the configuration number of the invariant parameter of the relevant database according to the business side
According to the configuration data and the data synchronization level of, the variable element, corresponding synchronization scheme is determined, and according to the synchronization
It includes: relevant database according to the business side that the data that scheme executes relevant database to Hadoop database, which synchronize,
Metadata information, the preset standard table structure of the Hadoop database, the business side relevant database with it is described
Field mapping relations, the configuration data of the invariant parameter, the configuration data of the variable element and the institute of Hadoop database
Data synchronization level is stated, corresponding synchronization scheme is determined using database replication synchronization data software, and according to the side of synchronization
The data that case executes relevant database to Hadoop database are synchronous.
In some embodiments, the invariant parameter includes at least one in metadata table name, literary name section and field type
?.
In some embodiments, the variable element include data validity interval, locking range, major key, from increasing field and same
At least one of in step mode.
The embodiment of the present application also provides a kind of data synchronization unit of relevant database to Hadoop database, the dress
It sets and includes:
First configuration module, for according to for the pre-stored invariant parameter in business side to be synchronized and its configuration number
According to identifying the invariant parameter in each table of the relevant database of the business side, and to the invariant parameter identified
It is configured；
Second configuration module, for providing corresponding for the variable element in each table of the relevant database
Configuration data modify interface, so that user configures the variable element；And boundary is modified according to the configuration data
The configuration of variable element described in face operates, and determines the configuration data of the variable element；
Level determination module, for determining that the data of relevant database to Hadoop database of the business side are synchronous
Grade；
Data simultaneous module, for the configuration number according to the invariant parameter of the relevant database of the business side
According to the configuration data and the data synchronization level of, the variable element, corresponding synchronization scheme is determined, and according to the synchronization
The data that scheme executes relevant database to Hadoop database are synchronous.
The embodiment of the present application also provides a kind of computer equipment, including memory and processor, stores in the memory
There is computer-readable instruction, when the computer-readable instruction is executed by the processor, so that processor execution is above-mentioned
The step of method of data synchronization.
The embodiment of the present application also provides a kind of storage medium for being stored with computer-readable instruction, the computer-readable finger
When order is executed by one or more processors, so that the step of one or more processors execute above-mentioned method of data synchronization.
Method of data synchronization, device, calculating of the relevant database provided by the embodiments of the present application to Hadoop database
Machine equipment and storage medium, by identifying the invariant parameter in relevant database to the pre-stored mode of invariant parameter,
And the invariant parameter identified is configured according to pre-stored configuration data, for variable element by providing configuration number
According to the mode at modification interface so that user carries out manual configuration, this data configuration mode can greatly improve the synchronous effect of data
Rate.Based on flexible parameter configuration mode, the universal synchronous side of all tables of data for relevant database can be designed that
Case, it is not necessary to design a synchronization scheme for each table, greatly reduce workload, also greatly improve data synchronization efficiency.
Detailed description of the invention
Fig. 1 is the internal structure block diagram of computer equipment in one embodiment；
Fig. 2 be one embodiment in relevant database to Hadoop database method of data synchronization flow chart；
Fig. 3 be one embodiment in relevant database to Hadoop database data synchronization unit structural representation
Figure.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
It is appreciated that term " first " used in this application, " second " etc. can be used to describe various elements herein,
But these elements should not be limited by these terms.These terms are only used to distinguish the first element from the other element.
Fig. 1 is the structural schematic diagram of computer equipment in the application one embodiment.As shown in Figure 1, the computer equipment
Including processor, non-volatile memory medium, memory and the network interface connected by system bus.Wherein, the computer
The non-volatile memory medium of equipment is stored with operating system, database and computer-readable instruction, can be stored in database
Control information sequence when the computer-readable instruction is executed by processor, may make processor to realize a kind of relevant database
To the method for data synchronization of Hadoop database.The processor of the computer equipment is for providing calculating and control ability, support
The operation of entire computer equipment.Computer-readable instruction can be stored in the memory of the computer equipment, which can
When reading instruction is executed by processor, the data that may make processor to execute a kind of relevant database to Hadoop database are synchronous
Method.The network interface of the computer equipment is used for and terminal connection communication.It will be understood by those skilled in the art that showing in Fig. 1
Structure out, only the block diagram of part-structure relevant to application scheme, does not constitute and is applied to application scheme
The restriction of computer equipment thereon, specific computer equipment may include than more or fewer components as shown in the figure, or
Person combines certain components, or with different component layouts.
The embodiment of the present application provides a kind of method of data synchronization of relevant database to Hadoop database, the relationship type
The method of data synchronization of database to Hadoop database can be applied in computer equipment shown in figure 1, the computer
The particular hardware form of equipment has very much, for example, data control platform, data control platform is a big data platform, can be with
Data control is carried out to multiple business sides of access.
As shown in Fig. 2, method of data synchronization provided by the embodiments of the present application includes:
S21, basis are directed to the pre-stored invariant parameter in business side to be synchronized and its configuration data, identify the industry
Invariant parameter in each table of the relevant database of business side, and the invariant parameter identified is configured；
It will be appreciated that there are many relevant databases, i.e. the type of relevant database have it is multiple, for example, Oracle
Database, MySql database, PostgreSQL database, Hive database etc..There is a corresponding pass in each business side
It is type database, the type of different business sides, relevant database may be different.
It will be appreciated that Hadoop database, i.e. HadoopDatabase, abbreviation hbase, hbase are a distributions
, PostgreSQL database towards column, hbase is different from general relational database, it is one and is suitable for unstructured data
The database of storage.
It will be appreciated that there are many parameters of the relevant database of business side to be synchronized, for example, metadata table name,
Literary name section, data validity interval, locking range, major key, increases the parameters such as field, the method for synchronization at field type certainly.To relationship type number
Before synchronizing according to library progress data, need to configure these parameters.Due to table different in relevant database, parameter
May be different, therefore in the prior art, it is to design a synchronization scheme, workload for each table in relevant database
It is huge.However, taking different processing modes in this application: invariant parameter is pre-configured, it is mentioned for variable element
Interface is modified for corresponding configuration data, for user's manual configuration.
It will be appreciated that invariant parameter and variable element are not unalterable, the application of business side difference, business side
Scene difference etc., invariant parameter and variable element may be different.So-called invariant parameter refers to for each table, has
The configuration data of parameters is fixed a bit, these parameters are invariant parameter, and the configuration data of some parameters is may to occur
Variation, these parameters are variable element.Under certain application scenarios, the invariant parameter of relevant database may include it is described not
Variable element include in metadata table name, literary name section and field type at least one of, variable element may include data validity interval,
Locking range, major key, from increase in field and the method for synchronization at least one of.
Before carrying out data and synchronizing, it is first determined which parameter is invariant parameter, which parameter is variable element, will not
Variable element and its configuration data are stored in data control platform, and realization is stored in advance.When progress data are synchronous, to relationship type
Each table of database identified, when recognizing parameter identical with any one pre-stored invariant parameter, then
Invariant parameter is had this parameter as, and according to pre-stored corresponding configuration data, which is configured.
For example, including order table and user's table, metadata table name, the literary name of order table and user's table in relevant database
Section and field type will not change, therefore these parameters are invariant parameter, stored in data control platform in advance it is good this
A little parameters.Carry out data it is synchronous when, in relevant database order table and user's table identify, when recognizing first number
When according to this parameter of table name, metadata table name according to the pre-stored data configures the metadata table name recognized.For it
His invariant parameter, configuration process are also in this way, to realize the configuration to the invariant parameter of relevant database.
S22, for the variable element in each table of the relevant database, corresponding configuration data modification is provided
Interface, so that user configures the variable element；And it can be changed ginseng according to described in configuration data modification interface
Several configuration operations, determines the configuration data of the variable element；
In practical applications, the default configuration ginseng of the variable element can be provided in configuration data modification interface
Number；In this way, it is variable that user can modify modification on interface in the configuration data after showing the configuration data modification interface
The default configuration data of parameter.That is, the above-mentioned corresponding configuration data of offer modifies interface, so that user is to described variable
Parameter is configured, can be with are as follows: provides corresponding configuration data modification interface, the silent configuration for user to the variable element
Data are modified.
For example, the variable element of the relevant database of business side to be synchronized include: data validity interval, locking range,
Major key is matched data validity interval by configuration operation of the user on configuration data modification interface from increasing field and the method for synchronization
It is set to 2 months, locking range is configured to the data within nearly 48 hours, and major key is configured to order ID, can be configured to order from increasing
Time field, the method for synchronization can be configured to incremental update or full dose updates.Wherein, each variable element is modified in configuration data
There is default parameters on interface, default parameters can be rule of thumb arranged, can also be with automatic identification, for example, therein from increasing and main
Key can determine default configuration by way of automatic identification, default configuration or according to circumstances be repaired for selection by the user
Change.
S23, determine the relevant database of the business side to the data synchronization level of Hadoop database；
It will be appreciated that in a database, some tables need to update every 1 hour it is primary, some tables need every
It updates once within one day, some tables only need to synchronize once, therefore a database may correspond to multiple synchronization levels.
In practical applications, the synchronous task of different stage corresponds to different data synchronization levels, more in addition to tables of data
New frequency, the factor being contemplated that when determining data synchronization level also there are many, for example, different business sides, business side are different
Business scenario, different methods of synchronization etc., it is thus determined that the process of the data synchronization level can also include: according to the industry
The type of business side, the business side business scenario and the method for synchronization at least one of, determine the data synchronization level.
It, can be according to data synchronization level to the place of each business side when needing to carry out data to multiple business sides to synchronize
It makes sequence in order to be arranged, can be improved synchronous efficiency, reduce influencing each other between different task, and in design synchronization scheme
When can determine when according to data synchronization level carry out data synchronize.
It will be appreciated that there is no sequencing between above-mentioned steps S21, S22, S23, for example, three steps are performed simultaneously,
Synchronous efficiency can be improved in this way.
S24, the configuration data according to the invariant parameter of the relevant database of the business side, the variable element
Configuration data and the data synchronization level, determine corresponding synchronization scheme, and relationship type is executed according to the synchronization scheme
The data of database to Hadoop database are synchronous.
In practical applications, synchronization scheme can be determined using database replication synchronization data software, for example, Kettle is soft
Part.Synchronization scheme is determined, in turn based on the data synchronization level of the parameters, determination that configure before using Kettle software
It is synchronous to execute data.
In practical application, Kettle software can specifically be believed according to the metadata of the relevant database of the business side
Breath, the preset standard table structure of the Hadoop database, the relevant database of the business side and the Hadoop data
Field mapping relations, the configuration data of the invariant parameter, the configuration data of the variable element in library are synchronous with the data
Grade determines corresponding synchronization scheme using database replication synchronization data software.
For example: the corresponding table configuration information of inquiry different data synchronization level uses kettle database group
Part configuration data source information, metadata information etc.；Kettle table reading assembly is used according to data source information；Table data are read,
The synchronization level information of each table in table data, that is, database, for example, database has ten tables, three tables are grade A, three tables
For grade B, four tables are this configuration information of grade C；Said modules are handled according to preset logic flow, including judge item
Part, data flow etc.；The table data read are subjected to Screening Treatment；It will using Hbase operating assembly according to metadata information
Obtained data write-in Hbase table is screened, and then has configured Hbase table and matching field etc., and then according to having configured Hbase table
And matching field determines synchronization scheme.Constant ginseng is all referred to during reading assembly, reading table data, processing component etc.
Number, variable element etc..
In some embodiments, the method for data synchronization can also include:
By the configuration data of the invariant parameter of the relevant database of the business side and the data synchronization level
It stores into default middle database, so that the configuration data and the data that obtain the invariant parameter in data synchronization process next time are same
Walk grade.
Here, invariant parameter and its configuration data, data synchronization level are stored into default middle database, when later again
The relevant database of the business side extract directly from middle database when data synchronize, without again to constant ginseng
Number is configured, without being determined again to data synchronization level.That is, being not that synchronize will be right for data each time
Invariant parameter carries out configuring and determining synchronization level, and it can be used repeatedly for storage to middle database, data after can be improved in this way
Synchronous efficiency.
Certainly, when the business scenario of relevant database change cause its invariant parameter change when, need again into
Row parameter configuration and determining data synchronization level.
Method of data synchronization provided by the embodiments of the present application, by identifying relationship to the pre-stored mode of invariant parameter
Invariant parameter in type database, and the invariant parameter identified is configured according to pre-stored configuration data, for
For variable element so that user carries out manual configuration by way of providing configuration data and modifying interface, this data configuration mode can
To greatly improve data synchronization efficiency.Based on flexible parameter configuration mode, the institute for relevant database can be designed that
There is the universal synchronous scheme of tables of data, it is not necessary to design a synchronization scheme for each table, greatly reduce workload, also significantly
Improve data synchronization efficiency.
Currently, a data control platform access tens business sides, the quantity of table have 10000 multiple, if according to tradition
Mode to carry out data synchronous, at least need could complete within 1~2 month, and pass through data provided by the embodiments of the present application synchronization
Method only needs can complete for several days, substantially increases synchronous efficiency.
As shown in figure 3, in one embodiment, the data for providing a kind of relevant database to Hadoop database are same
Device 30 is walked, which can integrate in above-mentioned computer equipment (for example, data control platform), specifically
May include:
First configuration module, for according to for the pre-stored invariant parameter in business side to be synchronized and its configuration number
According to identifying the invariant parameter in each table of the relevant database of the business side, and to the invariant parameter identified
It is configured；
Second configuration module, for providing corresponding for the variable element in each table of the relevant database
Configuration data modify interface, so that user configures the variable element；And boundary is modified according to the configuration data
The configuration of variable element described in face operates, and determines the configuration data of the variable element；
Level determination module, for determining that the data of relevant database to Hadoop database of the business side are synchronous
Grade；
Data simultaneous module, for the configuration number according to the invariant parameter of the relevant database of the business side
According to the configuration data and the data synchronization level of, the variable element, corresponding synchronization scheme is determined, and according to the synchronization
The data that scheme executes relevant database to Hadoop database are synchronous.
In some embodiments, described device further include:
Data memory module, for by the configuration data of the invariant parameter of the relevant database of the business side and
The data synchronization level is stored into default middle database, for obtaining the invariant parameter in data synchronization process next time
Configuration data and data synchronization level.
In some embodiments, the default configuration ginseng of the variable element is provided in configuration data modification interface
Number；Second configuration module is specifically used for: corresponding configuration data modification interface is provided, so that user writes from memory to the variable element
Configuration data is modified.
In some embodiments, level determination module is specifically used for: according to the type of the business side, the business side
At least one of in business scenario and the method for synchronization, determine the data synchronization level.
In some embodiments, data simultaneous module is specifically used for: according to the member of the relevant database of the business side
Data information, the preset standard table structure of the Hadoop database, the business side relevant database with it is described
Field mapping relations, the configuration data of the invariant parameter, the configuration data of the variable element and the institute of Hadoop database
Data synchronization level is stated, corresponding synchronization scheme is determined using database replication synchronization data software, and according to the side of synchronization
The data that case executes relevant database to Hadoop database are synchronous.
In some embodiments, the invariant parameter includes at least one in metadata table name, literary name section and field type
?.
In some embodiments, the variable element include data validity interval, locking range, major key, from increasing field and same
At least one of in step mode.
Data synchronization unit provided by the present application, the first configuration module is by identifying the pre-stored mode of invariant parameter
Invariant parameter in relevant database out, and the invariant parameter identified is matched according to pre-stored configuration data
It sets, the second configuration module is for variable element so that user is matched manually by way of providing configuration data and modifying interface
It sets, this data configuration mode can greatly improve data synchronization efficiency.Based on flexible parameter configuration mode, the synchronous mould of data
Block can be designed that the universal synchronous scheme of all tables of data for relevant database, it is not necessary to for each table design one
A synchronization scheme greatly reduces workload, also greatly improves data synchronization efficiency.
In some embodiments, propose a kind of computer equipment, the computer equipment include memory, processor and
It is stored in the computer program that can be run on the memory and on the processor, the processor executes the computer
It is performed the steps of when program according to for the pre-stored invariant parameter in business side to be synchronized and its configuration data, identification
Invariant parameter in each table of the relevant database of the business side out, and the invariant parameter identified is matched
It sets；For the variable element in each table of the relevant database, corresponding configuration data modification interface is provided, for
User configures the variable element；And it is grasped according to the configuration to variable element described in configuration data modification interface
Make, determines the configuration data of the variable element；Determine the relevant database of the business side to the number of Hadoop database
According to synchronization level；According to the configuration data of the invariant parameter of the relevant database of the business side, the variable element
Configuration data and the data synchronization level, determine corresponding synchronization scheme, and relationship type is executed according to the synchronization scheme
The data of database to Hadoop database are synchronous.
In some embodiments, it also performs the steps of when the processor executes the computer program by the industry
The configuration data of the invariant parameter of the relevant database of business side and the data synchronization level are stored to default middle database
In, for obtaining the configuration data and data synchronization level of the invariant parameter in data synchronization process next time.
In some embodiments, the default configuration ginseng of the variable element is provided in configuration data modification interface
Number；The corresponding configuration data of the offer that the processor executes modifies interface, so that user carries out the variable element
Configuration, comprising: corresponding configuration data modification interface is provided, so that silent configuration data of the user to the variable element is repaired
Change.
In some embodiments, the relevant database for the determination business side that the processor executes arrives
The data synchronization level of Hadoop database, comprising: according to the type of the business side, the business scenario of the business side and same
At least one of in step mode, determine the data synchronization level.
In some embodiments, what the processor executed is described according to the relevant database of the business side
The configuration data of invariant parameter, the configuration data of the variable element and the data synchronization level, determine the corresponding side of synchronization
Case, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database, comprising: according to the industry
The metadata information of the relevant database of business side, the preset standard table structure of the Hadoop database, the business side
It is the field mapping relations of relevant database and the Hadoop database, the configuration data of the invariant parameter, described variable
The configuration data of parameter and the data synchronization level determine the corresponding side of synchronization using database replication synchronization data software
Case, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database.
In some embodiments, the invariant parameter includes at least one in metadata table name, literary name section and field type
?.
In some embodiments, the variable element include data validity interval, locking range, major key, from increasing field and same
At least one of in step mode.
The number of the beneficial effect of computer equipment provided by the present application and above-mentioned relation type database to Hadoop database
Identical with device according to synchronous method, which is not described herein again.
In one embodiment it is proposed that a kind of storage medium for being stored with computer-readable instruction, this is computer-readable
When instruction is executed by one or more processors, so that one or more processors execute following steps: according to for be synchronized
The pre-stored invariant parameter in business side and its configuration data, each for identifying the relevant database of the business side
Invariant parameter in table, and the invariant parameter identified is configured；For in each table of the relevant database
Variable element, corresponding configuration data modification interface is provided, so that user configures the variable element；And according to right
The configuration operation of variable element described in configuration data modification interface, determines the configuration data of the variable element；It determines
Data synchronization level of the relevant database of the business side to Hadoop database；According to the relationship type number of the business side
According to the configuration data of the invariant parameter in library, the configuration data of the variable element and the data synchronization level, determining pair
The synchronization scheme answered, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database.
In some embodiments, when computer-readable instruction is executed by one or more processors, also make one or more
A processor executes following steps: by the configuration data of the invariant parameter of the relevant database of the business side and described
Data synchronization level is stored into default middle database, for obtaining the configuration of the invariant parameter in data synchronization process next time
Data and data synchronization level.
In some embodiments, the default configuration ginseng of the variable element is provided in configuration data modification interface
Number；The corresponding configuration data of the offer that one or more of processors execute modifies interface, for user to it is described can
Variable element is configured, comprising: provides corresponding configuration data modification interface, the silent configuration for user to the variable element
Data are modified.
In some embodiments, the relationship type number for the determination business side that one or more of processors execute
According to the data synchronization level in library to Hadoop database, comprising: according to the type of the business side, the business field of the business side
At least one of in scape and the method for synchronization, determine the data synchronization level.
In some embodiments, the relevant database according to the business side that one or more of processors execute
The configuration data of the invariant parameter, the configuration data of the variable element and the data synchronization level, determine corresponding
Synchronization scheme, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database, comprising: according to
The metadata information of the relevant database of the business side, the preset standard table structure of the Hadoop database, the industry
Field mapping relations, the configuration data of the invariant parameter, institute of the relevant database of business side and the Hadoop database
The configuration data and the data synchronization level for stating variable element are determined corresponding same using database replication synchronization data software
Step scheme, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database.
In some embodiments, the invariant parameter includes at least one in metadata table name, literary name section and field type
?.
In some embodiments, the variable element include data validity interval, locking range, major key, from increasing field and same
At least one of in step mode.
The data of the beneficial effect of storage medium provided by the present application and above-mentioned relation type database to Hadoop database
Synchronous method is identical with device, and which is not described herein again.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, which can be stored in a computer-readable storage and be situated between
In matter, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, storage medium above-mentioned can be
The non-volatile memory mediums such as magnetic disk, CD, read-only memory (Read-Only Memory, ROM) or random storage note
Recall body (Random Access Memory, RAM) etc..
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality
It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.Therefore, the scope of protection of the patent of the invention shall be subject to the appended claims.
Claims (10)
1. a kind of relevant database is to the method for data synchronization of Hadoop database, which is characterized in that the described method includes:
According to the pre-stored invariant parameter in business side to be synchronized and its configuration data is directed to, the pass of the business side is identified
It is the invariant parameter in each table of type database, and the invariant parameter identified is configured；
For the variable element in each table of the relevant database, corresponding configuration data modification interface is provided, with
The variable element is configured for user；And according to the configuration to variable element described in configuration data modification interface
Operation, determines the configuration data of the variable element；
Determine the relevant database of the business side to the data synchronization level of Hadoop database；
According to the configuration data of the invariant parameter of the relevant database of the business side, the configuration number of the variable element
According to the data synchronization level, corresponding synchronization scheme is determined, and relevant database is executed according to the synchronization scheme and is arrived
The data of Hadoop database are synchronous.
2. the method according to claim 1, wherein further include:
The configuration data of the invariant parameter of the relevant database of the business side and the data synchronization level are stored
Into default middle database, configuration data for obtaining the invariant parameter in data synchronization process next time is synchronous with data etc.
Grade.
3. the method according to claim 1, wherein being provided in configuration data modification interface described variable
The default configuration parameters of parameter；It is corresponding, it is described that corresponding configuration data modification interface is provided, so that user is to the variable ginseng
Number is configured, comprising: corresponding configuration data modification interface is provided, so that user is to the silent configuration data of the variable element
It modifies.
4. the method according to claim 1, wherein the relevant database of the determination business side arrives
The data synchronization level of Hadoop database, comprising:
According to the type of the business side, the business side business scenario and the method for synchronization at least one of, determine described in
Data synchronization level.
5. the method according to claim 1, wherein the institute of the relevant database according to the business side
The configuration data of invariant parameter, the configuration data of the variable element and the data synchronization level are stated, determines corresponding synchronization
Scheme, and synchronized according to the data that the synchronization scheme executes relevant database to Hadoop database and include:
According to the metadata information of the relevant database of the business side, the preset standard table knot of the Hadoop database
Structure, the field mapping relations of relevant database and the Hadoop database of the business side, the invariant parameter are matched
Data, the configuration data of the variable element and the data synchronization level are set, it is true using database replication synchronization data software
Fixed corresponding synchronization scheme, and it is synchronous according to the data that the synchronization scheme executes relevant database to Hadoop database.
6. described in any item methods according to claim 1~5, which is characterized in that the invariant parameter include metadata table name,
At least one of in literary name section and field type.
7. described in any item methods according to claim 1~5, which is characterized in that the variable element include data validity interval,
Locking range, major key, from increase in field and the method for synchronization at least one of.
8. a kind of relevant database is to the data synchronization unit of Hadoop database, which is characterized in that described device includes:
First configuration module, for knowing according to the pre-stored invariant parameter in business side to be synchronized and its configuration data is directed to
Not Chu the business side relevant database each table in invariant parameter, and the invariant parameter identified is matched
It sets；
Second configuration module, for providing and matching accordingly for the variable element in each table of the relevant database
Data modification interface is set, so that user configures the variable element；And according to in configuration data modification interface
The configuration of the variable element operates, and determines the configuration data of the variable element；
Level determination module, for determining that the data of relevant database to Hadoop database of the business side synchronize
Grade；
Data simultaneous module, for the configuration data according to the invariant parameter of the relevant database of the business side, institute
The configuration data and the data synchronization level for stating variable element, determine corresponding synchronization scheme, and according to the synchronization scheme
The data for executing relevant database to Hadoop database are synchronous.
9. a kind of computer equipment, which is characterized in that including memory and processor, being stored with computer in the memory can
Reading instruction, when the computer-readable instruction is executed by the processor, so that the processor executes such as claim 1 to 7
Any one of method of data synchronization described in claim the step of.
10. a kind of storage medium for being stored with computer-readable instruction, which is characterized in that the computer-readable instruction is by one
Or multiple processors are when executing, so that one or more processors are executed as described in any one of claims 1 to 7 claim
The step of method of data synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910341127.2A CN110209732A (en) | 2019-04-25 | 2019-04-25 | Method of data synchronization and device of the relevant database to Hadoop database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910341127.2A CN110209732A (en) | 2019-04-25 | 2019-04-25 | Method of data synchronization and device of the relevant database to Hadoop database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110209732A true CN110209732A (en) | 2019-09-06 |
Family
ID=67786469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910341127.2A Pending CN110209732A (en) | 2019-04-25 | 2019-04-25 | Method of data synchronization and device of the relevant database to Hadoop database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209732A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078313A (en) * | 2019-12-09 | 2020-04-28 | 神州数码融信软件有限公司 | Service parameter configuration method |
| CN113077201A (en) * | 2020-01-06 | 2021-07-06 | 北京京东振世信息技术有限公司 | Method, device and system for analyzing service parameters |
| CN113987069A (en) * | 2021-10-09 | 2022-01-28 | 中汽创智科技有限公司 | Data synchronization method, device, system, equipment, storage medium and vehicle |
| CN114020843A (en) * | 2021-11-04 | 2022-02-08 | 北京房江湖科技有限公司 | Spark framework-based data table synchronization method, device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2447458A1 (en) * | 2003-10-29 | 2005-04-29 | Ibm Canada Limited - Ibm Canada Limitee | System and method for managing query access to information |
| CN101697120A (en) * | 2009-11-05 | 2010-04-21 | 河南辉煌科技股份有限公司 | Relational database based generation method of J2EE code |
| WO2014097475A1 (en) * | 2012-12-21 | 2014-06-26 | 株式会社Ｍｕｒａｋｕｍｏ | Information processing method, information processing device, and program |
| WO2017148327A1 (en) * | 2016-03-03 | 2017-09-08 | 腾讯科技（深圳）有限公司 | Service parameter selection method and related device |
| US20180011655A1 (en) * | 2016-07-08 | 2018-01-11 | RtBrick Inc. | System and methods for defining object memory format in memory and store for object interactions, manipulation, and exchange in distributed network devices |
| CN107908631A (en) * | 2017-07-25 | 2018-04-13 | 平安科技（深圳）有限公司 | Data processing method, device, storage medium and computer equipment |
-
2019
- 2019-04-25 CN CN201910341127.2A patent/CN110209732A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2447458A1 (en) * | 2003-10-29 | 2005-04-29 | Ibm Canada Limited - Ibm Canada Limitee | System and method for managing query access to information |
| CN101697120A (en) * | 2009-11-05 | 2010-04-21 | 河南辉煌科技股份有限公司 | Relational database based generation method of J2EE code |
| WO2014097475A1 (en) * | 2012-12-21 | 2014-06-26 | 株式会社Ｍｕｒａｋｕｍｏ | Information processing method, information processing device, and program |
| WO2017148327A1 (en) * | 2016-03-03 | 2017-09-08 | 腾讯科技（深圳）有限公司 | Service parameter selection method and related device |
| US20180011655A1 (en) * | 2016-07-08 | 2018-01-11 | RtBrick Inc. | System and methods for defining object memory format in memory and store for object interactions, manipulation, and exchange in distributed network devices |
| CN107908631A (en) * | 2017-07-25 | 2018-04-13 | 平安科技（深圳）有限公司 | Data processing method, device, storage medium and computer equipment |
Non-Patent Citations (2)
| Title |
|---|
| 刘广志;刘军;: "铁路仿真通用三维可视化系统建模研究", 微计算机应用, no. 11, 15 November 2007 (2007-11-15) * |
| 生拥宏;刘川意;鞠大鹏;汪东升;: "差量存储的集中式文件级连续数据保护方法", 计算机科学与探索, no. 04, 15 July 2009 (2009-07-15) * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078313A (en) * | 2019-12-09 | 2020-04-28 | 神州数码融信软件有限公司 | Service parameter configuration method |
| CN113077201A (en) * | 2020-01-06 | 2021-07-06 | 北京京东振世信息技术有限公司 | Method, device and system for analyzing service parameters |
| CN113077201B (en) * | 2020-01-06 | 2024-05-24 | 北京京东振世信息技术有限公司 | Method, device and system for analyzing service parameters |
| CN113987069A (en) * | 2021-10-09 | 2022-01-28 | 中汽创智科技有限公司 | Data synchronization method, device, system, equipment, storage medium and vehicle |
| CN114020843A (en) * | 2021-11-04 | 2022-02-08 | 北京房江湖科技有限公司 | Spark framework-based data table synchronization method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209732A (en) | Method of data synchronization and device of the relevant database to Hadoop database | |
| CN110334152B (en) | Data synchronization method and device and server | |
| CN103177068B (en) | According to the system and method for existence compatible rule merging source record | |
| CN107967316A (en) | A kind of method of data synchronization, equipment and computer-readable recording medium | |
| CN109947399A (en) | Code structure generation method, device, computer installation and readable storage medium storing program for executing | |
| CN110209650A (en) | The regular moving method of data, device, computer equipment and storage medium | |
| CN106528070A (en) | Data table generation method and equipment | |
| CN109766349A (en) | The anti-weighing method of task, device, computer equipment and storage medium | |
| CN110245145A (en) | Structure synchronization method and apparatus of the relevant database to Hadoop database | |
| CN109299083A (en) | A kind of data governing system | |
| CN105677465B (en) | The data processing method and device of batch processing are run applied to bank | |
| CN104657164B (en) | Software upgrading treating method and apparatus | |
| CN112364049B (en) | Data synchronization script generation method, system, terminal and storage medium | |
| CN110532041A (en) | Regulation engine method for parameter configuration, device, computer equipment and storage medium | |
| CN109885614A (en) | A kind of method and apparatus that data are synchronous | |
| CN101976240A (en) | Form number generating method and system | |
| EP2965492B1 (en) | Selection of data storage settings for an application | |
| CN111767144A (en) | Transaction routing determination method, device, equipment and system for transaction data | |
| CN109799948A (en) | A kind of date storage method and device | |
| CN112102099B (en) | Policy data processing method and device, electronic equipment and storage medium | |
| CN108874738A (en) | Distributed parallel operation method, device, computer equipment and storage medium | |
| CN107832448A (en) | Database operation method, device and equipment | |
| CN110109981A (en) | Information displaying method, device, computer equipment and the storage medium of work queue | |
| CN103678591A (en) | Device and method for automatically executing multi-service receipt statistical treatment | |
| CN110308926A (en) | Software information methods of exhibiting, device, computer installation and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |