CN110245145A - Structure synchronization method and apparatus of the relevant database to Hadoop database - Google Patents
Structure synchronization method and apparatus of the relevant database to Hadoop database Download PDFInfo
- Publication number
- CN110245145A CN110245145A CN201910341135.7A CN201910341135A CN110245145A CN 110245145 A CN110245145 A CN 110245145A CN 201910341135 A CN201910341135 A CN 201910341135A CN 110245145 A CN110245145 A CN 110245145A
- Authority
- CN
- China
- Prior art keywords
- database
- relevant database
- synchronized
- business side
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
This application involves structure synchronization method, apparatus, computer equipment and the storage medium of a kind of relevant database to Hadoop database, method comprises determining that the type of the relevant database of business side to be synchronized；Determine data source information；The relevant database of business side to be synchronized is connected, and extracts metadata information from the relevant database being connected to；It converts metadata information to the preset standard table structure of Hadoop database in data control platform, and the field mapping relations between the relevant database of business side to be synchronized and Hadoop database is stored into default mapping table；Whether the table structure for periodically monitoring the relevant database of business side to be synchronized changes, if table structure changes, the preset standard table structure and default mapping table be converted to metadata information is updated.The table knot of relevant database can be synchronized in the preset standard table structure of Hadoop database by the application, that is, be synchronized to unified standard table structure, be convenient for operation management.
Description
Technical field
The present invention relates to big data processing technology fields, more particularly to a kind of relevant database to Hadoop database
Structure synchronization method, apparatus, computer equipment and storage medium.
Background technique
In the synchronizing process of relevant database to Hadoop database (i.e. HadoopDatabase, abbreviation Hbase),
Structure synchronization is the first step, and so-called structure synchronization, which refers to, is synchronized to the metadata of business side in data control platform.
During to relation technological researching, inventor's discovery: traditional method is Manual synchronization, i.e., first by relational data
The structure in library exports, then the manual table statement of building for being converted into Hbase executes, and this Manual synchronization scheme often has many disadvantages
End causes the mapping relations of two database structures to be safeguarded for example, the Hbase structure after conversion is nonstandard (for example, field is excessive)
It is difficult.
Summary of the invention
The embodiment of the present application provides a kind of structure synchronization method, apparatus to Hadoop database of relevant database, meter
Calculate machine equipment and storage medium, the preset standard table structure that can be synchronized to the table knot of relevant database in Hadoop database
In, that is, it is synchronized to unified standard table structure, is convenient for operation management
The embodiment of the present application provides a kind of structure synchronization method of relevant database to Hadoop database, the method
Include:
Determine the type of the relevant database of business side to be synchronized；
Data source information needed for determining the relevant database progress structure synchronization for the type；
According to the data source information, connect the relevant database of the business side to be synchronized, and from being connected to
Metadata information is extracted in relevant database；
Convert the metadata information to the preset standard table structure of Hadoop database in data control platform, and will
Field mapping relations between the relevant database of the business side to be synchronized and the Hadoop database are stored to pre-
If in mapping table；
Whether the table structure for periodically monitoring the relevant database of the business side to be synchronized changes, if table knot
Structure changes, then the preset standard table structure and the default mapping table progress being converted to the metadata information
It updates.
In some embodiments, the method also includes:
The data source information and/or the metadata information are stored to the default middle database of the data control platform
In.
In some embodiments, before the relevant database of the connection business side to be synchronized, the method
Further include:
According to the data source information, validation verification is carried out to the relevant database of the business side to be synchronized,
After being verified, the step of allowing to execute the relevant database of the connection business side to be synchronized.
In some embodiments, the relevant database to the business side to be synchronized carries out validation verification,
Include:
According to the data source information, judge whether be stored with the data source information in the data control platform；If
It is not stored to have the data source information, then according to the data source information, the pass of the tentative connection business side to be synchronized
It is type database, if tentative successful connection, is verified.
In some embodiments, the preset standard table structure includes identity field and information field；It is corresponding, it is described to incite somebody to action
The metadata information is converted into the preset standard table structure of Hadoop database in data control platform, comprising: by the member
Major key field in data information is stored to the identity field of the preset standard table structure, will be in the metadata information
Remaining field store to the information field of the preset standard table structure, and remaining described field is with keyword/correspondence
The mode of numerical value is stored into the information field.
In some embodiments, the type of the relevant database of determination business side to be synchronized, comprising:
In response to presetting the selection of any one relevant database kind Class Options in selection interface, the relationship that will be selected
The relevant database of type database kind Class Options corresponding relevant database types as the business side to be synchronized
Type is provided with multiple relevant database kind Class Options in the selection interface.
In some embodiments, number needed for the determination carries out structure synchronization for the relevant database of the type
According to source information, comprising:
After determining the type of relevant database of business side to be synchronized, provide the type corresponding information record
Enter interface, for the corresponding data source information of user's typing；The data source information being logged on the data input interface is made
For for data source information needed for the relevant database of type progress structure synchronization.
The embodiment of the present application also provides a kind of structure synchronization device of relevant database to Hadoop database, the dress
It sets and includes:
First determining module, the type of the relevant database for determining business side to be synchronized；
Second determining module, for data needed for determining the relevant database progress structure synchronization for the type
Source information；
Data extraction module, for connecting the relationship type number of the business side to be synchronized according to the data source information
Metadata information is extracted according to library, and from the relevant database being connected to；
Thaumatropy module, for converting the pre- of Hadoop database in data control platform for the metadata information
It is marked with quasi- table structure, and by the field between the relevant database of the business side to be synchronized and the Hadoop database
Mapping relations are stored into default mapping table；
Variation monitoring module, the table structure for the periodically relevant database of the monitoring business side to be synchronized are
It is no to change, if table structure changes, the preset standard table structure that the metadata information is converted to and
The default mapping table is updated.
The embodiment of the present application also provides a kind of computer equipment, including memory and processor, stores in the memory
There is computer-readable instruction, when the computer-readable instruction is executed by the processor, so that processor execution is above-mentioned
The step of structure synchronization method.
The embodiment of the present application also provides a kind of storage medium for being stored with computer-readable instruction, the computer-readable finger
When order is executed by one or more processors, so that the step of one or more processors execute above structure synchronous method.
Relevant database provided by the embodiments of the present application to Hadoop database structure synchronization method, apparatus, calculate
Machine equipment and storage medium, it is first determined the type of the relevant database of business side to be synchronized is so based on relational data
The type in library obtains data source information and metadata information, and then realizes that table structure is unitized, it is seen that the application is not for of the same race
Its table knot can be synchronized in the preset standard table structure in Hadoop database by the relevant database of class, i.e., synchronous
To unified standard table structure, it is convenient for operation management.The application not uses Manual synchronization, it is possible to reduce in conventional method manually
Synchronous workload reduces the generation of manual operation accident.Meanwhile whether the table structure of periodically monitoring relevant database is sent out
Changing, if table structure changes, the preset standard table structure that be converted to the metadata information and described
Default mapping table is updated, and automatic synchronization table structure may be implemented, and avoids the table structure of business side from changing same to follow-up data
Step impacts.
Detailed description of the invention
Fig. 1 is the internal structure block diagram of computer equipment in one embodiment；
Fig. 2 be one embodiment in relevant database to Hadoop database structure synchronization method flow chart；
Fig. 3 be one embodiment in relevant database to Hadoop database structure synchronization device structural representation
Figure.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
It is appreciated that term " first " used in this application, " second " etc. can be used to describe various elements herein,
But these elements should not be limited by these terms.These terms are only used to distinguish the first element from the other element.
Fig. 1 is the structural schematic diagram of computer equipment in the application one embodiment.As shown in Figure 1, the computer equipment
Including processor, non-volatile memory medium, memory and the network interface connected by system bus.Wherein, the computer
The non-volatile memory medium of equipment is stored with operating system, database and computer-readable instruction, can be stored in database
Control information sequence when the computer-readable instruction is executed by processor, may make processor to realize a kind of relevant database
To the structure synchronization method of Hadoop database.The processor of the computer equipment is for providing calculating and control ability, support
The operation of entire computer equipment.Computer-readable instruction can be stored in the memory of the computer equipment, which can
When reading instruction is executed by processor, may make processor execute a kind of relevant database to Hadoop database structure synchronization
Method.The network interface of the computer equipment is used for and terminal connection communication.It will be understood by those skilled in the art that showing in Fig. 1
Structure out, only the block diagram of part-structure relevant to application scheme, does not constitute and is applied to application scheme
The restriction of computer equipment thereon, specific computer equipment may include than more or fewer components as shown in the figure, or
Person combines certain components, or with different component layouts.
The embodiment of the present application provides a kind of structure synchronization method of relevant database to Hadoop database, the relationship type
The structure synchronization method of database to Hadoop database can be applied in computer equipment shown in figure 1, the computer
The particular hardware form of equipment has very much, for example, data control platform, data control platform is a big data platform, can be with
Data control is carried out to multiple business sides.
Wherein, Hadoop database, i.e. HadoopDatabase, abbreviation hbase, hbase be one it is distributed, towards
The PostgreSQL database of column, hbase are different from general relational database, it is the number for being suitable for unstructured data storage
According to library.
As shown in Fig. 2, structure synchronization method provided by the embodiments of the present application includes:
S21, determine business side to be synchronized relevant database type；
It will be appreciated that a data control platform corresponds to multiple business sides, i.e., a data control platform can be to more
The relevant database of a business side is managed, for example, structure synchronization, data synchronize.Each business side have one it is right
The type of the relevant database answered, the relevant database of different business sides is different.
It will be appreciated that there are many relevant databases, i.e. the type of relevant database have it is multiple, for example, Oracle
Database, MySql database, PostgreSQL database, Hive database etc..Here the pass for needing to carry out structure synchronization is determined
It is the type of type database, so as to data source information required for subsequent determination.
In practical applications, there are many modes for determining the type of the relevant database of business side to be synchronized, for example,
Data control platform provide a selection interface, multiple relevant database kind Class Options are provided in the selection interface, when with
After family selects any one relevant database kind Class Options in selection interface, relationship that data control platform can will be selected
The type of relevant database of the corresponding relevant database type of type database kind Class Options as business side to be synchronized.
That is, selection of the data control platform in response to any one relevant database kind Class Options in default selection interface,
Using the corresponding relevant database type of relevant database kind Class Options selected as the business side to be synchronized
The type of relevant database.Here, it is contemplated that different business sides will use different relevant databases, therefore select
Multiple relevant database kind Class Options are provided on interface, for selection by the user.
Data source information needed for S22, the determining relevant database for the type carry out structure synchronization；
It will be appreciated that data source information needed for carrying out structure synchronization to different types of relevant database is different,
Which data source information needed for the relevant database type deciding step S22 determined in step S21 has.For example, being directed to
The relevant database of type A, data source information needed for carrying out structure synchronization may include: the driving of relevant database
Packet, address, title, user name, password, connection pool size etc..For the relevant database of type B, structure synchronization institute is carried out
The data source information needed may include: the driving packet of relevant database, address, title, Connecting quantity, user name, password etc..
In practical applications, there are many modes for determining required data source information, and one kind is provided below: determining to same
After the type of the relevant database of the business side of step, provide the type corresponding data input interface, for user's record
Enter corresponding data source information；Using the data source information being logged on the data input interface as the pass for being directed to the type
It is data source information needed for type database carries out structure synchronization.
It is found that it can be seen which the data source information of typing has, when user is in information in data input interface
In input interface after the corresponding data source information of typing, data source information needed for step S22 can be learnt.
S23, according to the data source information, connect the relevant database of the business side to be synchronized, and from connection
To relevant database in extract metadata information；
For example, connected using information such as address, relational data library name, user name, passwords in data source information
Connect the relevant database of business side to be synchronized.
It will be appreciated that metadata (Metadata) information is the information about data in relevant database.In data
In warehouse system, metadata information can help the developer of data warehouse administrator and data warehouse easily to find
Their data of concern；Metadata information is the structure of data and the data of method for building up in description data warehouse.Metadata
Information can specifically include the information such as metadata table title, field, field type.
It will be appreciated that different types of relevant database in storing data database table structure, will use different sides
Formula is stored, therefore is directed to different types of relevant database, and different modes can be used and extract metadata information therein.
For extracting the table information under table space A, the corresponding extraction code of several relevant databases is as follows:
Oracle database: select table_name as tablename from all_tables where
Owner='A'；
Mysql database: select TABLE_NAME as tablename from information_
Schema.tables where table_schema='A'；
PostgreSQL database: SELECT tablename FROM pg_tables where schemaname='
A'。
For extracting the field information of table space A following table a, the corresponding extraction code of several relevant databases is as follows:
Oracle database: select cu.column_name as colname from all_cons_columns
Cu, all_constraints au where cu.constraint_name=au.constraint_name and
Au.constraint_type='P'and au.table_name='a'and cu.owner='A'；
Mysql database: select COLUMN_NAME as colname, DATA_TYPE as type from
Information_schema.columns where table_schema='A'and table_name='a'；
PostgreSQL database: select column_name as colname, udt_name as type from
Information_schema.columns where table_schema='a'and table_name='A'.
It will be appreciated that can know the relational data of business side to be synchronized after extracting metadata information
Which field how many table, every table have in library, and wherein which or several fields are major key.So-called major key, refers to one
The combination of column or multiple row, every a line in value energy unique identification table, can be with the entity integrity of enforced table by major key.
S24, the preset standard table structure for converting the metadata information to Hadoop database in data control platform,
And the field mapping relations between the relevant database of the business side to be synchronized and the Hadoop database are stored
Into default mapping table；
In practical applications, the preset standard table structure may include identity field and information field, i.e., is being marked in advance
Two fields are only provided in quasi- table structure, the conversion process in step S24 may include: in the metadata information at this time
Major key field is stored to the identity field of the preset standard table structure, remaining field in the metadata information is deposited
It stores up to the information field of the preset standard table structure, and remaining described field is deposited in a manner of keyword/correspondence numerical value
Storage is into the information field.In this way, corresponding to Hadoop regardless of how many field in the table of the relevant database of business side
Table structure on database is the same.
Based on above-mentioned conversion process, above-mentioned field mapping relations may include: the major key field in the metadata information
The identity field of the preset standard table structure is corresponded to, remaining field in the metadata information corresponds to the pre- bidding
The information field of quasi- table structure.Here, above-mentioned field mapping relations are stored in default mapping table.
Wherein, keyword/correspondence numerical value, refers to the form of key/value, for example, having order title in an order table
Field and the order amount of money, storage form after storing to preset standard table structure, in information field are as follows: Order_name: "
Iphone mobile phone order "/order_money:10000.
For example, table a has tetra- fields of a1, a2, a3, a4, and wherein a1 is major key.For Hadoop database, design
One general purpose table, i.e., above-mentioned preset standard table structure only includes identity field ID and information field Info, and field a1 is corresponding
Identity field ID, field a2, a3, a4 corresponding informance field Info.
In practical applications, above-mentioned preset standard table structure can be created by way of calling small routine Hbase sdk,
And field mapping relations are stored into default mapping table.
Whether the table structure for the relevant database that S25, periodicity monitor the business side to be synchronized changes, if
Table structure changes, then the preset standard table structure being converted to the metadata information and the default mapping table
It is updated.
In practical applications, have due to demand upgrading, technology upgrading or script design defect of business side etc.
The table structure of its relevant database may be changed, and the table structure of relevant database is controlled by business side, so industry
Business side may often modify table structure, preset standard table knot in the table structure and data control platform of such relevant database
Structure will appear inconsistent situation, it is therefore desirable to be updated to preset standard table structure.
Here, the preset standard table structure and the default mapping table progress metadata information being converted to
The detailed process of update can refer to step S23 and S24 as above, that is to say, that reacquire the relational data of business personnel side
The metadata information in library, and then carried out more according to the metadata information of reacquisition with the preset standard table structure being converted to before
Field mapping relations in default mapping table are also updated replacement by new replacement.
So far, the structure synchronization of the Hadoop database to the relevant database of business side to data control platform is realized
Process, the embodiment of the present application are accomplished that structure synchronization, synchronous with data not related.
In some embodiments, it after getting the metadata information of relevant database, can also will acquire
Metadata information is stored into the default middle database of data control platform.Equally, it after obtaining data source information, can also incite somebody to action
Data source information is stored into above-mentioned default middle database.It is, of course, also possible to work as relevant database to Hadoop database
Data synchronization information is stored into above-mentioned default middle database.
Here, a middle database is set, extracted metadata information, data source letter each time are stored in middle database
Breath, that is, realize and record to the historical changing of table structure, facilitate processing abnormal conditions.
It in some embodiments, can be with before executing step S23: according to the data source information, to described to same
The relevant database of the business side of step carries out validation verification to be allowed to execute described in the connection to same after being verified
The step of relevant database of the business side of step.
Wherein, the detailed process of validation verification may include:
According to the data source information, judge whether be stored with the data source information in the data control platform；If
It is not stored to have the data source information, then according to the data source information, the pass of the tentative connection business side to be synchronized
It is type database, if tentative successful connection, is verified.
As it can be seen that above-mentioned validation verification is divided into two steps:
The first step is to judge the data source information whether is stored in data control platform, such as search from middle database
Whether its data source information is had.It can specifically be existed according to information such as title, the addresses of the relevant database in data source information
Data control platform is searched, if the information such as the title of identical relevant database, address can be found, illustrates data
Control platform has existed its data source information, terminates structure synchronization process at this time.If not finding identical relationship type number
According to information such as the title in library, addresses, then explanation is to carry out structure synchronization for the first time, can continue to execute second step.As it can be seen that first
Step is uniqueness verifying.
Second step is to carry out tentative connection to the relevant database of business side, and relevant database specifically can be used
The tentative connection business side of information such as title, address, user name, password relevant database, can specifically pass through Java
Language is realized.If tentative successful connection, user's data source information of typing in the data input interface in step S22 is
Effectively.
Structure synchronization method provided by the embodiments of the present application, it is first determined the relevant database of business side to be synchronized
Type, so the type based on relevant database obtains data source information and metadata information, and then realizes that table structure is unitized,
It can be seen that the application is directed to different types of relevant database, its table knot can be synchronized to default in Hadoop database
In standard table structure, that is, it is synchronized to unified standard table structure, is convenient for operation management.The application not uses Manual synchronization, can
To reduce the workload of manual synchronization in conventional method, the generation of manual operation accident is reduced.Meanwhile periodically monitoring relationship type
Whether the table structure of database changes, if table structure changes, the metadata information is converted to described in
Preset standard table structure and the default mapping table are updated, and automatic synchronization table structure may be implemented, avoid the table of business side
Structure change, which synchronizes follow-up data, to be impacted.
As shown in figure 3, in one embodiment, the structure for providing a kind of relevant database to Hadoop database is same
Device 30 is walked, which can integrate in above-mentioned computer equipment (for example, data control platform), specifically
May include:
First determining module 31, the type of the relevant database for determining business side to be synchronized；
Second determining module 32, for number needed for determining the relevant database progress structure synchronization for the type
According to source information；
Data extraction module 33, for connecting the relationship type of the business side to be synchronized according to the data source information
Database, and metadata information is extracted from the relevant database being connected to；
Thaumatropy module 34, for converting Hadoop database in data control platform for the metadata information
Preset standard table structure, and by the word between the relevant database of the business side to be synchronized and the Hadoop database
Section mapping relations are stored into default mapping table；
Variation monitoring module 35, the table structure of the relevant database for periodically monitoring the business side to be synchronized
Whether change, if table structure changes, the preset standard table structure that the metadata information is converted to
It is updated with the default mapping table.
In some embodiments, described device further include: information storage module, for by the data source information and/or
The metadata information is stored into the default middle database of the data control platform.
In some embodiments, described device further include: validation verification module, for connecting institute in data extraction module
Before the relevant database for stating business side to be synchronized, according to the data source information, to the business side to be synchronized
Relevant database carries out validation verification, after being verified, data extraction module is allowed to execute described in the connection to same
The step of relevant database of the business side of step.
In some embodiments, validation verification module is specifically used for: according to the data source information, judging the data
Whether the data source information is stored in control platform；If not stored have the data source information, according to the data source
Information, the relevant database of the tentative connection business side to be synchronized, if tentative successful connection, is verified.
In some embodiments, the preset standard table structure includes identity field and information field；It is corresponding, the knot
Structure conversion module is used for: the major key field in the metadata information is stored to the identity of the preset standard table structure
Field stores remaining field in the metadata information to the information field of the preset standard table structure, and institute
Remaining field is stated to store in a manner of keyword/correspondence numerical value into the information field.
In some embodiments, the first determining module is specifically used for: in response to any one relationship in default selection interface
The corresponding relevant database type of the relevant database kind Class Options selected is made in the selection of type database kind Class Options
For the type of the relevant database of the business side to be synchronized, multiple relevant databases are provided in the selection interface
Kind Class Options.
In some embodiments, the second determining module is specifically used for: in the relational data for determining business side to be synchronized
After the type in library, provide the type corresponding data input interface, for the corresponding data source information of user's typing；By institute
It states the data source information being logged on data input interface and carries out structure synchronization as the relevant database for the type
Required data source information.
Structure synchronization device provided by the present application, first the first determining module determine the relationship type number of business side to be synchronized
According to the type in library, then remaining module obtains data source information and metadata information based on the type of relevant database, in turn
Realize that table structure is unitized, it is seen that the application is directed to different types of relevant database, can be synchronized to its table knot
In preset standard table structure in Hadoop database, that is, it is synchronized to unified standard table structure, is convenient for operation management.The application
Not use Manual synchronization, it is possible to reduce the workload of manual synchronization in conventional method reduces the generation of manual operation accident.Together
When, whether the table structure of variation monitoring module periodically monitoring relevant database changes, if table structure changes,
The preset standard table structure and the default mapping table be converted to the metadata information is updated, and may be implemented
Automatic synchronization table structure avoids the table structure change of business side from synchronizing follow-up data and impacts.
In some embodiments, propose a kind of computer equipment, the computer equipment include memory, processor and
It is stored in the computer program that can be run on the memory and on the processor, the processor executes the computer
The type for determining the relevant database of business side to be synchronized is performed the steps of when program；It determines for the type
Data source information needed for relevant database carries out structure synchronization；According to the data source information, connect described to be synchronized
The relevant database of business side, and metadata information is extracted from the relevant database being connected to；The metadata is believed
Breath is converted into the preset standard table structure of Hadoop database in data control platform, and by the pass of the business side to be synchronized
It is that field mapping relations between type database and the Hadoop database are stored into default mapping table；Periodically monitoring institute
Whether the table structure for stating the relevant database of business side to be synchronized changes, if table structure changes, to described
The preset standard table structure and the default mapping table that metadata information is converted to are updated.
In some embodiments, it also performs the steps of when the processor executes the computer program by the number
It stores according to source information and/or the metadata information into the default middle database of the data control platform.
In some embodiments, it is also performed the steps of according to when the processor executes the computer program
Data source information carries out validation verification to the relevant database of the business side to be synchronized, after being verified, allows
The step of executing the relevant database of the connection business side to be synchronized.
In some embodiments, the relevant database to the business side to be synchronized that the processor executes
Carry out validation verification, comprising: according to the data source information, judge whether be stored with the number in the data control platform
According to source information；If not stored have the data source information, according to the data source information, tentative connection is described to be synchronized
The relevant database of business side, if tentative successful connection, is verified.
In some embodiments, the preset standard table structure includes identity field and information field；The processor is held
The capable preset standard table structure for converting the metadata information to Hadoop database in data control platform, packet
It includes: the major key field in the metadata information is stored to the identity field of the preset standard table structure, it will be described
Remaining field in metadata information is stored to the information field of the preset standard table structure, and remaining described field with
Keyword/correspondence numerical value mode is stored into the information field.
In some embodiments, the relevant database for the determination business side to be synchronized that the processor executes
Type, comprising: in response to presetting the selection of any one relevant database kind Class Options in selection interface, the pass that will be selected
It is relevant database of the corresponding relevant database type of type database kind Class Options as the business side to be synchronized
Type, be provided with multiple relevant database kind Class Options in the selection interface.
In some embodiments, the determination that the processor executes is carried out for the relevant database of the type
Data source information needed for structure synchronization, comprising: after determining the type of relevant database of business side to be synchronized, provide
The corresponding data input interface of one type, for the corresponding data source information of user's typing；By the data input interface
On the data source information that is logged carry out structure synchronization as the relevant database for the type needed for data source believe
Breath.
The knot of the beneficial effect of computer equipment provided by the present application and above-mentioned relation type database to Hadoop database
Structure synchronous method is identical with device, and which is not described herein again.
In one embodiment it is proposed that a kind of storage medium for being stored with computer-readable instruction, this is computer-readable
When instruction is executed by one or more processors, so that one or more processors execute following steps: determining industry to be synchronized
The type of the relevant database of business side；Data needed for determining the relevant database progress structure synchronization for the type
Source information；According to the data source information, the relevant database of the business side to be synchronized is connected, and from the pass being connected to
It is to extract metadata information in type database；Hadoop database in data control platform is converted by the metadata information
Preset standard table structure, and by the word between the relevant database of the business side to be synchronized and the Hadoop database
Section mapping relations are stored into default mapping table；Periodically monitor the table knot of the relevant database of the business side to be synchronized
Whether structure changes, if table structure changes, the preset standard table knot that the metadata information is converted to
Structure and the default mapping table are updated.
In some embodiments, when computer-readable instruction is executed by one or more processors, also make one or more
A processor executes following steps: the data source information and/or the metadata information being stored to the data to manage and put down
In the default middle database of platform.
In some embodiments, it is also performed the steps of according to when the processor executes the computer program
Data source information carries out validation verification to the relevant database of the business side to be synchronized, after being verified, allows
The step of executing the relevant database of the connection business side to be synchronized.
In some embodiments, the pass to the business side to be synchronized that one or more of processors execute
It is that type database carries out validation verification, comprising: according to the data source information, judge whether deposit in the data control platform
Contain the data source information；If not stored have the data source information, according to the data source information, tentative connection institute
The relevant database of business side to be synchronized is stated, if tentative successful connection, is verified.
In some embodiments, the preset standard table structure includes identity field and information field；It is one or more
The preset standard for converting the metadata information to Hadoop database in data control platform that a processor executes
Table structure, comprising: store the major key field in the metadata information to the identity word of the preset standard table structure
Section stores remaining field in the metadata information to the information field of the preset standard table structure, and described
Remaining field is stored in a manner of keyword/correspondence numerical value into the information field.
In some embodiments, the relationship for the determination business side to be synchronized that one or more of processors execute
The type of type database, comprising:, will in response to presetting the selection of any one relevant database kind Class Options in selection interface
Pass of the corresponding relevant database type of relevant database kind Class Options selected as the business side to be synchronized
It is the type of type database, multiple relevant database kind Class Options is provided in the selection interface.
In some embodiments, the determination that one or more of processors execute is directed to the relationship type of the type
Data source information needed for database carries out structure synchronization, comprising: determining the relevant database of business side to be synchronized
After type, provide the type corresponding data input interface, for the corresponding data source information of user's typing；By the letter
The data source information being logged on breath input interface carries out needed for structure synchronization as the relevant database for the type
Data source information.
The structure of the beneficial effect of storage medium provided by the present application and above-mentioned relation type database to Hadoop database
Synchronous method is identical with device, and which is not described herein again.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, which can be stored in a computer-readable storage and be situated between
In matter, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, storage medium above-mentioned can be
The non-volatile memory mediums such as magnetic disk, CD, read-only memory (Read-Only Memory, ROM) or random storage note
Recall body (Random Access Memory, RAM) etc..
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality
It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.Therefore, the scope of protection of the patent of the invention shall be subject to the appended claims.
Claims (10)
1. a kind of relevant database is to the structure synchronization method of Hadoop database, which is characterized in that the described method includes:
Determine the type of the relevant database of business side to be synchronized；
Data source information needed for determining the relevant database progress structure synchronization for the type；
According to the data source information, the relevant database of the business side to be synchronized is connected, and from the relationship being connected to
Metadata information is extracted in type database；
Convert the metadata information to the preset standard table structure of Hadoop database in data control platform, and will be described
Field mapping relations between the relevant database of business side to be synchronized and the Hadoop database, which are stored to default, reflects
In firing table；
Whether the table structure for periodically monitoring the relevant database of the business side to be synchronized changes, if table structure is sent out
Changing, then the preset standard table structure being converted to the metadata information and the default mapping table carry out more
Newly.
2. the method according to claim 1, wherein further include:
The data source information and/or the metadata information are stored into the default middle database of the data control platform.
3. the method according to claim 1, wherein the relationship type number of the connection business side to be synchronized
Before library, the method also includes:
According to the data source information, validation verification is carried out to the relevant database of the business side to be synchronized, is being tested
The step of demonstrate,proving after passing through, allowing to execute the relevant database of the connection business side to be synchronized.
4. according to the method described in claim 3, it is characterized in that, the relational data to the business side to be synchronized
Library carries out validation verification, comprising:
According to the data source information, judge whether be stored with the data source information in the data control platform；If not depositing
The data source information is contained, then according to the data source information, the relationship type of the tentative connection business side to be synchronized
Database, if tentative successful connection, is verified.
5. the method according to claim 1, wherein the preset standard table structure includes identity field and information
Field；It is corresponding, the preset standard table for converting the metadata information to Hadoop database in data control platform
Structure, comprising: the major key field in the metadata information is stored to the identity field of the preset standard table structure,
Remaining field in the metadata information is stored to the information field of the preset standard table structure, and it is described remaining
Field is stored in a manner of keyword/correspondence numerical value into the information field.
6. the method according to claim 1, wherein the relevant database of determination business side to be synchronized
Type, comprising:
In response to presetting the selection of any one relevant database kind Class Options in selection interface, the relationship type number that will be selected
Type according to the corresponding relevant database type of library kind Class Options as the relevant database of the business side to be synchronized,
Multiple relevant database kind Class Options are provided in the selection interface.
7. according to the method described in claim 6, it is characterized in that, the determination for the type relevant database into
Data source information needed for row structure synchronization, comprising:
After determining the type of relevant database of business side to be synchronized, provide the type corresponding data input circle
Face, for the corresponding data source information of user's typing；Using the data source information being logged on the data input interface as needle
Data source information needed for carrying out structure synchronization to the relevant database of the type.
8. a kind of relevant database is to the structure synchronization device of Hadoop database, which is characterized in that described device includes:
First determining module, the type of the relevant database for determining business side to be synchronized；
Second determining module is believed for data source needed for determining the relevant database progress structure synchronization for the type
Breath；
Data extraction module, for connecting the relevant database of the business side to be synchronized according to the data source information,
And metadata information is extracted from the relevant database being connected to；
Thaumatropy module, for converting the metadata information to the pre- bidding of Hadoop database in data control platform
Quasi- table structure, and the field between the relevant database of the business side to be synchronized and the Hadoop database is mapped
Relationship is stored into default mapping table；
Variation monitoring module, for periodically monitoring whether the table structure of relevant database of the business side to be synchronized is sent out
Changing, if table structure changes, the preset standard table structure that be converted to the metadata information and described
Default mapping table is updated.
9. a kind of computer equipment, which is characterized in that including memory and processor, being stored with computer in the memory can
Reading instruction, when the computer-readable instruction is executed by the processor, so that the processor executes such as claim 1 to 7
Any one of structure synchronization method described in claim the step of.
10. a kind of storage medium for being stored with computer-readable instruction, which is characterized in that the computer-readable instruction is by one
Or multiple processors are when executing, so that one or more processors are executed as described in any one of claims 1 to 7 claim
The step of structure synchronization method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910341135.7A CN110245145A (en) | 2019-04-25 | 2019-04-25 | Structure synchronization method and apparatus of the relevant database to Hadoop database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910341135.7A CN110245145A (en) | 2019-04-25 | 2019-04-25 | Structure synchronization method and apparatus of the relevant database to Hadoop database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110245145A true CN110245145A (en) | 2019-09-17 |
Family
ID=67883389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910341135.7A Pending CN110245145A (en) | 2019-04-25 | 2019-04-25 | Structure synchronization method and apparatus of the relevant database to Hadoop database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245145A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704417A (en) * | 2019-10-10 | 2020-01-17 | 南方电网数字电网研究院有限公司 | Metadata management method, equipment and storage medium |
| CN111290813A (en) * | 2020-01-21 | 2020-06-16 | 福建省农村信用社联合社 | Software interface field data standardization method, device, equipment and medium |
| CN111552746A (en) * | 2020-04-30 | 2020-08-18 | 北京思特奇信息技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN111767267A (en) * | 2020-06-18 | 2020-10-13 | 杭州数梦工场科技有限公司 | Metadata processing method and device and electronic equipment |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112434004A (en) * | 2020-10-28 | 2021-03-02 | 金蝶云科技有限公司 | Data migration method and device for heterogeneous system, computer equipment and storage medium |
| CN113641652A (en) * | 2021-08-09 | 2021-11-12 | 挂号网（杭州）科技有限公司 | Data synchronization method, device and system and server |
-
2019
- 2019-04-25 CN CN201910341135.7A patent/CN110245145A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704417A (en) * | 2019-10-10 | 2020-01-17 | 南方电网数字电网研究院有限公司 | Metadata management method, equipment and storage medium |
| CN111290813A (en) * | 2020-01-21 | 2020-06-16 | 福建省农村信用社联合社 | Software interface field data standardization method, device, equipment and medium |
| CN111290813B (en) * | 2020-01-21 | 2022-09-23 | 福建省农村信用社联合社 | Software interface field data standardization method, device, equipment and medium |
| CN111552746A (en) * | 2020-04-30 | 2020-08-18 | 北京思特奇信息技术股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN111767267A (en) * | 2020-06-18 | 2020-10-13 | 杭州数梦工场科技有限公司 | Metadata processing method and device and electronic equipment |
| CN112434004A (en) * | 2020-10-28 | 2021-03-02 | 金蝶云科技有限公司 | Data migration method and device for heterogeneous system, computer equipment and storage medium |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN113641652A (en) * | 2021-08-09 | 2021-11-12 | 挂号网（杭州）科技有限公司 | Data synchronization method, device and system and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245145A (en) | Structure synchronization method and apparatus of the relevant database to Hadoop database | |
| US10963800B2 (en) | Service layer augmentation of response to semantically-informed query of arbitrary external data sources | |
| CN102968374B (en) | A kind of data warehouse method of testing | |
| WO2018095299A1 (en) | Time sequence data management method, device and apparatus | |
| CN105989076A (en) | Data statistical method and device | |
| CN110321383A (en) | Big data platform method of data synchronization, device, computer equipment and storage medium | |
| CN102236672A (en) | Method and device for importing data | |
| CN109542907A (en) | Database caches construction method, device, computer equipment and storage medium | |
| CN108319661A (en) | A kind of structured storage method and device of spare part information | |
| CN108647357A (en) | The method and device of data query | |
| CN107832448A (en) | Database operation method, device and equipment | |
| WO2014137258A1 (en) | Selection of data storage settings for an application | |
| CN111046081A (en) | Access method and system for industrial time sequence data | |
| CN114547204A (en) | Data synchronization method and device, computer equipment and storage medium | |
| CN106250501A (en) | Report processing method and reporting system | |
| CN104166650B (en) | Data storage device and date storage method | |
| KR101588375B1 (en) | Method and system for managing database | |
| CN111125226B (en) | Configuration data acquisition method and device | |
| CN110502529B (en) | Data processing method, device, server and storage medium | |
| CN105446981B (en) | Map of website generation method, access method and device | |
| CN111309674A (en) | System and method for file management by mobile computing device | |
| CN108964990A (en) | Implementor name display methods and device in a kind of multiple equipment management system | |
| CN116107801A (en) | Transaction processing method and related product | |
| KR101218927B1 (en) | Method for Monitoring User Sensory Performance of Web Service and Recording Medium Therefore and Computing Device | |
| CN111143328A (en) | Agile business intelligent data construction method, system, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |