CN107515933A - Change method, server, system and the storage medium of source data database table structure - Google Patents
Change method, server, system and the storage medium of source data database table structure Download PDFInfo
- Publication number
- CN107515933A CN107515933A CN201710755733.XA CN201710755733A CN107515933A CN 107515933 A CN107515933 A CN 107515933A CN 201710755733 A CN201710755733 A CN 201710755733A CN 107515933 A CN107515933 A CN 107515933A
- Authority
- CN
- China
- Prior art keywords
- table structure
- database
- metadata information
- target database
- modification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/235—Update request formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses in the method for modification source data database table structure, server, system and storage medium, the method of the automatic modification source data database table structure preserves the metadata information of at least nearest two days by the metadata information that receives each source database or target database pushes daily；The same day metadata information and metadata information yesterday of storage are contrasted afterwards, judge whether table structure has variation；When judgement table structure has variation, according to table structural change information, the script of the table structure of generation modification target database, and the script of the table structure is sent and performed into target database.The present invention during source database carries out data syn-chronization to target database, make target database table structure be automatically performed relative to source database table structure to when changing, so as to both reducing human cost, reduce error rate again.
Description
Technical field
The present invention relates to data synchronization technology field, and in particular to the method for automatic modification source data database table structure, service
Device, system and storage medium.
Background technology
Before data analysis being carried out to operation system, it is necessary first to obtain the data message of operation system.Before not influenceing
The performance of network is held, and the data of front network disperse to associate, impact analysis, therefore need data syn-chronization to data warehouse
Collected, be easy to concentration to be analyzed and handled.But the content of operation system can be because the change of business demand be by more
Change, i.e., the situation of source database table structural modification can be frequently encountered during being synchronized to the data of operation system.
Now, analysis personnel are generally required exploitation version is synchronously established according to source database, in same version time modification
Synchronization program, including table script is built, synchronous script etc., but at present, this operating process is complicated, and timing modification can expend more
Exploitation manpower, and easily malfunction.
Therefore, prior art has yet to be improved and developed.
The content of the invention
In view of in place of above-mentioned the deficiencies in the prior art, it is an object of the invention to provide a kind of automatic modification source database table
Method, server, system and the storage medium of structure, the table structure of target database can be made to be automatically performed relative to source database
Table structure to when changing, so as in data synchronization process, can both reduce human cost, error rate can be reduced again.
In order to achieve the above object, this invention takes following technical scheme：
A kind of method of automatic modification source data database table structure, it comprises the following steps：
Receive each source database or metadata information that target database pushes daily, and preserve the member of at least nearest two days
Data message；
The same day metadata information and metadata information yesterday of storage are contrasted, judges whether table structure has variation；
When table structure has variation, according to table structural change information, the script of the table structure of target database is changed in generation, and will
The script of the table structure is sent to be performed into target database.
In the method for described automatic modification source data database table structure, it is described contrast storage same day metadata information with
Yesterday, metadata information, judged whether table structure has the step of variation to include：
Application server start metadata contrast modification program, separate in the metadata information of storage yesterday metadata information and
Today metadata information；
The column name of each table owner of metadata information is contrasted one by one, judges each column name whether in metadata information yesterday
With exist in today metadata information.
It is described when table structure has variation in the method for described automatic modification source data database table structure, according to table knot
Structure transition information, the script of the table structure of generation modification target database, and the script of the table structure is sent to number of targets
Include according to the step of execution in storehouse：
If a certain column name exists in yesterday metadata information and is not present in today metadata information, generation modification
The table structure of target database and the script that this column name is set to sky in correspondence position, send into target database to perform and delete
Division operation；
If a certain column name is not present in yesterday metadata information and existed in today metadata information, the row are extracted
Title, the table structure of generation modification target database simultaneously increase the script of the column name on correspondence position, sent to number of targets
Operated according to increase is performed in storehouse.
In the method for described automatic modification source data database table structure, pushed daily in the reception target database
Metadata information, and before the step of metadata information of preservation at least nearest two days, described method also includes：
Metadata information is synchronized in application server by the daily timing of each source database.
In the method for described automatic modification source data database table structure, pushed daily in the reception target database
Metadata information, and before the step of metadata information of preservation at least nearest two days, described method also includes：
Metadata information is synchronized in target database by the daily timing of each source database；
Target database pushes to metadata information in application server.
It is described when table structure has variation in the method for described automatic modification source data database table structure, according to table knot
Structure transition information, the script of the table structure of generation modification target database, and the script of the table structure is sent to number of targets
After the step of execution in storehouse, in addition to：
Terminate the timer-triggered scheduler task of current sync program, change synchronization program to adapt to the table structure of new source database；
The timer-triggered scheduler task of new synchronization program is initiated to source database and/or target database.
In the method for described automatic modification source data database table structure, the source database is oracle database,
MySQL database or PostgreSQL databases.
A kind of application server of automatic modification source data database table structure, including：Processor, memory, communication bus;Institute
State that be stored with memory can be by the computer-readable program of the computing device；
The communication bus realizes the connection communication between processor and memory；
The side of automatic modification source data database table structure described above is realized described in the computing device during computer-readable program
Step in method.
A kind of computer-readable recording medium, the computer-readable recording medium storage have one or more program,
One or more of programs automatic can be repaiied by one or more computing device with realize described in as above any one
Change the step in the method for source data database table structure.
A kind of system of automatic modification source data database table structure, including several source databases, target database and as above
The application server of described modification source data database table structure；
Each source database, metadata information is synchronized in target database for daily timing；
Target database, for metadata information to be pushed in application server；
The application server, for the metadata information for receiving each source database or target database pushes daily, and
Preserve the metadata information of at least nearest two days；And the same day metadata information of contrast storage and metadata information yesterday, sentence
Whether disconnected table structure has variation；When table structure has variation, according to table structural change information, the table of generation modification target database
The script of structure, and the script of the table structure is sent and performed into target database.
Compared to prior art, the method for automatic modification source data database table structure provided by the invention, server, system and
In storage medium, the method for the automatic modification source data database table structure is by receiving each source database or target database
The metadata information pushed daily, and preserve the metadata information of at least nearest two days；The same day metadata of storage is contrasted afterwards
Information and metadata information yesterday, judge whether table structure has variation；When judgement table structure has variation, according to table structural change
Information, the script of the table structure of generation modification target database, and the script of the table structure is sent into target database
Perform so that the table structure of target database be automatically performed relative to source database table structure to when changing so that
It in data synchronization process, can both reduce human cost, error rate can be reduced again.
Brief description of the drawings
Fig. 1 is the application environment schematic diagram of the method for automatic modification source data database table structure provided by the invention.
Fig. 2 is the flow chart of the method for automatic modification source data database table structure provided by the invention.
Fig. 3 is the flow chart of step S20 in the automatic method for changing source data database table structure provided by the invention.
Fig. 4 is the flow chart of step S30 in the automatic method for changing source data database table structure provided by the invention
Fig. 5 is the running environment signal of the preferred embodiment of the application server of modification source data database table structure provided by the invention
Figure.
Fig. 6 is the functional module of the preferred embodiment for the application server that source data database table structure is changed in present invention installation
Figure.
Fig. 7 is the structured flowchart of the system of modification source data database table structure provided by the invention.
Embodiment
In view of operating process is complicated in the data synchronization process of database in the prior art, labor intensive is more and error-prone
Shortcoming, it is an object of the invention to provide a kind of method, server, system and the storage of automatic modification source data database table structure to be situated between
Matter, by make the table structure of target database be automatically performed relative to source database table structure to when changing so that
In data synchronization process, human cost can be both reduced, error rate can be reduced again.
To make the purpose of the present invention, technical scheme and effect clearer, clear and definite, develop simultaneously embodiment pair referring to the drawings
The present invention is further described.It should be appreciated that specific embodiment described herein is not used to only to explain the present invention
Limit the present invention.
Referring to Fig. 1, its application environment signal for the method for automatic modification source data database table structure provided by the invention
Figure.One or more application programs can be installed to handle related data in Fig. 1, in application server.In the present embodiment, can lead to
Cross application server and receive the metadata information from source database or target database, metadata information is deposited in the form of a list
Storage in the application server, and is contrasted current meta data information with the metadata information stored before by application server, when
When table structure has variation, according to table structural change information, generation changes the modification script of target database into target database,
Perform modification script in target database to modify to table structure, so as to realize that the table structure of target database is automatically performed
Relative to source database table structure to when changing.
Referring to Fig. 2, the method for automatic modification source data database table structure provided by the invention, in this way applied to Fig. 1
In application server exemplified by illustrate, comprise the following steps：
The metadata information that S10, each source database of reception or target database push daily, and preserve at least nearest two days
Metadata information.
Preferably, the source database is oracle database（Oracle database also known as Oracle RDBMS, or referred to as
Oracle, it is a relational database management system of Oracle）, MySQL（MySQL is the small-sized of open source code
Correlation data base management system）Database or PostgreSQL（PostgreSQL is a free Object-Relation data
Storehouse server）Database, the target database are hive（Hive is a Tool for Data Warehouse based on Hadoop）Data
Storehouse.These databases are common and easily operated data base management system and instrument, are easy in the present embodiment to data
Analyzed and handled.
In the present embodiment, each source database or target data are received by additionally setting application server simultaneously
The metadata information that storehouse pushes daily, and the follow-up processing to metadata information is completed by application server.Metadata is believed
Breath is mainly the various information of front end service system, and the various identity and deposit information of the client of such as banking system are first
Data message is embodied in by table owner, table name, table note, column name, the metadata of the part such as row annotation and row order composition
In information table.
In addition, before step S10, in addition to metadata information is synchronized to using clothes by the daily timing of each source database
The step being engaged in device.Now, built and contacted with application server respectively by each source database, and in the hive for analysis
Start Oozie in data warehouse（Oozie is a kind of Java web applications）Timing Synchronization task, daily 2:00 AM is by before
Hold the metadata information in the multiple source datas storehouse of operation system to be synchronized in hive data warehouses and application server, at this moment apply
Server directly receives the metadata information pushed daily from each source database, and does not interfere with the day of application server
Regular data processing.
Or before step S10, also metadata information first can be synchronized to target by the daily timing of each source database
In database, then by target database metadata information is pushed in application server.Now, by by for analysis
Hive data warehouses are built with application server to be contacted, and starts Oozie Timing Synchronization tasks, and daily 2:00 AM is by front end industry
After the metadata information in the multiple source datas storehouse of business system is synchronized to hive data warehouses, then is pushed to and answered by hive data warehouses
With in server.At this moment only with advance application server and the foundation of single hive data warehouses is contacted, without making each source
Database is built with application server contact respectively, it is not required that increased source is directed in the case of source database quantity is increased
Database builds contact again, greatlys save time and the energy of developer.
S20, the same day metadata information and metadata information yesterday of contrast storage, judge whether table structure has variation.
After application server have received the metadata information on the same day, its metadata for pre-setting startup contrast is repaiied
Change program, by same day metadata information compared with the previous day metadata information, obtain each source database table structure it is newly-increased,
Modification and deletion information.Referring to Fig. 2, step in its method for automatic modification source data database table structure provided by the invention
S20 flow chart.
As shown in figure 3, the step S20 includes：
S21, application server start metadata contrast modification program, separate metadata letter yesterday in the metadata information of storage
Breath and metadata information today；
Whether the column name of S22, one by one each table owner of contrast metadata information, judge each column name in metadata yesterday
Exist in information and today metadata information.
Table structure defines the field of a table, type, major key, external key, index, and these basic attributes constitute database
Table structure.In operation system, its table structural change is mainly reflected in the change of column name, is such as increased or be have modified such as visitor
The gender information column at family, Income situation Information etc., therefore mainly for each table owner of metadata information in the present embodiment
Column name contrasted.
S30, when table structure has variation, according to table structural change information, the table structure of generation modification target database
Script, and the script of the table structure is sent and performed into target database.Simultaneously referring to Fig. 3, it is provided for the present invention
Automatic modification source data database table structure method in step S30 flow chart.
As shown in figure 4, the step S30 includes：
If S31, a certain column name exist in yesterday metadata information and be not present in today metadata information, generate
Change the table structure of target database and this column name is set to the script of sky in correspondence position, send and held into target database
Row deletion action；
If S32, a certain column name are not present in yesterday metadata information and existed in today metadata information, extract
The column name, the table structure of generation modification target database simultaneously increase the script of the column name on correspondence position, sent to mesh
Mark and increase operation is performed in database.
Specifically, if there is step S31 situation, we can draw the row name that source data database table structure now be present
Claim situation about being deleted or modified, thus contrast modification program will generate the table structure of changing hive databases and correspondence position by this
Column name is set to the script of sky, then will send into hive databases and perform, the deletion pair on the object table of hive databases
The row answered.
If there is step S32 situation, we can draw now exist source data database table structure column name increase or
The situation of modification, therefore this row will be extracted by contrasting modification program, the table structure of generation modification hive databases simultaneously increases in correspondence position
The script for adding this to arrange, it then will send into hive databases and perform, and increase corresponding arrange on the object table of hive databases.
Further, after step S30, in addition to the timer-triggered scheduler task of current sync program is terminated, modification is synchronous
Program is to adapt to the table structure of new source database；New synchronization program is initiated to source database and/or target database
The step of timer-triggered scheduler task.It is specially that kill falls after the completion of the modification to hive databases in the present embodiment（Terminate）
Oozie scheduling originally, and Oozie timed tasks are generated, call OozieAPI to send to Oozie and dispatch execution.
Please continue to refer to Fig. 5, Fig. 5 is the internal structure schematic diagram of one embodiment Computer equipment.The computer is set
Standby can be terminal or server, wherein, terminal can be smart mobile phone, tablet personal computer, notebook computer, desk-top
Computer, personal digital assistant and Wearable etc. have the electronic equipment of communication function.Server can be independent service
Device or multiple server groups into server cluster.Reference picture 5, the computer equipment include connecting by system bus
Processor, non-volatile memory medium, built-in storage and the network interface connect.Wherein, the non-volatile of the computer equipment is deposited
Storage media can storage program area and computer-readable program, when the computer-readable program is performed, may be such that processor is held
The method that the seed nucleus of row one protects difficulty prediction.The processor of the computer equipment is used to provide calculating and control ability, and support is whole
The operation of computer equipment.Computer-readable program can be stored in the built-in storage, the computer-readable program is by processor
During execution, the method that the seed nucleus of computing device one protects difficulty prediction may be such that.The network interface of computer equipment is used to carry out net
Network communicates, and such as sends the task dispatching of distribution.It will be understood by those skilled in the art that the structure shown in Fig. 5, only with this Shen
Please the related part-structure of scheme block diagram, do not form the limit of the computer equipment being applied thereon to application scheme
Fixed, specific computer equipment can include, than more or less parts shown in figure, either combining some parts or tool
There is different part arrangements.
The present invention also provides a kind of application server of automatic modification source data database table structure, and it includes processor 10, deposited
Reservoir 20 and display 30.Fig. 5 illustrate only the members of the application server of modification source data database table structure, but should manage
Solution is, it is not required that implements all components shown, the more or less component of the implementation that can be substituted.
The memory 20 can be the application server of the modification source data database table structure in certain embodiments
The internal storage unit of each component, such as the hard disk or internal memory of application server.The memory 20 is in further embodiments
It can also be the External memory equipment of each component of the application server of the modification source data database table structure, such as change source number
The plug-in type hard disk being equipped with each component according to the application server of database table structure, intelligent memory card（Smart Media Card,
SMC）, secure digital（Secure Digital, SD）Card, flash card（Flash Card）Deng.Further, the memory
20 can also both include the internal storage unit of the application server of the modification source data database table structure or including external storage
Equipment.The memory 20 be used for store be installed on it is described modification source data database table structure application server application software and
Various types of data, such as program code of application server of the installation modification source data database table structure etc..The memory 20
It can be also used for temporarily storing the data that has exported or will export.In one embodiment, it is stored with memory 20
Source database table structure program 40 is changed, the modification source database table structure program 40 can be performed by processor 10, so as to real
The modification source database table structural approach of each embodiment of existing the application.
The processor 10 can be a central processing unit in certain embodiments（Central Processing Unit,
CPU）, microprocessor or other data processing chips, for running the program code stored in the memory 20 or processing number
According to, such as perform described purview certification method etc..
The display 30 can be light-emitting diode display, liquid crystal display, touch-control liquid crystal display in certain embodiments
And OLED（Organic Light-Emitting Diode, Organic Light Emitting Diode）Touch device etc..The display 30 is used
In being shown in the information that is handled in the wechat customer action feedback device and for showing visual user interface, such as
Assignment information interface, certification report interface etc..The part 10-30 of the wechat customer action feedback device passes through system bus phase
Mutual communication.
In one embodiment, it is real when processor 10, which performs, changes source database table structure program 40 in the memory 20
Existing following steps：
Receive each source database or metadata information that target database pushes daily, and preserve the member of at least nearest two days
Data message；
The same day metadata information and metadata information yesterday of storage are contrasted, judges whether table structure has variation；
When table structure has variation, according to table structural change information, the script of the table structure of target database is changed in generation, and will
The script of the table structure is sent to be performed into target database.
Further, in the application server of described modification source data database table structure, the same day of the contrast storage
Metadata information and metadata information yesterday, judge whether table structure has the step of variation to include：
Application server start metadata contrast modification program, separate in the metadata information of storage yesterday metadata information and
Today metadata information；
The column name of each table owner of metadata information is contrasted one by one, judges each column name whether in metadata information yesterday
With exist in today metadata information.
The column name of each table owner for contrasting metadata information one by one, judge each column name whether in member yesterday
Include after step present in data message and today metadata information：
If a certain column name exists in yesterday metadata information and is not present in today metadata information, generation modification
The table structure of target database and the script that this column name is set to sky in correspondence position, send into target database to perform and delete
Division operation；
If a certain column name is not present in yesterday metadata information and existed in today metadata information, the row are extracted
Title, the table structure of generation modification target database simultaneously increase the script of the column name on correspondence position, sent to number of targets
Operated according to increase is performed in storehouse.
In the metadata information that the reception target database pushes daily, and preserve the metadata letter of at least nearest two days
Before the step of breath, described method also includes：
Metadata information is synchronized in application server by the daily timing of each source database.
Or the metadata information pushed daily in the reception target database, and preserve the member of at least nearest two days
Before the step of data message, described method also includes：
Metadata information is synchronized in target database by the daily timing of each source database；
Target database pushes to metadata information in application server.
It is described when table structure has variation, according to table structural change information, the table structure of generation modification target database
Script, and by the script of the table structure send into target database perform the step of after, in addition to：
Terminate the timer-triggered scheduler task of current sync program, change synchronization program to adapt to the table structure of new source database；
The timer-triggered scheduler task of new synchronization program is initiated to source database and/or target database.
Referring to Fig. 6, it changes the preferable implementation of the application server of source database table structure program for present invention installation
The functional block diagram of example.In the present embodiment, installation modification source database table structure program can be divided into one or more
Module, one or more of modules are stored in the memory, and by one or more processors（The present embodiment is
The processor）It is performed, to complete the present invention.For example, in figure 6, the application of installation modification source database table structure program
Server can be divided into receiving module 31, comparative analysis module 32 and feedback modified module 33.Module alleged by the present invention
It is the series of computation machine program segment for referring to complete specific function, than program more suitable for describing the modification source database table
Implementation procedure of the structure program in the application server of the modification source data database table structure.Describe specific place of matchmakers below
State module 31-33 function.
Receiving module 31, for the metadata information for receiving each source database or target database pushes daily, and
Preserve the metadata information of at least nearest two days；
Comparative analysis module 32, for contrasting the same day metadata information and metadata information yesterday of application server for storage, sentence
Whether disconnected table structure has variation；
Modified module 33 is fed back, in the case of having variation in table structure, according to table structural change information, generation modification target
The script of the table structure of database, and the script of the table structure is sent and performed into target database.
The comparative analysis module 32 includes separative element and comparison unit, wherein：
The separative element, the metadata information and member today yesterday in the metadata information for separating application server for storage
Data message.
The comparison unit, the column name of each table owner for contrasting metadata information one by one, judges each row name
Claim whether yesterday metadata information and today metadata information in exist.
The feedback modified module 33 includes deleting unit and adding unit, wherein：
The deletion unit, for existing when a certain column name in yesterday metadata information and in today metadata information
In the absence of when, generation modification target database table structure simultaneously correspondence position by this column name be set to sky script, send extremely
Deletion action is performed in target database.
The adding unit, for being not present and in metadata today in yesterday metadata information when a certain column name
In the presence of in information, the column name is extracted, the table structure of generation modification target database simultaneously increases the row name on correspondence position
The script of title, send into target database and perform increase operation.
Further, the feedback modified module 33 also includes：Unit is changed, for terminating the timing of current sync program
Scheduler task, synchronization program is changed to adapt to the table structure of new source database；And to source database and/or target data
Initiate the timer-triggered scheduler task of new synchronization program in storehouse.
Based on the methods and applications server of above-mentioned automatic modification source data database table structure, the present invention further correspondingly provides one
The system of kind modification source data database table structure, referring to Fig. 7, it includes several source databases 110, a target database 120
With the application server 130 of a modification source data database table structure.
Wherein, each source database 110, metadata information is synchronized in target database for daily timing.Mesh
Database 120 is marked, for metadata information to be pushed in application server.The application server 130, it is each for receiving
The metadata information that source database or target database push daily, and preserve the metadata information of at least nearest two days；With
And the same day metadata information of contrast storage and metadata information yesterday, judge whether table structure has variation；When table structure has change
When dynamic, according to table structural change information, the script of the table structure of generation modification target database, and by the script of the table structure
Transmission performs into target database.
, specifically need to be straight by each source database, target database 120 and application server 130 when changing table structure
Connect and interact, such as metadata information is synchronized in target database by each source database, then by target database
It is pushed in application server, the analysis and processing to metadata information is completed by application server, so as in data syn-chronization mistake
Cheng Zhong, by application server make the table structure of target database be automatically performed relative to source database table structure to when
Modification.
The workflow that the system of source data database table structure is changed in the present embodiment is as follows：
The metadata information that application server receives each source database or target database pushes daily, and preserve at least most
The metadata information of nearly two days；
The same day metadata information and metadata information yesterday of application server contrast storage, judge whether table structure has variation；
When table structure has variation, application server is according to table structural change information, the table structure of generation modification target database
Script, and the script of the table structure is sent and performed into target database.
Further, in the system of described modification source data database table structure, the application server contrast storage
With metadata information yesterday, the flow whether judge table structure has variation includes same day metadata information：
Application server start metadata contrast modification program, separate in the metadata information of storage yesterday metadata information and
Today metadata information；
Whether application server contrasts the column name of each table owner of metadata information one by one, judge each column name in yesterday
Exist in metadata information and today metadata information.
The application server contrasts the column name of each table owner of metadata information one by one, judges that each column name is
It is no yesterday metadata information and today metadata information present in include after flow：
If a certain column name exists in yesterday metadata information and is not present in today metadata information, application service
This column name is simultaneously set to the script of sky in correspondence position by the table structure of device generation modification target database, sends to target data
Deletion action is performed in storehouse；
If a certain column name is not present in yesterday metadata information and existed in today metadata information, application service
Device extracts the column name, and the table structure of generation modification target database simultaneously increases the script of the column name on correspondence position, sent out
Deliver in target database and perform increase operation.
In one embodiment, the metadata information pushed daily in application server reception target database, and protect
Before the step of depositing the metadata information of at least nearest two days, described workflow also includes：
Metadata information is synchronized in application server by the daily timing of each source database.
In another embodiment, the metadata information pushed daily in application server reception target database, and
Before the step of preserving the metadata information of at least nearest two days, described workflow also includes：
Metadata information is synchronized in target database by the daily timing of each source database；
Target database pushes to metadata information in application server.
Further, when table structure has variation, application server is according to table structural change information, generation modification number of targets
According to the script of the table structure in storehouse, and by the script of the table structure send into target database perform the step of after, also wrap
Include：
Application server terminates the timer-triggered scheduler task of current sync program, changes synchronization program to adapt to new source database
Table structure；
Application server initiates the timer-triggered scheduler task of new synchronization program to source database and/or target database.
In summary, method, server, system and the storage of automatic modification source data database table structure provided by the invention are situated between
In matter, the method for the automatic modification source data database table structure is pushed away daily by receiving each source database or target database
The metadata information sent, and preserve the metadata information of at least nearest two days；Contrast afterwards the same day metadata information of storage with
Yesterday, metadata information, judged whether table structure has variation；When judgement table structure has variation, according to table structural change information,
The script of the table structure of generation modification target database, and the script of the table structure is sent and performed into target database.
So that the table structure of target database be automatically performed relative to source database table structure to when changing so that in data
In synchronizing process, human cost can be both reduced, error rate can be reduced again.
Certainly, one of ordinary skill in the art will appreciate that realizing all or part of flow in above-described embodiment method,
It is that related hardware can be instructed by computer program（Such as processor, controller etc.）To complete, described program can be stored in
In one computer-readable storage medium, the program may include such as the flow of above-mentioned each method embodiment upon execution.Wherein
Described storage medium can be memory, magnetic disc, CD etc..
It should be appreciated that the application of the present invention is not limited to above-mentioned citing, for those of ordinary skills, can
To be improved or converted according to the above description, all these modifications and variations should all belong to the guarantor of appended claims of the present invention
Protect scope.
Claims (10)
- A kind of 1. method of automatic modification source data database table structure, it is characterised in that comprise the following steps：Receive each source database or metadata information that target database pushes daily, and preserve the member of at least nearest two days Data message；The same day metadata information and metadata information yesterday of storage are contrasted, judges whether table structure has variation；When table structure has variation, according to table structural change information, the script of the table structure of target database is changed in generation, and will The script of the table structure is sent to be performed into target database.
- 2. the method for automatic modification source data database table structure according to claim 1, it is characterised in that the contrast storage Same day metadata information with yesterday metadata information, judge whether table structure has the step of variation to include：Application server start metadata contrast modification program, separate in the metadata information of storage yesterday metadata information and Today metadata information；The column name of each table owner of metadata information is contrasted one by one, judges each column name whether in metadata information yesterday With exist in today metadata information.
- 3. the method for the automatic modification source data database table structure stated according to claim 2, it is characterised in that described when table structure has During variation, according to table structural change information, the script of the table structure of generation modification target database, and by the pin of the table structure The step of this transmission performs into target database includes：If a certain column name exists in yesterday metadata information and is not present in today metadata information, generation modification The table structure of target database and the script that this column name is set to sky in correspondence position, send into target database to perform and delete Division operation；If a certain column name is not present in yesterday metadata information and existed in today metadata information, the row are extracted Title, the table structure of generation modification target database simultaneously increase the script of the column name on correspondence position, sent to number of targets Operated according to increase is performed in storehouse.
- 4. the method for automatic modification source data database table structure according to claim 1, it is characterised in that in the reception mesh The metadata information that mark database pushes daily, and before the step of metadata information of preservation at least nearest two days, it is described Method also includes：Metadata information is synchronized in application server by the daily timing of each source database.
- 5. the method for automatic modification source data database table structure according to claim 1, it is characterised in that in the reception mesh The metadata information that mark database pushes daily, and before the step of metadata information of preservation at least nearest two days, it is described Method also includes：Metadata information is synchronized in target database by the daily timing of each source database；Target database pushes to metadata information in application server.
- 6. the method for automatic modification source data database table structure according to claim 1, it is characterised in that described to work as table structure When having variation, according to table structural change information, the script of the table structure of generation modification target database, and by the table structure After script sends into target database the step of performing, in addition to：Terminate the timer-triggered scheduler task of current sync program, change synchronization program to adapt to the table structure of new source database；The timer-triggered scheduler task of new synchronization program is initiated to source database and/or target database.
- 7. the method for the automatic modification source data database table structure according to claim 1 to 6 any one, it is characterised in that The source database is oracle database, MySQL database or PostgreSQL databases.
- A kind of 8. application server of automatic modification source data database table structure, it is characterised in that including：Processor, memory, lead to Believe bus;Being stored with the memory can be by the computer-readable program of the computing device；The communication bus realizes the connection communication between processor and memory；The automatic modification as described in claim 1-7 any one is realized described in the computing device during computer-readable program Step in the method for source data database table structure.
- A kind of 9. computer-readable recording medium, it is characterised in that the computer-readable recording medium storage have one or Multiple programs, one or more of programs can be by one or more computing devices, to realize such as claim 1~7 Step in the method for automatic modification source data database table structure described in any one.
- 10. a kind of system of automatic modification source data database table structure, including several source databases, it is characterised in that also include The application server of target database and modification source data database table structure as claimed in claim 8；Each source database, metadata information is synchronized in target database for daily timing；Target database, for metadata information to be pushed in application server；The application server, for the metadata information for receiving each source database or target database pushes daily, and Preserve the metadata information of at least nearest two days；And the same day metadata information of contrast storage and metadata information yesterday, sentence Whether disconnected table structure has variation；When table structure has variation, according to table structural change information, the table of generation modification target database The script of structure, and the script of the table structure is sent and performed into target database.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710755733.XA CN107515933A (en) | 2017-08-29 | 2017-08-29 | Change method, server, system and the storage medium of source data database table structure |
| PCT/CN2018/083879 WO2019041832A1 (en) | 2017-08-29 | 2018-04-20 | Method, server and system for modifying source database table structure, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710755733.XA CN107515933A (en) | 2017-08-29 | 2017-08-29 | Change method, server, system and the storage medium of source data database table structure |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107515933A true CN107515933A (en) | 2017-12-26 |
Family
ID=60724461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710755733.XA Pending CN107515933A (en) | 2017-08-29 | 2017-08-29 | Change method, server, system and the storage medium of source data database table structure |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107515933A (en) |
| WO (1) | WO2019041832A1 (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509612A (en) * | 2018-03-30 | 2018-09-07 | 北京润科通用技术有限公司 | A kind of data processing method and device |
| WO2019041832A1 (en) * | 2017-08-29 | 2019-03-07 | 平安科技（深圳）有限公司 | Method, server and system for modifying source database table structure, and storage medium |
| CN109800234A (en) * | 2019-01-25 | 2019-05-24 | 苏州科达科技股份有限公司 | Service platform database system, upgrade method, equipment and storage medium |
| WO2019127837A1 (en) * | 2017-12-28 | 2019-07-04 | 平安科技（深圳）有限公司 | Method, apparatus, and device for generating table creation script, and computer readable storage medium |
| CN110059134A (en) * | 2019-03-18 | 2019-07-26 | 深圳市买买提信息科技有限公司 | A kind of data are synchronized to method, relevant apparatus and the equipment of cloud platform |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110347693A (en) * | 2019-07-11 | 2019-10-18 | 招商局金融科技有限公司 | Method of data synchronization, device and storage medium |
| CN110866012A (en) * | 2019-11-11 | 2020-03-06 | 北京知道创宇信息技术股份有限公司 | Table structure management method, device, server and readable storage medium |
| CN111125101A (en) * | 2019-12-16 | 2020-05-08 | 杭州涂鸦信息技术有限公司 | Data center table structure consistency monitoring method and system |
| CN111367924A (en) * | 2020-02-26 | 2020-07-03 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN104317843A (en) * | 2014-10-11 | 2015-01-28 | 上海瀚之友信息技术服务有限公司 | Data synchronism ETL (Extract Transform Load) system |
| CN105393250A (en) * | 2013-07-09 | 2016-03-09 | 甲骨文国际公司 | Dynamic migration script management |
| CN106649771A (en) * | 2016-12-27 | 2017-05-10 | 广州杰赛科技股份有限公司 | Data model updating method and system for database |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2350860B1 (en) * | 2008-11-06 | 2015-04-08 | Deluxe Media Inc. | Placeholders in index table for updating a portable storage medium |
| CN104965735B (en) * | 2015-06-18 | 2018-10-19 | 北京京东尚科信息技术有限公司 | Device for generating upgrading SQL scripts |
| CN106326376A (en) * | 2016-08-15 | 2017-01-11 | 东软集团股份有限公司 | Method and device for copying information after table structure modification |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
-
2017
- 2017-08-29 CN CN201710755733.XA patent/CN107515933A/en active Pending
-
2018
- 2018-04-20 WO PCT/CN2018/083879 patent/WO2019041832A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN105393250A (en) * | 2013-07-09 | 2016-03-09 | 甲骨文国际公司 | Dynamic migration script management |
| CN104317843A (en) * | 2014-10-11 | 2015-01-28 | 上海瀚之友信息技术服务有限公司 | Data synchronism ETL (Extract Transform Load) system |
| CN106649771A (en) * | 2016-12-27 | 2017-05-10 | 广州杰赛科技股份有限公司 | Data model updating method and system for database |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019041832A1 (en) * | 2017-08-29 | 2019-03-07 | 平安科技（深圳）有限公司 | Method, server and system for modifying source database table structure, and storage medium |
| WO2019127837A1 (en) * | 2017-12-28 | 2019-07-04 | 平安科技（深圳）有限公司 | Method, apparatus, and device for generating table creation script, and computer readable storage medium |
| CN108509612A (en) * | 2018-03-30 | 2018-09-07 | 北京润科通用技术有限公司 | A kind of data processing method and device |
| CN109800234A (en) * | 2019-01-25 | 2019-05-24 | 苏州科达科技股份有限公司 | Service platform database system, upgrade method, equipment and storage medium |
| CN110059134A (en) * | 2019-03-18 | 2019-07-26 | 深圳市买买提信息科技有限公司 | A kind of data are synchronized to method, relevant apparatus and the equipment of cloud platform |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110347693A (en) * | 2019-07-11 | 2019-10-18 | 招商局金融科技有限公司 | Method of data synchronization, device and storage medium |
| CN110866012A (en) * | 2019-11-11 | 2020-03-06 | 北京知道创宇信息技术股份有限公司 | Table structure management method, device, server and readable storage medium |
| CN111125101A (en) * | 2019-12-16 | 2020-05-08 | 杭州涂鸦信息技术有限公司 | Data center table structure consistency monitoring method and system |
| CN111125101B (en) * | 2019-12-16 | 2023-10-13 | 杭州涂鸦信息技术有限公司 | Data center table structure consistency monitoring method and system |
| CN111367924A (en) * | 2020-02-26 | 2020-07-03 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN111367924B (en) * | 2020-02-26 | 2024-01-02 | 杭州数梦工场科技有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019041832A1 (en) | 2019-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107515933A (en) | Change method, server, system and the storage medium of source data database table structure | |
| CN108536761B (en) | Report data query method and server | |
| US11182404B2 (en) | Data replication technique in database management system | |
| US11481408B2 (en) | Event driven extract, transform, load (ETL) processing | |
| CN105493028B (en) | Data consistency and rollback for cloud analysis | |
| CN105787091B (en) | Support the content delivering system and method for multiple terminals | |
| CN108897874A (en) | Method and apparatus for handling data | |
| CN109816481A (en) | Bill processing method, device and computer readable storage medium | |
| CN109783258A (en) | A kind of message treatment method, device and server | |
| CN110457333A (en) | Data real time updating method, device and computer readable storage medium | |
| KR20130140508A (en) | Apparatus for collecting log information | |
| CN109614270A (en) | Data read-write method, device, equipment and storage medium based on Hbase | |
| CN108345658A (en) | Algorithm calculates decomposing process, server and the storage medium of track | |
| CN103123651A (en) | Method of rapidly searching multiple same-kind paper, device and mobile equipment | |
| CN110008284A (en) | Method for synchronizing data of database and equipment based on data page preloading and rollback | |
| WO2022156087A1 (en) | Data blood relationship establishing method and apparatus, computer device, and storage medium | |
| CN111488386A (en) | Data query method and device | |
| CN110457345A (en) | Database read method, device and computer readable storage medium | |
| CN109285043A (en) | A kind of configuration method of product modular, terminal device and storage medium | |
| US10402391B2 (en) | Processing method, device and system for data of distributed storage system | |
| CN105656946A (en) | Method for processing logging account, business service terminal and account service terminal | |
| CN101499086A (en) | Heterogeneous module data sharing system and method | |
| CN104636190A (en) | Software system and method for transmitting information between plug-ins | |
| CN108509513A (en) | Platform customer data reflow method, electronic device, equipment and storage medium | |
| CN107609008A (en) | A kind of data importing device and method from relevant database to Kafka based on Apache Sqoop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171226 |
|
| RJ01 | Rejection of invention patent application after publication |