CN107122497A - Data processing method, device, electronic equipment and computer-readable recording medium - Google Patents
Data processing method, device, electronic equipment and computer-readable recording medium Download PDFInfo
- Publication number
- CN107122497A CN107122497A CN201710381991.6A CN201710381991A CN107122497A CN 107122497 A CN107122497 A CN 107122497A CN 201710381991 A CN201710381991 A CN 201710381991A CN 107122497 A CN107122497 A CN 107122497A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- source database
- variation
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the invention disclose a kind of data processing method, device, electronic equipment and computer-readable recording medium, it is related to technical field of data processing, can shorten source database server and the data query of targeting database server and reproduction process needs the time of consuming.The data processing method includes：Obtain the configuration file that preset distributed post subscribes to messaging system server；Obtain the variation message of source database；According to the configuration file, the variation message to the source database is handled, and the variation message of the source database after processing is sent into target database.The present invention requires higher scene suitable for the data processing speed of source database server and targeting database server.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of data processing method, device, electronic equipment and meter
Calculation machine readable storage medium storing program for executing.
Background technology
Existing Relational DBMS Mysql, in WWW (World Wide Web, abbreviation WEB) application
Aspect is best relational database management system (Relational Database Management System, abbreviation
RDBMS application software).At present, Relational DBMS Mysql replication theme mainly includes two kinds：The first
Master-slave synchronisation scheme；Program efficient stable, in two main database servers and the feelings from database server network UNICOM
Under condition, data in Relational DBMS Mysql databases can by main database server with from database
The mode of server sync completes data duplication；Second is SQL (Structured Query
Language, abbreviation SQL) data processing scheme；The program completes data query by SQL statement and data are inserted, so that real
Data query and the duplication of existing source database server and targeting database server.
However, in database data processing procedure, inventor has found at least there is following technical problem in the prior art：
The data query of source database server and targeting database server is realized in prior art by SQL statement
And reproduction process needs the time expended longer, it is impossible to meet the data handling requirements of user.
The content of the invention
In view of this, the embodiment of the present invention provides a kind of data processing method, device, electronic equipment and computer-readable deposited
Storage media, can accelerate data processing speed needs the time expended longer to solve the process of existing data query and duplication
The problem of.
In a first aspect, the embodiment of the present invention provides a kind of data processing method, including：
Obtain the configuration file that preset distributed post subscribes to messaging system server；
Obtain the variation message of source database；
According to the configuration file, the variation message to the source database is handled, and by the source data after processing
The variation message in storehouse is sent to target database.
With reference in a first aspect, in the first embodiment of first aspect, the configuration file includes：Pending change
Number of threads in the structure of dynamic message, variable format and thread pool；Wherein, it is described according to the configuration file, to the source
The variation message of database is handled, and the variation message of the source database after processing is sent to the step of target database
Suddenly, including：
Obtain the variation type of message of the source database；
According to the variation type of message of the source database, the processing mode of data is determined；
According to the structure of the pending variation message, variable format, number of threads in thread pool and it is described really
The processing mode of fixed data, data conversion is carried out by the variation message of the source database, and by the source database after conversion
Variation message be sent to target database.
With reference to the first embodiment of first aspect, in second of embodiment of first aspect, the variation disappears
Breath type includes：Insertion, renewal are deleted；The processing mode of the data includes：The task queue processing mode of caching system
Processing mode is exported with high concurrent data input；The variation type of message according to the source database, determines the place of data
The step of reason mode is：
The variation type of message of the source database is inserted or updated, and the processing mode for determining data is the high concurrent
Data input exports processing mode；
The variation type of message of the source database determines the processing mode of data appointing for the caching system to delete
Business queue processing mode.
It is described to convert in the third embodiment of first aspect with reference to second of embodiment of first aspect
The step of variation message of source database afterwards is sent to target database, including：
By the downloading data order of Relational DBMS, the variation of the source database after the conversion is disappeared
Breath is sent to target database.
Second aspect, the embodiment of the present invention provides a kind of data processing equipment, including：
Dispensing unit, the distributed post preset for obtaining subscribes to the configuration file of messaging system server；
Message retrieval unit, the variation message for obtaining source database；
Message processing unit, for according to the configuration file, the variation message to the source database to be handled, and
The variation message of source database after processing is sent to target database；
Wherein, the configuration file includes：Thread in the structure of pending variation message, variable format and thread pool
Quantity.
With reference to second aspect, in the first embodiment of second aspect, the configuration file includes：Pending change
Number of threads in the structure of dynamic message, variable format and thread pool；
The message processing unit, including：
Type of message obtains subelement, the variation type of message for obtaining the source database；
Data processing method determination subelement, for the variation type of message according to the source database, determines data
Processing mode；
Data processing subelement, for the structure according to the pending variation message, variable format, in thread pool
The processing mode of number of threads and the data of the determination, data conversion is carried out by the variation message of the source database, and
The variation message of source database after conversion is sent to target database.
With reference to the first embodiment of second aspect, in second of embodiment of second aspect, the variation disappears
Breath type includes：Insertion, renewal are deleted；The processing mode of the data includes：The task queue processing mode of caching system
Processing mode is exported with high concurrent data input；
The data processing method determination subelement specifically for：
The variation type of message of the source database is inserted or updated, and the processing mode for determining data is the high concurrent
Data input exports processing mode；
The variation type of message of the source database determines the processing mode of data appointing for the caching system to delete
Business queue processing mode.
With reference to second of embodiment of second aspect, in the third embodiment of second aspect, at the data
Subelement is managed when the variation message of the source database after by conversion is sent to target database, passes through relational data library management
The downloading data order of system, target database is sent to by the variation message of the source database after the conversion.
The third aspect, the embodiment of the present invention provides a kind of electronic equipment, including memory, processor and is stored in described deposit
On reservoir and the computer program that can run on the processor；Wherein, realized such as during the computing device described program
Data processing method described in upper any one.
Fourth aspect, the embodiment of the present invention provides a kind of computer-readable recording medium, for storing computer program, its
In, the computer program can be executed by processor the data processing method described in as above any one.
A kind of data processing method provided in an embodiment of the present invention, device, electronic equipment and computer-readable recording medium,
The distributed post preset by obtaining subscribes to the configuration file of messaging system server so that the distributed post, which is subscribed to, to disappear
Data conversion can be carried out according to configuration file to the variation message of source database by ceasing system server, and according to source database
Change type of message and carry out data processing, so that the variation message of the source database after processing is sent into target database；Adopt
Targeting database server acquisition source database clothes can not only be shortened by subscribing to messaging system server with the distributed post
The time of the variation message of the source database of business device, can also be flexible according to configuration file, the variation to the source database
Message carries out data conversion.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is the flow chart of the data processing method of embodiments of the invention；
Fig. 2 is the flow chart of the data processing method embodiment two of embodiments of the invention；
Fig. 3 is the data handling system schematic diagram of data processing method embodiment three of the present invention；
Fig. 4 is the structural representation of data processing equipment embodiment one of the present invention；
Fig. 5 is the structural representation of data processing equipment embodiment two of the present invention；
Fig. 6 is the structural representation of electronic equipment one embodiment of the present invention.
Embodiment
The embodiment of the present invention is described in detail below in conjunction with the accompanying drawings.
It will be appreciated that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.Base
Embodiment in the present invention, those of ordinary skill in the art obtained under the premise of creative work is not made it is all its
Its embodiment, belongs to the scope of protection of the invention.
Fig. 1 is the schematic diagram of embodiments of the invention data processing method, and message system is subscribed to applied to distributed post
Server, as shown in figure 1, the method for the present embodiment can include：
Step 101, the configuration file for obtaining preset distributed post subscription messaging system server；
In the present embodiment, the configuration file that the distributed post subscribes to messaging system server can be according to actual need
Ask and be configured, the variation message of the source database for getting carries out the self-defined conversions such as structure, form.For example：To institute
Variation message progress field filter, newer field generation, the multilist for stating source database merge, carry out point table handling etc. by field.
Step 102, the variation message for obtaining source database；
In the present embodiment, changed if capturing its data in source database server, pass through source database
Variation message be sent to distributed post subscribe to messaging system server.The distributed post subscribes to messaging system server
The variation message of source database is got, then variation message that can be further to the source database is handled.
Step 103, according to the configuration file, the variation message to the source database is handled, and will processing after
The variation message of source database be sent to target database.
In the present embodiment, the distributed post subscribes to messaging system server according to the configuration file pre-set, right
The variation message of the source database got is handled；For example：Structure, form in configuration file etc. are self-defined
Conversion, the variation message to the source database enters row information identification, information format conversion and information processing, then will processing
Information afterwards is Resealed, and is sent to target database.
The present embodiment, the distributed post preset by obtaining subscribes to the configuration file of messaging system server so that institute
Data turn can be carried out according to configuration file to the variation message of source database by stating distributed post subscription messaging system server
Change, and data processing is carried out according to the variation type of message of source database, so that by the variation message of the source database after processing
It is sent to target database；Target database clothes can not only be shortened by subscribing to messaging system server using the distributed post
Business device obtains the time of the variation message of the source database of source database server, can also be flexible according to configuration file, right
The variation message of the source database carries out data conversion.
Fig. 2 is the flow chart of data processing method embodiment two of the present invention, as shown in Fig. 2 the method for the present embodiment can be with
Including：
Step 201, the configuration file for obtaining preset distributed post subscription messaging system server；
In the present embodiment, the configuration file includes：In the structure of pending variation message, variable format and thread pool
Number of threads；The process of preset configuration file is similar with the step 101 of above method embodiment, and here is omitted.
Step 202, the variation message for obtaining source database；
In the present embodiment, the process and the step 102 of above method embodiment of the variation message for obtaining source database
Similar, here is omitted.
Step 203, the variation type of message for obtaining the source database；
In the present embodiment, the variation type of message of the source database of the acquisition is that the distributed post subscribes to message system
Parsing of the system server after the variation message of the source database is got to the message, by obtaining the source database
Type of message is changed, after the distributed post subscription messaging system server can just be determined according to the configuration file of its own
Continue processing mode and the data conversion of the message.
Step 204, the variation type of message according to the source database, determine the processing mode of data；
In the present embodiment, the variation type of message includes：Insertion, renewal are deleted；The processing mode bag of the data
Include：Task queue processing mode and high concurrent data input the output processing mode of caching system；The variation of the source database
Type of message is inserted or updated, determines that the processing mode of data exports processing mode for the high concurrent data input；It is described
The variation type of message of source database is deletes, and the processing mode for determining data is the task queue processing side of the caching system
Formula.
Number of threads in step 205, the structure according to the pending variation message, variable format, thread pool with
And the processing mode of the data of the determination, the variation message of the source database is subjected to data conversion, and by after conversion
The variation message of source database is sent to target database；
In the present embodiment, the type of the variation message of the source database after the conversion includes：Replace or delete；Specifically
It will be exactly, the type of the variation message of source database includes in source database server：Insertion, renewal are deleted；Described point
The variation message of the source database is carried out data by cloth distribution subscription messaging system server according to the setting of configuration file
Conversion, is that insertion or renewal are converted into replacement type by type of message in source database server, deletes type and keep constant；
The variation message of source database after the conversion is sent to download of the target database by Relational DBMS
Data command is realized.
The present embodiment, messaging system server is subscribed to by the source database server acquisition source number by distributed post
The variation message of the source database of acquisition is subjected to data identification according to the variation message in storehouse, and according to its configuration file, it is determined that disappearing
Type is ceased, and then determines the processing mode of data.The distribution used due to the processing mode of technical solution of the present invention meeting data
Distribution subscription messaging system server, and different processing modes is used to different type of messages, shorten the processing of data
Time, so as to substantially increase the treatment effeciency of message；Technical solution of the present invention can also be by the setting of configuration file, such as
Number of threads in the structure of pending variation message, variable format and thread pool is set by configuration file, can be caused
Distributed post, which subscribes to messaging system server, to disappear according to the variation for the source database that source database server is its transmission
Breath configuration number of threads；Can also be according to the structure of pending variation message, change of the variable format to the source database of acquisition
Dynamic message carries out data processing, completes the conversion of the data structure of the variation message of source database, format conversion.
Below using a specific embodiment, the technical scheme to embodiment of the method shown in any of Fig. 1~Fig. 2 is entered
Row is described in detail.
Be below the process description of data processing method embodiment three of the present invention with reference to shown in Fig. 3, the present embodiment be applicable
Scene can be source database server for Open Framework Debezium and the source database server obtains user's mandate,
Distributed post subscribes to messaging system server Kafka, wherein, the source data server can be N number of, and target database can
Think M；With reference to shown in Fig. 3, specific data processing method can be：
Step 301, the configuration file for obtaining preset distributed post subscription messaging system server；The configuration file
Including：Number of threads in the structure of pending variation message, variable format and thread pool.
In the present embodiment, the structure of the pending variation message is used for the variation message for parsing identification source database；
The variation message that the variable format is used for the source database that will be parsed carries out message turn according to the variable format requirement
Change, for example：By the insertion type and updating type of the variation message of source database server, replacement type is converted into；Source data
It is constant that the variation message of storehouse server deletes type；Number of threads in the thread pool, then can with according to the variation message
Source database server be mutually matched, corresponding number of threads is determined to handle the variation message.
Step 302, the distributed post subscribe to messaging system server and obtain source using java editions KafkaConsumer
The variation message of database；The variation type of message includes：Insertion, renewal are deleted.
Step 303, the distributed post subscribe to messaging system server according to the configuration file, to the source data
The variation message in storehouse is handled, and the variation message of the source database after processing is sent into target database；
In the present embodiment, the variation type of message of the source database of the acquisition can be subscribed to by the distributed post
Messaging system server is analyzed and acquired by after the variation message of the source database is got to the message；By obtaining
The variation type of message of source database is stated, the distributed post subscribes to messaging system server just can matching somebody with somebody according to its own
Put processing mode and data conversion that file determines the follow-up message.
For example：The type of message that the variation message of the source database is parsed is insertion or updates, then basis point
The configuration file of cloth distribution subscription messaging system server, the distributed post is subscribed to messaging system server and will parsed
The message transformation come determines that the data processing method of the variation message exports processing side for high concurrent data input to replace type
Formula Java New IO, and be the number of threads in its corresponding thread pool of matching according to the server of source database, become to described
Dynamic message is handled；If the type of message that the variation message of the source database is parsed is deletes, according to distribution
The configuration file of formula distribution subscription messaging system server, the distributed post is subscribed to messaging system server and will parsed
Message be defined as delete type, determine the variation message data processing method be the caching system redis task team
Column processing mode, and be the number of threads in its corresponding thread pool of matching according to the server of source database, changed to described
Message is handled.The variation message of source database after the conversion is sent to target database and passes through relational data depositary management
Reason system mysql downloading data order load data are realized.
The data described above for replacing type can be handled flexibly data according to configuration file, such as field is sieved
Choosing, newer field generation (by calculating), multilist merge, point table handling etc. are carried out by field, and realization is tied to source database
The self-defined conversion such as structure, form.
The data described above for deleting type are inserted into redis task queue in real time, then restart one group of thread
Asynchronous consumption redis task queues are gone in pond, and the data in target database are deleted.
Fig. 4 is the structural representation of data processing equipment embodiment one of the present invention, as shown in figure 4, the device of the present embodiment
It can include：Dispensing unit 401, message retrieval unit 402 and message processing unit 403；Wherein, the dispensing unit 401, is used
In the configuration file for obtaining preset distributed post subscription messaging system server；The message retrieval unit 402, for obtaining
Take the variation message of source database；The message processing unit 403, for according to the configuration file, to the source database
Variation message handled, and the variation message of the source database after processing is sent to target database.
The device of the present embodiment, can be used for the technical scheme for performing embodiment of the method shown in Fig. 1, its realization principle and skill
Art effect is similar, and here is omitted.
Fig. 5 is the structural representation of data processing equipment embodiment two of the present invention, as shown in figure 5, the device of the present embodiment
On the basis of Fig. 4 shown device structures, further, the message processing unit 403 can include：
Type of message obtains subelement 501, data processing method determination subelement 502 and data processing subelement 503,
Wherein, the type of message obtains subelement 501, the variation type of message for obtaining the source database；The variation disappears
Breath type includes：Insertion, renewal are deleted；The data processing method determination subelement 502, for according to the source database
Variation type of message, determine the processing mode of data；The data processing subelement 503, for according to described pending
The processing mode of the number of threads in the structure of message, variable format, thread pool and the data of the determination is changed, will be described
The variation message of source database carries out data conversion, and the variation message of the source database after conversion is sent into target data
Storehouse.
It should be noted that the variation type of message includes：Insertion, renewal are deleted；The processing mode of the data
Including：Task queue processing mode and high concurrent data input the output processing mode of caching system；
The data processing method determination subelement specifically for：
The variation type of message of the source database is inserted or updated, and the processing mode for determining data is the high concurrent
Data input exports processing mode；
The variation type of message of the source database determines the processing mode of data appointing for the caching system to delete
Business queue processing mode.
It should also be noted that, the variation message of source database of the data processing subelement after by conversion is sent to
During target database, by the downloading data order of Relational DBMS, by the source database after the conversion
Change message and be sent to target database.
The device of the present embodiment, can be used for the technical scheme for performing embodiment of the method shown in Fig. 1 to Fig. 3, it realizes former
Reason is similar with technique effect, and here is omitted.
It should be noted that herein, such as first and second or the like relational terms are used merely to a reality
Body or operation make a distinction with another entity or operation, and not necessarily require or imply these entities or deposited between operating
In any this actual relation or order.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Nonexcludability is included, so that process, method, article or equipment including a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include being this process, method, article or equipment
Intrinsic key element.In the absence of more restrictions, the key element limited by sentence " including one ... ", it is not excluded that
Also there is other identical element in the process including the key element, method, article or equipment.
Each embodiment in this specification is described by the way of related, identical similar portion between each embodiment
Divide mutually referring to what each embodiment was stressed is the difference with other embodiment.
For device embodiment, because it is substantially similar to embodiment of the method, so the comparison of description is simple
Single, the relevent part can refer to the partial explaination of embodiments of method.
Represent in flow charts or logic and/or step described otherwise above herein, for example, being considered use
In the order list for the executable instruction for realizing logic function, it may be embodied in any computer-readable medium, for
Instruction execution system, device or equipment (such as computer based system including the system of processor or other can be held from instruction
The system of row system, device or equipment instruction fetch and execute instruction) use, or combine these instruction execution systems, device or set
It is standby and use.For the purpose of this specification, " computer-readable medium " can any can be included, store, communicate, propagate or pass
Defeated program is for instruction execution system, device or equipment or the dress for combining these instruction execution systems, device or equipment and using
Put.The more specifically example (non-exhaustive list) of computer-readable medium includes following：Electricity with one or more wirings
Connecting portion (electronic installation), portable computer diskette box (magnetic device), random access memory (RAM), read-only storage
(ROM), erasable edit read-only storage (EPROM or flash memory), fiber device, and portable optic disk is read-only deposits
Reservoir (CDROM).In addition, can even is that can be in the paper of printing described program thereon or other are suitable for computer-readable medium
Medium, because can then enter edlin, interpretation or if necessary with it for example by carrying out optical scanner to paper or other media
His suitable method is handled electronically to obtain described program, is then stored in computer storage.
It should be appreciated that each several part of the present invention can be realized with hardware, software, firmware or combinations thereof.
In the above-described embodiment, multiple steps or method can in memory and by suitable instruction be performed with storage
The software or firmware that system is performed are realized.If for example, being realized with hardware, with another embodiment, can use
Any one of following technology well known in the art or their combination are realized：With for realizing logic work(to data-signal
The discrete logic of the logic gates of energy, the application specific integrated circuit with suitable combinational logic gate circuit, programmable gate
Array (PGA), field programmable gate array (FPGA) etc..
The embodiment of the present invention also provides a kind of electronic equipment, including memory, processor and is stored on the memory
And the computer program that can be run on the processor；Wherein, realized during the computing device described program as above any
Data processing method described in.
Fig. 6 is the structural representation of electronic equipment one embodiment of the present invention, it is possible to achieve implemented shown in Fig. 1-3 of the present invention
The flow of example, as shown in fig. 6, the electronic equipment can include：Memory 601, processor 602 and it is stored in the memory
On 601 and the computer program that can be run on the processor 602；Wherein, realized during the computing device described program
Data processing method as described above.
The processor 602 is executable by operation to the specific implementation procedure and the processor 602 of above-mentioned steps
The step of program code is further to perform, may refer to the description of Fig. 1-3 illustrated embodiments of the present invention, will not be repeated here.
The electronic equipment exists in a variety of forms, includes but is not limited to：
(1) server：The equipment for providing the service of calculating, the composition of server is total including processor, hard disk, internal memory, system
Line etc., server is similar with general computer architecture, but is due to need to provide highly reliable service, therefore in processing energy
Require higher in terms of power, stability, reliability, security, scalability, manageability.
(2) other electronic equipments with data interaction function.
Those skilled in the art are appreciated that to realize all or part of step that above-described embodiment method is carried
Rapid to can be by program to instruct the hardware of correlation to complete, described program can be stored in a kind of computer-readable storage medium
In matter, the program upon execution, including one or a combination set of the step of embodiment of the method.
For convenience of description, description apparatus above is to be divided into various units/modules with function to describe respectively.Certainly, exist
Implement the function of each unit/module can be realized in same or multiple softwares and/or hardware during the present invention.
As seen through the above description of the embodiments, those skilled in the art can be understood that the present invention can
Realized by the mode of software plus required general hardware platform.Understood based on such, technical scheme essence
On the part that is contributed in other words to prior art can be embodied in the form of software product, the computer software product
It can be stored in storage medium, such as ROM/RAM, magnetic disc, CD, including some instructions are to cause a computer equipment
(can be personal computer, server, or network equipment etc.) performs some of each of the invention embodiment or embodiment
Method described in part.
The present invention also provides a kind of computer-readable recording medium, for storing computer program, wherein, the computer
Program can be executed by processor data processing method as described above.
One of ordinary skill in the art will appreciate that realize all or part of flow in above-described embodiment method, being can be with
The hardware of correlation is instructed to complete by computer program, described program can be stored in a computer read/write memory medium
In, the program is upon execution, it may include such as the flow of the embodiment of above-mentioned each method.Wherein, described storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
The foregoing is only a specific embodiment of the invention, but protection scope of the present invention is not limited thereto, any
Those familiar with the art the invention discloses technical scope in, the change or replacement that can be readily occurred in, all should
It is included within the scope of the present invention.Therefore, protection scope of the present invention should be defined by scope of the claims.
Claims (10)
1. a kind of data processing method, it is characterised in that including：
Obtain the configuration file that preset distributed post subscribes to messaging system server；
Obtain the variation message of source database；
According to the configuration file, the variation message to the source database is handled, and by the source database after processing
Change message and be sent to target database.
2. data processing method according to claim 1, it is characterised in that the configuration file includes：Pending change
Number of threads in the structure of dynamic message, variable format and thread pool；
It is described that according to the configuration file, the variation message to the source database is handled, and by the source data after processing
The step of variation message in storehouse is sent to target database, including：
Obtain the variation type of message of the source database；
According to the variation type of message of the source database, the processing mode of data is determined；
According to the structure of the pending variation message, variable format, number of threads and the determination in thread pool
The processing mode of data, data conversion is carried out by the variation message of the source database, and by the change of the source database after conversion
Dynamic message is sent to target database.
3. data processing method according to claim 2, it is characterised in that the variation type of message includes：Insert, more
New or deletion；The processing mode of the data includes：The task queue processing mode and high concurrent data input of caching system are defeated
Go out processing mode；It is according to the variation type of message of the source database, the step of the processing mode for determining data：
The variation type of message of the source database is inserted or updated, and the processing mode for determining data is the high concurrent data
Input and output processing mode；
The variation type of message of the source database is deletes, and the processing mode for determining data is the task team of the caching system
Column processing mode.
4. data processing method according to claim 3, it is characterised in that the variation of the source database by after conversion
The step of message is sent to target database, including：
By the downloading data order of Relational DBMS, the variation message of the source database after the conversion is sent out
Give target database.
5. a kind of data processing equipment, it is characterised in that including：
Dispensing unit, the distributed post preset for obtaining subscribes to the configuration file of messaging system server；
Message retrieval unit, the variation message for obtaining source database；
Message processing unit, for according to the configuration file, the variation message to the source database to be handled, and will place
The variation message of source database after reason is sent to target database.
6. data processing equipment according to claim 5, it is characterised in that the configuration file includes：Pending change
Number of threads in the structure of dynamic message, variable format and thread pool；
The message processing unit, including：
Type of message obtains subelement, the variation type of message for obtaining the source database；
Data processing method determination subelement, for the variation type of message according to the source database, determines the processing of data
Mode；
Data processing subelement, for the structure according to the pending variation message, variable format, the thread in thread pool
The processing mode of quantity and the data of the determination, carries out data conversion, and will turn by the variation message of the source database
The variation message of source database after change is sent to target database.
7. data processing equipment according to claim 6, it is characterised in that the variation type of message includes：Insert, more
New or deletion；The processing mode of the data includes：The task queue processing mode and high concurrent data input of caching system are defeated
Go out processing mode；
The data processing method determination subelement specifically for：
The variation type of message of the source database is inserted or updated, and the processing mode for determining data is the high concurrent data
Input and output processing mode；
The variation type of message of the source database is deletes, and the processing mode for determining data is the task team of the caching system
Column processing mode.
8. data processing equipment according to claim 7, it is characterised in that the data processing subelement will be after it will convert
The variation message of source database when being sent to target database, the downloading data for passing through Relational DBMS is ordered
Order, target database is sent to by the variation message of the source database after the conversion.
9. a kind of electronic equipment, including memory, processor and it is stored on the memory and can transports on the processor
Capable computer program；Characterized in that, being realized during the computing device described program as described in claim any one of 1-4
Data processing method.
10. a kind of computer-readable recording medium, for storing computer program, it is characterised in that the computer program can
It is executed by processor the data processing method as described in claim any one of 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710381991.6A CN107122497A (en) | 2017-05-25 | 2017-05-25 | Data processing method, device, electronic equipment and computer-readable recording medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710381991.6A CN107122497A (en) | 2017-05-25 | 2017-05-25 | Data processing method, device, electronic equipment and computer-readable recording medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107122497A true CN107122497A (en) | 2017-09-01 |
Family
ID=59728927
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710381991.6A Pending CN107122497A (en) | 2017-05-25 | 2017-05-25 | Data processing method, device, electronic equipment and computer-readable recording medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107122497A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107766554A (en) * | 2017-11-02 | 2018-03-06 | 山东浪潮通软信息科技有限公司 | A kind of data processing method, apparatus and system |
| CN108009028A (en) * | 2017-11-29 | 2018-05-08 | 中国平安人寿保险股份有限公司 | Message treatment method, device, equipment and computer-readable recording medium |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109063066A (en) * | 2018-07-20 | 2018-12-21 | 中国邮政储蓄银行股份有限公司 | Data query method and device, data management system |
| CN109284334A (en) * | 2018-09-05 | 2019-01-29 | 拉扎斯网络科技（上海）有限公司 | Real-time data base synchronous method, device, electronic equipment and storage medium |
| CN109359139A (en) * | 2018-10-24 | 2019-02-19 | 拉扎斯网络科技（上海）有限公司 | Method of data synchronization, system, electronic equipment and computer readable storage medium |
| CN110874257A (en) * | 2018-08-31 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Container configuration method and device on server |
| CN111103635A (en) * | 2019-12-04 | 2020-05-05 | 上海眼控科技股份有限公司 | Meteorological data processing method, system, electronic equipment and storage medium |
| CN111857879A (en) * | 2020-07-21 | 2020-10-30 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN112783828A (en) * | 2019-11-11 | 2021-05-11 | 千寻位置网络有限公司 | IGS file processing method and device |
| CN113110926A (en) * | 2021-04-19 | 2021-07-13 | 上海华兴数字科技有限公司 | Dynamic adjustment method, device, medium and electronic equipment for data consumption thread |
| CN113177032A (en) * | 2021-06-29 | 2021-07-27 | 南京云联数科科技有限公司 | Database-based data sharing method and system |
| WO2021218143A1 (en) * | 2020-04-29 | 2021-11-04 | 平安科技（深圳）有限公司 | Data synchronization method and apparatus, and server and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101308513A (en) * | 2008-06-27 | 2008-11-19 | 福建星网锐捷网络有限公司 | Distributed system cache data synchronous configuration method and apparatus |
| CN102724304A (en) * | 2012-06-06 | 2012-10-10 | 哈尔滨工程大学 | Information warehouse federation in subscription/release system and data synchronization method |
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN106339498A (en) * | 2016-09-05 | 2017-01-18 | 北京微播易科技股份有限公司 | Method, device and system for synchronizing data |
-
2017
- 2017-05-25 CN CN201710381991.6A patent/CN107122497A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101308513A (en) * | 2008-06-27 | 2008-11-19 | 福建星网锐捷网络有限公司 | Distributed system cache data synchronous configuration method and apparatus |
| CN102724304A (en) * | 2012-06-06 | 2012-10-10 | 哈尔滨工程大学 | Information warehouse federation in subscription/release system and data synchronization method |
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN106339498A (en) * | 2016-09-05 | 2017-01-18 | 北京微播易科技股份有限公司 | Method, device and system for synchronizing data |
Non-Patent Citations (2)
| Title |
|---|
| 袁震: "基于消息中间件的数据同步更新方法", 《兵工自动化》 * |
| 陈广伟 等: "基于发布／订阅的通用数据交互接口模型", 《华中科技大学学报（自然科学版）》 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107766554A (en) * | 2017-11-02 | 2018-03-06 | 山东浪潮通软信息科技有限公司 | A kind of data processing method, apparatus and system |
| CN108009028A (en) * | 2017-11-29 | 2018-05-08 | 中国平安人寿保险股份有限公司 | Message treatment method, device, equipment and computer-readable recording medium |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109063066B (en) * | 2018-07-20 | 2022-04-01 | 中国邮政储蓄银行股份有限公司 | Data query method and device and data management system |
| CN109063066A (en) * | 2018-07-20 | 2018-12-21 | 中国邮政储蓄银行股份有限公司 | Data query method and device, data management system |
| CN110874257A (en) * | 2018-08-31 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Container configuration method and device on server |
| CN110874257B (en) * | 2018-08-31 | 2024-04-02 | 阿里巴巴集团控股有限公司 | Method and device for configuring container on server |
| CN109284334B (en) * | 2018-09-05 | 2022-01-28 | 拉扎斯网络科技（上海）有限公司 | Real-time database synchronization method and device, electronic equipment and storage medium |
| CN109284334A (en) * | 2018-09-05 | 2019-01-29 | 拉扎斯网络科技（上海）有限公司 | Real-time data base synchronous method, device, electronic equipment and storage medium |
| CN109359139A (en) * | 2018-10-24 | 2019-02-19 | 拉扎斯网络科技（上海）有限公司 | Method of data synchronization, system, electronic equipment and computer readable storage medium |
| CN112783828A (en) * | 2019-11-11 | 2021-05-11 | 千寻位置网络有限公司 | IGS file processing method and device |
| CN112783828B (en) * | 2019-11-11 | 2022-08-19 | 千寻位置网络有限公司 | IGS file processing method and device |
| CN111103635A (en) * | 2019-12-04 | 2020-05-05 | 上海眼控科技股份有限公司 | Meteorological data processing method, system, electronic equipment and storage medium |
| WO2021218143A1 (en) * | 2020-04-29 | 2021-11-04 | 平安科技（深圳）有限公司 | Data synchronization method and apparatus, and server and storage medium |
| CN111857879B (en) * | 2020-07-21 | 2023-11-24 | 抖音视界有限公司 | Data processing method, device, electronic equipment and computer readable medium |
| CN111857879A (en) * | 2020-07-21 | 2020-10-30 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN113110926A (en) * | 2021-04-19 | 2021-07-13 | 上海华兴数字科技有限公司 | Dynamic adjustment method, device, medium and electronic equipment for data consumption thread |
| CN113177032A (en) * | 2021-06-29 | 2021-07-27 | 南京云联数科科技有限公司 | Database-based data sharing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107122497A (en) | Data processing method, device, electronic equipment and computer-readable recording medium | |
| US6029174A (en) | Apparatus and system for an adaptive data management architecture | |
| Verma et al. | Breaking the MapReduce stage barrier | |
| Sumbaly et al. | The big data ecosystem at linkedin | |
| US6363388B1 (en) | Apparatus and system for an adaptive data management architecture | |
| US9508048B2 (en) | System and method for integrated real time reporting and analytics across networked applications | |
| Barmpis et al. | Hawk: Towards a scalable model indexing architecture | |
| US8370326B2 (en) | System and method for parallel computation of frequency histograms on joined tables | |
| CN104050248B (en) | A kind of document storage system and storage method | |
| Rith et al. | Speaking in tongues: SQL access to NoSQL systems | |
| CN106104526A (en) | The transparent discovery of semi-structured data pattern | |
| Yang et al. | F1 Lightning: HTAP as a Service | |
| CN104462185A (en) | Digital library cloud storage system based on mixed structure | |
| CN109033109A (en) | Data processing method and system | |
| CN109726192A (en) | The system and method for master data model version and field separate management function is realized based on big data environment | |
| Adams et al. | Oc π: object-centric process insights | |
| Holzmann et al. | ABCDEF: The 6 key features behind scalable, multi-tenant web archive processing with ARCH: Archive, Big Data, Concurrent, Distributed, Efficient, Flexible | |
| Haugh | Tractable theories of multiple defeasible inheritance in ordinary nonmonotonic logics | |
| CN104699811B (en) | A kind of no intrusive mood form data change record and the method and device of backtracking | |
| KR20180077830A (en) | Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method | |
| Tseng et al. | A successful application of big data storage techniques implemented to criminal investigation for telecom | |
| CN109684331A (en) | A kind of object storage meta data management device and method based on Kudu | |
| Baranowski et al. | Scale out databases for CERN use cases | |
| Heinonen | From Classical DW to Cloud Data Warehouse | |
| Gupta et al. | Data lake ingestion strategies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170901 |