CN110209507A - Data processing method, device, system and storage medium based on message queue - Google Patents
Data processing method, device, system and storage medium based on message queue Download PDFInfo
- Publication number
- CN110209507A CN110209507A CN201910407305.7A CN201910407305A CN110209507A CN 110209507 A CN110209507 A CN 110209507A CN 201910407305 A CN201910407305 A CN 201910407305A CN 110209507 A CN110209507 A CN 110209507A
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- data
- source
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of data processing method based on message queue, device, system and storage mediums, which comprises obtains the source data of at least one source database；The source data is stored to message and transmits queue, and message transmission queue is sent to target message queue by message channel；Corresponding source data is obtained based on the target message queue, corresponding source data is stored to corresponding target database.According to the method for the present invention, device, system and storage medium realize efficient data convergence, the data of separate sources are integrated, the communication specification of uniform data, data is allowed to play bigger benefit so that communicating decoupling between data source library and data warehouse.
Description
Technical field
The present invention relates to data processing fields, relate more specifically to the processing of isomeric data.
Background technique
As computer, the mobile terminal PC, a large amount of of smart phone popularize, the information content rapid growth that the information age generates,
Blowout mode has been presented it, under internet, online, people clothing, food, lodging and transportion -- basic necessities of life are scattered is present in major website, blog, in community, number
According to acquisition, storage, processing quantity be also in multiple increase.Simultaneously as the departments such as government, public security organs Large Information Systems are built
Setting tool has interim and distributivity feature so that bulk information becomes isolated island, how effectively and quickly converge these information at
For problem；The data such as common penthao kettle, talend open studio, clover converge tool on the market；
Kettle is developed based on Java, and interface chemical industry tool can be light easy to get started, but the non-High Availabitity of host-guest architecture, and does not support that real time data is same
Step；Talend, which has graphical interfaces, to be provided with the plug-in mode of eclipse, supports distributed, also support processing in real time, but need
Advanced Edition and price are bought, also will use java exploitation；Clover is the tool after kettle and talend,
User is less, and major advantage is lightweight, is easy insertion, is easy to use, but function is far from, kettle and talend are strong
Greatly.So that gradually expanding for data, data volume will be increased with TB grades even speed of PB rank for the development of Information technology, even if
Powerful ETL tool still can not meet current demand again, and need to put into a large amount of implementation personnel, the user of tool
The not yet done bottleneck for leading to not quickly upper hand and also becoming efficiency of formula, skill.
Therefore, there is the increased requirement for being unable to satisfy data volume in data convergence processing in the prior art, and artificial investment is big,
The problem of low efficiency.
So how quickly and efficient convergence will face the challenge.
Summary of the invention
The present invention is proposed in view of the above problem.The present invention provides based on message queue data processing method,
Device, system and storage medium, by the channel for converging message queue technology as data, so that data source library and data bins
Decoupling is communicated between library, realizes efficient data convergence, the data of separate sources are integrated, the communication rule of uniform data
Model allows data to play bigger benefit.
According to a first aspect of the embodiments of the present invention, a kind of data processing method based on message queue is provided, it is described
Method includes:
Obtain the source data of at least one source database；
The source data is stored to message and transmits queue, and message transmission queue is sent to by message channel
Target message queue；
Corresponding source data is obtained based on the target message queue, corresponding source data is stored to corresponding mesh
Mark database.
Optionally, the source data for obtaining at least one source database, comprising:
It is adapted to using ORM frame or DBHelper class at least one described source database；
The source data is read according to preset data reading mode.
Optionally, be adapted to at least one described source database using DBHelper class include:
It is adapted at least one described source database based on first interface and/or second interface, the first interface is for advising
DBHelper class described in model, the second interface is for extending the DBHelper for needing to obtain the source database information
Class.
Optionally, the source data is stored to message and transmits queue, and message transmission queue is led to by message
Road is sent to target message queue, comprising:
The source data is stored to the message based on scheduled message format and transmits queue；
When a triggered event is detected, the transmission message queue target is sent to by the information channel to disappear
Cease queue.
Optionally, the message channel of the message transmission queue includes message transmission queue sendaisle and message transmission team
Column receiving channel, the method also includes: setting message transmits the messaging parameter of queue sendaisle and message transmission queue connects
The messaging parameter for receiving channel matches to establish the message channel.
Optionally, it is described obtain at least one source database source data or it is described by corresponding source data store to
Corresponding target database is included in line or offline mode.
Optionally, the method also includes: configure the keepalive parameter of TCP/IP to realize that suspension resumes.
According to a second aspect of the embodiments of the present invention, a kind of data processing equipment based on message queue is provided, it is described
Device includes:
Data acquisition module, for obtaining the source data of at least one source database；
Message queue module transmits queue for storing the source data to message, and the message is transmitted queue
Target message queue is sent to by message channel；
Data write. module obtains corresponding source data based on the target message queue, by corresponding source data
It stores to corresponding target database.
According to a third aspect of the embodiments of the present invention, a kind of data processing system based on message queue is provided, including
Memory, processor and it is stored in the computer program run on the memory and on the processor, the processor
The step of realizing first aspect the method for the embodiment of the present invention when executing the computer program.
According to a fourth aspect of the embodiments of the present invention, a kind of computer storage medium is provided, computer is stored thereon with
Program, when the computer program is computer-executed the step of first aspect the method for the realization embodiment of the present invention.
The present invention provides data processing method, device, system and storage medium based on message queue, by by message
Efficient number is realized so that communicating decoupling between data source library and data warehouse in the channel that queue technology is converged as data
According to convergence, the data of separate sources are integrated, the communication specification of uniform data, data is allowed to play bigger benefit.
Detailed description of the invention
The embodiment of the present invention is described in more detail in conjunction with the accompanying drawings, the above and other purposes of the present invention,
Feature and advantage will be apparent.Attached drawing is used to provide to further understand the embodiment of the present invention, and constitutes explanation
A part of book, is used to explain the present invention together with the embodiment of the present invention, is not construed as limiting the invention.In the accompanying drawings,
Identical reference label typically represents same parts or step.
Fig. 1 is a kind of schematic flow chart of data processing method based on message queue according to an embodiment of the present invention；
Fig. 2 is a kind of example of data processing method based on message queue according to an embodiment of the present invention；
Fig. 3 is a kind of schematic block diagram of data processing equipment based on message queue according to an embodiment of the present invention.
Specific embodiment
In order to enable the object, technical solutions and advantages of the present invention become apparent, root is described in detail below with reference to accompanying drawings
According to example embodiments of the present invention.Obviously, described embodiment is only a part of the embodiments of the present invention, rather than this hair
Bright whole embodiments, it should be appreciated that the present invention is not limited by example embodiment described herein.Based on described in the present invention
The embodiment of the present invention, those skilled in the art's obtained all other embodiment in the case where not making the creative labor
It should all fall under the scope of the present invention.
In existing big data treatment process, data access and management are largely all to cross configuration file or code
Mode, can no longer meet the intercommunication of data, to just put forward higher requirements to mass data processing, how effectively converge
Poly-, management, shared data are extremely important.
Data convergence during, common source have RDB type such as Oracle, Mysql, Db2, Sybase,
SqlServer etc.；Nosql type such as Redis, MongoDb, Hbase etc.；File type such as Csv, Txt, picture, audio, video
Deng；Even there are also the data sources such as Web Services, Stream data, Hadoop；When due to the response of different types of source
Between it is also different, target side has little time synchronization process under high concurrent environment, and request often blocks, or even directly results in
Countless row lock table lock, situations such as making to call request cumulative, cause usually beyond maximum connection.It is therefore desirable to a kind of technology
The data of all types of, each platform, each mode can be integrated, break the estrangement between them, the communication specification of uniform data,
Data are allowed to play bigger benefit.
Based on above-mentioned consideration, the embodiment of the invention provides a kind of data processing methods based on message queue.Referring to figure
1, Fig. 1 shows a kind of data processing method based on message queue of the embodiment of the present invention, and the method 100 includes:
Firstly, obtaining the source data of at least one source database in step S110；
In step S120, the source data is stored to message and transmits queue, and by message transmission queue by disappearing
Breath channel is sent to target message queue；
In step S130, corresponding source data is obtained based on the target message queue, corresponding source data is deposited
It stores up to corresponding target database.
Optionally, the data processing method according to an embodiment of the present invention based on message queue can with memory and
It is realized in the unit or system of processor.
Optionally, the data processing method according to an embodiment of the present invention based on message queue can be deployed in personal terminal
Place, smart phone, tablet computer, personal computer etc..
Optionally, the data processing method according to an embodiment of the present invention based on message queue can also be deployed in being distributed
Server end (or cloud).
Wherein, it is provided in an embodiment of the present invention based on the data processing method of message queue by the way that message queue technology is made
Efficient data convergence is realized so that communicating decoupling between data source library and data warehouse for the channel of data convergence, it will not
Data with source integrate, the communication specification of uniform data, and data is allowed to play bigger benefit.
According to embodiments of the present invention, in step S110, the source data for obtaining at least one source database, comprising:
It is adapted to using ORM frame or DBHelper class at least one described source database；
The source data is read according to preset data reading mode.
Wherein, ORM (Object Relational Mapping) be object-oriented object model and relational data it
Between mutual conversion, including based on relevant database data storage, realize a virtual object-oriented data access
Interface.ORM frame, come one relationship map details of description object, is reflected using metadata by using between description object and database
Object in program is persisted in relational database to solve object-oriented and relation data inventory by the metadata penetrated automatically
Mutual unmatched phenomenon technology.
Optionally, the ORM frame includes and is not limited to EF (Entity framework) solid framework, linq to
Sql frame, CYQ.Data automatic programming frame or NHibernate frame.Wherein, EF (Entity framework) entity
Frame includes ado.net frame, and ado.net is the technology of the software application of one group of support exploitation data-oriented；linq to
Sql frame be based in vs create EF mapping below there is also linq to sql classes, suitable for light-duty, small ORM frame
Frame；CYQ.Data automatic programming frame is a kind of ORM frame formed on the basis of EF by reorganization；
NHibernate is the object/relational mapping tool towards .NET environment.
In one embodiment, be adapted to at least one described source database using ORM frame may include: to establish
Database connects and obtains Connection object；Query SQL sentence is assembled according to the input of user；It is established according to SQL statement
Statement object or PreparedStatement object；SQL statement is executed with Connection object, obtains result set
ResultSet object；The data in result set ResultSet object are read one by one；According to the data read, by specific industry
Business logic is calculated；Update SQL statement is assembled again according to the result being calculated；Connection object is reused to execute more
New SQL statement, with the data in more new database；Finally successively close each Statement object and Connection object.
And DBHelper refer to provide database assist aspect software, such as using .NET Framework be operation frame
The SqlHelper component of structure.DBHelper can support multitype database operation include Microsoft Access,
Microsoft SqlServer,Oracle,Mysql.Database is adapted to and can be matched according to different using DBHelper
Set the operational requirements to call different DBHelper to meet for disparate databases.DBHelper class includes database
Helper operates base class HelperBase, database Helper and operates base class SqlServerHelper, Microsoft Access
Database manipulation assistant's class AccessHelper, MySql database manipulation assistant's class MysqlHelper, oracle database behaviour
Make assistant's class OracleHelper.
Optionally, be adapted to at least one described source database using DBHelper class include:
It is adapted at least one described source database based on first interface and/or second interface, the first interface is for advising
DBHelper described in model, the second interface is for extending the DBHelper for needing to obtain the source database information.
Wherein it is possible to define the first interface and the second interface be IDataProvider and
ISchemaProvider, IDataProvider are used for each DBHelper of specification, and ISchemaProvider is needed for extending
Obtain the DBHelper of database information.DataProviderBase is the base class of all DBHelper.Each independence
DBHelper can inherit DataProviderBase, when the DBHelper needs to provide database information, then need to realize
ISchemaProvider interface.
In addition, (the built-in support for Oracle of .NET is it is stated that expired by built-in support SqlServer by .net
), and others such as MySql, SQLi te etc. needs the driving of loading third-party that could use, and will increase database in this way
Dependence of the adaptation procedure for third party's procedure set, it is possible to which the DBHelper for relying on third party's procedure set is made into similar insert
The mode of part had not only reduced procedure set in this way and has relied on but also improve the scalability of adaptive device, written in Java also need according to
Rely in corresponding Jar packet.Such as Sqluldr, gpdist, gpload, copy, bulkinsert that source database has been provided
Related tool can also be integrated to the adaptive device to improve reading, write efficiency.
According to embodiments of the present invention, in step S120, the source data is stored to message and transmits queue, and is disappeared described
Breath transmission queue is sent to target message queue by message channel, comprising:
The source data is stored to the message based on scheduled message format and transmits queue；
When a triggered event is detected, the transmission message queue target is sent to by the information channel to disappear
Cease queue.
Optionally, the message transmission queue and target message queue can store in corresponding queue management device.
Wherein, according to the adaptation of the different types of data such as RDB type, Nosql type, file type, webservices,
Different types of data in not genbank database can uniformly be formed into scheduled text mode (message format) and pass through MQ
(Message queue, message queue) channel is transmitted, and is sent in object queue.Message transmission queue has centainly
Store function, and user can customized storage time according to demand, can cope with and high handle up, high concurrent and support tb grades or more big
Data application scene.
And trigger the mechanism that (Triggering) event is a kind of automatic starting application program.In MQ technology, queue pipe
It manages device and certain condition is called trigger event.If queue is arranged to trigger type, and trigger event has occurred, then team
Column manager will send a triggering message and be referred to as in the queue of starting queue to one.Triggering message is placed to starting queue
Process mean to produce trigger event.Message in processing queue management device is triggering monitoring programme (Trigger-
Monitor Application), his work is to read triggering message and make corresponding place according to the information of triggering message
Reason.Triggering monitoring programme can be the application program that message is read from starting queue.
After queue management device discovery reaches the queue being triggered by a piece of news, the triggering message that it is generated will be deposited
Be put into starting queue in, triggering monitoring programme will from starting queue in take out triggering message, and according to triggering message in content,
Start corresponding message processing program to handle the message being triggered in queue.
In one embodiment, application program A may include: to application program B transmission message queue according to trigger event
Local or Remote application program A, the push into application queue (Application Queue) (i.e. message queue)
A piece of news；
If the triggering type of queue is set as first, when the original depth of queue is 0 (queue is empty), at this moment push
A piece of news will form trigger event into queue, while generate a triggering message, and triggering in message will include process definition
In information because in process definition comprising starting application program B needed for information, trigger message in also contain starting
Information needed for application program B；
Queue management device creation triggering message, and it is pushed into starting queue Initiation relevant to application queue
Queue；
Triggering monitor (Trigger Monitor) obtains triggering from starting queue (Initiation Queue) and disappears
Breath；
Monitor processing triggering message is triggered, the order of starting application program B is issued；
Application program B opens application queue (Application Queue), and handles the message in application queue.
Optionally, the message channel of the message transmission queue includes message transmission queue sendaisle and message transmission team
Column receiving channel, the method also includes: setting message transmits the messaging parameter of queue sendaisle and message transmission queue connects
The messaging parameter for receiving channel matches to establish the message channel.
Wherein, message channel can be for connection to two queue management devices, read and write and disappear in single queue manager
Breath does not need to establish channel；But message is written in a queue management device, and is taken from the queue in another queue management device
Outbound message, this just needs to establish channel；For two queue management devices, whether it is distributed on the same machine, configuration side
Formula is the same, and different is exactly the address configuration of ip.
In one embodiment, message channel is realized using Apache Kafka.Apache Kafka is special disposal stream
The distributed system of formula data, it has good fault-tolerance, high-throughput, supports extending transversely, and allows geographical location point
The stream data of cloth is handled.There is good persistence using the message channel that Apache Kafka is realized, message is directly lasting
Change on ordinary magnetic disc and performance is good, so that data will not lose；And message is to be sequentially written in, consumption data is also that sequence is read
It takes, also ensures that sequence is read while persistence；Also there is high-throughput, even very common hardware Kafka
It can support handling up for the nearly 100,000,000/s of millions of message per second, close to the limit of network interface card, single node supports thousands of a clients.
Apache Kafka realizes that message is distinguished in message channel support by Kafka server and charge machine cluster, and Producer can
To determine which partition is message be dealt into, the sequence of message is exactly that Producer is sent in a partition
The sequence of message can have multiple partition in one topic, and the quantity of specific partition is configurable.
Partition can be set replica copy, and replica copy is there are on different kafka broker nodes, and first
Partition is leader, and others are follower, and message first writes on partition leader, then by
On partition leader push to parition follower.Thus kafka can be with horizontal extension, that is, extends
partition。
In addition, it is parallel to realize that message channel can also support that the load of Hadoop parallel data carries out using Apache Kafka
Processing, and realize and decouple the operation so that asynchronous between Producer and Consumer between application.Moreover, it is also possible to realize
Data copy redundancy, flow load balance are gone above expansible same part data to different broker, when a data disk
The data that break down will not lose, such as 3 copies, and exactly in the case where 3 machine disks all break down, data can just lose, and possess negative
Carry balanced, expansible, online extension, without withdrawing business.
In one embodiment, establishing message channel may include:
Step 1, software preparation comprises at least one of the following: 6.0 or more CentOS, JDK1.8, Zookeeper, or
Apache Kafka；
Step 2, deployment installation, comprises at least one of the following:
1) JDK installation configuration: decompression JDK installation kit simultaneously configures JAVA_HOME environment under the user for needing to install kafka
Variable；
2) zookeeper installation configuration: decompression zookeeper installation kit configures under the config catalogue of decompressing files
Zoo.cfg file starts zookeeper service, such as order wherein needing to configure port numbers and Log Directory under bin catalogue
It enables: bin/zkServer.sh start；
3) kafka installation and deployment: decompression kafka installation kit is modified under decompressing files config catalogue
Port numbers are arranged in server.properties file, and data store path, and zookeeper services the parameters such as link address, if
It is cluster installation, it is only necessary to configure multiple server.properties files；Start kafka service under bin catalogue, such as:
bin/kafka-server-start.sh config/server.properties。
Step 3, building message queue includes:
Can be after message channel be put up, the queue of manual creation needs；Such as theme, queue is can be considered in when creation
Title (topic), the reasonability of the number of partitions (partition) and number of copies (replication-factor), with shell
For script, such as: bin/kafka-topics.sh--create--zookeeper zookeeper ip:port--
partitions 1--replication-factor 1--topic topic_name；
Step 4, test post queue includes:
Whether test post channel is normal, first confirms whether zookeeper and kafka process starts, reuses production, disappears
Expense order is tested；
Wherein, check that process mode is as follows:
Zookeeper process: ps ef | grep zookeeper；
Kafka process: ps ef | grep kafka；
Production, consumption pattern are as follows:
Production: Kafka-console-producer.sh-topic topic_name broker-list ip:port；
Consumption: Kafka-console-consumer.sh topic topic_name-bootstrap-server ip:
port。
Optionally, it is described obtain at least one source database source data or it is described by corresponding source data store to
Corresponding target database is included in line or offline mode.
Optionally, during the source data for obtaining at least one source database, DataProviderBase base can be based on
Class defines data reading mode, that is, defines GetData method, is controlled by different parameters to realize online or offline number
According to acquisition.Such as setting DataSet mode is off-line operation data, then source data is disposably read in memory；Or setting
The mode of DataReader on-line operation data, is written MQ for source data in a manner of CSV, Json, completes the reading of data.
Optionally, corresponding source data is stored to corresponding target database, can be based on
DataProviderBase base class defines data writing mode, that is, defines ExecData method, controlled by different parameters
To realize online or offline data write-in.For example, the data consumed in MQ are passed through the online modes such as offline or CSV, Json
It is written in corresponding database, in practical applications, considers effective support of each object library for data source, most of library pair
It can be supported well in text class CSV, it is proposed that can be efficiently put in storage by CSV mode, that is, complete the write-in of data.
According to embodiments of the present invention, the method also includes: configure the keepalive parameter of TCP/IP to realize that suspension is continuous
It passes.
MQ technology itself is framework on TCP/IP, therefore is had with the TCP/IP characteristic of operating system or network bottom layer
Close relationship.When we to be established between two queue management devices of MQ communication when, due to the channel of MQ be it is unidirectional,
It has to establish two passes (Chanel), for example establishing two types is respectively to send (sender) type and reception
(receiver) communication between two queue management devices is realized in the channel of type, and the both ends MQ in channel utilizes MCA (message
Channel agent) manage the operating statuses such as the start and stop with monitor channel, and to the message sequence number (Message at channel both ends
Sequence Number) etc. carry out coordinated management, to guarantee WebSphere MQ to " the Once and Once Only " of message
Transmission.For MQ, in its system configuration configuration file mqs.ini file, in mqs.ini file, contain to team
Log size, channel attributes and the setting by XA standard and related parameter when database cooperative work of column manager,
In addition to this, a section can be used for controlling the information in relation to TCP/IP characteristic.
Since the MCA of MQ receiving channel is in the passive side of communication, receiving end waits always the message transmitted from transmitting terminal,
Therefore receiving end does not know when that transmitting terminal can stop sending message, does not know when a network fails, transmitting terminal yet
When halted state can be become from working condition.At this moment due to there is network failure, network connection is broken, and transmitting terminal is logical
Road state can be become retrying state from running state, and transmitting terminal can attempt to re-establish network connection, and at this moment receive
The channel at end is not stopped but, and still in the state of a kind of false " running ", corresponding we can obtain " a Channel
The error message of is in use ", causes transmitting terminal that can not restart.Because not having when the starting channel transmitting terminal MCA and for a long time
It disconnects, network failure at this moment occurs, the socket connection of TCP/IP is destroyed, when transmitting terminal stops channel and opens again
When dynamic, it needs to establish a new socket connection, and receiving end remains in original RECEIVE calling, it
Failure is established in the inconsistent therefore new socket connection of the socket feature socket feature new with sender.It is examined based on above-mentioned
Consider, the response speed of TCP/IP can be improved by configuring TCP/IP keepalive parameter, thus when realizing network failure
MQ can disconnect channel connection rapidly to restart channel, realize the function that suspension resumes.At this point, in transmitting terminal, MQ's
Channel could be become retrying state from running state, meanwhile, in receiving end, the channel of MQ could be by running state
Become not found state, in this way, channel can just start against connection when network or host restore again, restores
Running state.
According to embodiments of the present invention, the method also includes: when by the message transmission queue by message channel transmission
To when target message queue failure, the information of the failure is sent to message transmission queue or target message queue.
Wherein, if network failure or transmission abnormality occurs in transmission process in message, it will lead to transmission failure；Generally
It is production message and consumption message respectively including two kinds of situations.
When failure, which appears in the producer, produces message, it is confirm mode, i.e. sender that proxy server, which can be set,
Affirmation mode, the benefit of the mode are that message has a unique ID, after message is assigned to corresponding queue, agency service
Device can return to one confirmation message of the producer, and whether the producer is succeeded by judging that the information produces come confirmation message, so
When message channel is there are when failure, the producer can be got by confirmation message and unsuccessfully be identified, and message can be sent again, thus
Guarantee that creation data is not lost.
It is according to queue current pointer when subscribing to message due to consumer when failure, which appears in consumer, subscribes to message
Position is come what is carried out, if breaking down at this time, proxy server does not receive the confirmation message of consumer's transmission, proxy server
Automatically current consumer can be disconnected, and message is sent to the consumer of another subscription, while current consumption can be returned to
One error message of person informs consumer's mistake producing cause, to guarantee that consumer can get new information in time.
Below with reference to Fig. 2, with specific example to the embodiment of the present invention based on the data processing method of message queue into
Row explanation.Fig. 2 shows a kind of example of the data processing method based on message queue according to an embodiment of the present invention, such as Fig. 2
Shown, the method 200 includes:
Firstly, the source database that adaptation is different, specifically includes: defining the first interface and the second interface is
IDataProvider and ISchemaProvider, IDataProvider are used for each DBHelper of specification,
ISchemaProvider is for extending the DBHelper for needing to obtain database information.DataProviderBase is all
The base class of DBHelper, each independent DBHelper can inherit DataProviderBase, when the DBHelper needs to provide
When database information, then need to realize ISchemaProvider interface；
Then, the source data for obtaining at least one source database 210, specifically includes: above-mentioned between disparate databases
After being adapted to, source data is read from source database 210；Data reading mode is defined based on DataProviderBase base class,
GetData method is defined, is controlled by different parameters to realize online or offline data acquisition, DataSet is set
Mode is off-line operation data, then source data is disposably read in memory；Or setting DataReader on-line operation data
MQ is written in source data by mode in a manner of CSV, Json, completes the reading of data；
Then, the source data is stored to message and transmits queue, and message transmission queue is passed through into message channel
220 are sent to target message queue, specifically include: realizing message channel 220 using Apache Kafka, be based on scheduled message
The source data is stored to the message and transmits queue by format；When a triggered event is detected, by the transmission message queue
The target message queue is sent to by the information channel 220；
Then, corresponding source data is obtained based on the target message queue, corresponding source data is stored to right
The target database 230 answered, specifically includes: define ExecData method, controlled by different parameters with realizes online or
Corresponding number of targets is written by online modes such as offline or CSV, Json in the data consumed in MQ by offline data write-in
According in library 230, the write-in of data is completed.
In addition, having improved the response speed of TCP/IP since the keepalive parameter that the method 200 can also configure TCP/IP
Degree ensure that the continuity and validity of data transmission, improve the sink-efficiency of data to realize that suspension resumes.
It should be noted that the data transmission method for being produced, being consumed by MQ, it can also be according to user demand to data
It is subscribed to, can be produced with active control, consumption access quantity or request frequency, saving queue service resource avoid providing on demand
Source contention leads to delay machine, has the function of message fault tolerant mechanism, avoids process blocking, message queue data stacking, service ability drop
It is low, have without service fault-tolerant processing mechanism, avoids when queue dependence storage service is unavailable, lead to message production, consumption
The case where failing.
It follows that a kind of data processing method based on message queue according to an embodiment of the present invention passes through MQ technology
Efficient data are realized so that communicating decoupling between data source library and database in the channel that message channel is converged as data
Convergence, the data of separate sources is integrated, the communication specification of uniform data, and data is allowed to play bigger benefit.
Fig. 3 shows a kind of the schematic of data processing equipment 300 based on message queue according to an embodiment of the present invention
Block diagram.As shown in figure 3, a kind of data processing equipment 300 based on message queue according to an embodiment of the present invention includes:
Data acquisition module 310, for obtaining the source data of at least one source database；
Message queue module 320 transmits queue for storing the source data to message, and the message is transmitted team
Column are sent to target message queue by message channel；
Data write. module 330 obtains corresponding source data based on the target message queue, by corresponding source number
According to storing to corresponding target database.
According to embodiments of the present invention, the data acquisition module 310 includes:
Adaptation module 311, for being fitted using ORM frame or DBHelper class at least one described source database
Match；
Read module 312, for reading the source data according to preset data reading mode.
Wherein, ORM (Object Relational Mapping) be object-oriented object model and relational data it
Between mutual conversion, including based on relevant database data storage, realize a virtual object-oriented data access
Interface.ORM frame, come one relationship map details of description object, is reflected using metadata by using between description object and database
Object in program is persisted in relational database to solve object-oriented and relation data inventory by the metadata penetrated automatically
Mutual unmatched phenomenon technology.
Optionally, the ORM frame includes and is not limited to EF (Entity framework) solid framework, linq to
Sql frame, CYQ.Data automatic programming frame or NHibernate frame.Wherein, EF (Entity framework) entity
Frame includes ado.net frame, and ado.net is the technology of the software application of one group of support exploitation data-oriented；linq to
Sql frame be based in vs create EF mapping below there is also linq to sql classes, suitable for light-duty, small ORM frame
Frame；CYQ.Data automatic programming frame is a kind of ORM frame formed on the basis of EF by reorganization；
NHibernate is the object/relational mapping tool towards .NET environment.
Optionally, adaptation module 311 is also used to:
It is adapted at least one described source database based on first interface and/or second interface, the first interface is for advising
DBHelper described in model, the second interface is for extending the DBHelper for needing to obtain the source database information.
Wherein it is possible to define the first interface and the second interface be IDataProvider and
ISchemaProvider, IDataProvider are used for each DBHelper of specification, and ISchemaProvider is needed for extending
Obtain the DBHelper of database information.DataProviderBase is the base class of all DBHelper.Each independence
DBHelper can inherit DataProviderBase, when the DBHelper needs to provide database information, then need to realize
ISchemaProvider interface.
According to embodiments of the present invention, the message queue module 320 includes:
Message establishes module 321, transmits for being stored the source data to the message based on scheduled message format
Queue；
Sending module 322, for when a triggered event is detected, the transmission message queue being passed through the information channel
It is sent to the target message queue.
Optionally, the message transmission queue and target message queue can store in corresponding queue management device.
Wherein, according to the adaptation of the different types of data such as RDB type, Nosql type, file type, webservices,
Different types of data in not genbank database can uniformly be formed to scheduled text mode and pass through MQ (Message
Queue, message queue) channel is transmitted, and is sent in object queue.Message transmission queue has certain store function,
And user can customized storage time according to demand, can cope with it is high handle up, high concurrent and the big data application for supporting tb grades or more
Scene.
And trigger the mechanism that (Triggering) event is a kind of automatic starting application program.In MQ technology, queue pipe
It manages device and certain condition is called trigger event.If queue is arranged to trigger type, and trigger event has occurred, then team
Column manager will send a triggering message and be referred to as in the queue of starting queue to one.Triggering message is placed to starting queue
Process mean to produce trigger event.Message in processing queue management device is triggering monitoring programme (Trigger-
Monitor Application), his work is to read triggering message and make corresponding place according to the information of triggering message
Reason.Triggering monitoring programme is that the application program of message is read from starting queue.
After queue management device discovery reaches the queue being triggered by a piece of news, the triggering message that it is generated will be deposited
Be put into starting queue in, triggering monitoring programme will from starting queue in take out triggering message, and according to triggering message in content,
Start corresponding message processing program to handle the message being triggered in queue.
Optionally, the sending module 322 is also used to: setting message transmits the messaging parameter and message of queue sendaisle
The messaging parameter of transmission queue receiving channel matches to establish the message channel.
Wherein, message channel is for connection to two queue management devices, reads and writes message not in single queue manager
It needs to establish channel；But message is written in a queue management device, and disappears from the queue taking-up in another queue management device
Breath, this just needs to establish channel；It for two queue management devices, is whether distributed on the same machine, configuration mode is all
Be it is the same, different is exactly the address configuration of ip.
In one embodiment, message channel is realized using Apache Kafka.
In one embodiment, establishing message channel may include:
Step 1, software preparation comprises at least one of the following: 6.0 or more CentOS, JDK1.8, Zookeeper, or
Apache Kafka；
Step 2, deployment installation, comprises at least one of the following:
1) JDK installation configuration: decompression JDK installation kit simultaneously configures JAVA_HOME environment under the user for needing to install kafka
Variable；
2) zookeeper installation configuration: decompression zookeeper installation kit configures under the config catalogue of decompressing files
Zoo.cfg file starts zookeeper service, such as order wherein needing to configure port numbers and Log Directory under bin catalogue
It enables: bin/zkServer.sh start；
3) kafka installation and deployment: decompression kafka installation kit is modified under decompressing files config catalogue
Port numbers are arranged in server.properties file, and data store path, and zookeeper services the parameters such as link address, if
It is cluster installation, it is only necessary to configure multiple server.properties files；Start kafka service under bin catalogue, such as:
bin/kafka-server-start.sh config/server.properties。
Step 3, building message queue includes:
Can be after message channel be put up, the queue of manual creation needs；Such as theme, queue is can be considered in when creation
Title (topic), the reasonability of the number of partitions (partition) and number of copies (replication-factor), with shell
For script, such as: bin/kafka-topics.sh--create--zookeeper zookeeper ip:port--
partitions 1--replication-factor 1--topic topic_name；
Step 4, test post queue includes:
Whether test post channel is normal, first confirms whether zookeeper and kafka process starts, reuses production, disappears
Expense order is tested；
Wherein, check that process mode is as follows:
Zookeeper process: ps ef | grep zookeeper；
Kafka process: ps ef | grep kafka；
Production, consumption pattern are as follows:
Production: Kafka-console-producer.sh-topic topic_name broker-list ip:port；
Consumption: Kafka-console-consumer.sh topic topic_name-bootstrap-server ip:
port。
Optionally, the data acquisition module 310 or the Data write. module 330 are included in line or offline side
Formula.
Optionally, during the source data for obtaining at least one source database, DataProviderBase base can be based on
Class defines data reading mode, that is, defines GetData method, is controlled by different parameters to realize online or offline number
According to acquisition.Such as setting DataSet mode is off-line operation data, then source data is disposably read in memory；Or setting
The mode of DataReader on-line operation data, is written MQ for source data in a manner of CSV, Json, completes the reading of data.
Optionally, corresponding source data is stored to corresponding target database, can be based on
DataProviderBase base class defines data writing mode, that is, defines ExecData method, controlled by different parameters
To realize online or offline data write-in.For example, the data consumed in MQ are passed through the online modes such as offline or CSV, Json
It is written in corresponding database, in practical applications, considers effective support of each object library for data source, most of library pair
It can be supported well in text class CSV, it is proposed that can be efficiently put in storage by CSV mode, that is, complete the write-in of data.
According to embodiments of the present invention, described device 300 further include: parameter configuration module 340, for configuring TCP/IP's
Keepalive parameter is to realize that suspension resumes.
According to embodiments of the present invention, described device 300 can also include: abnormal module, transmit the message for working as
Queue by message channel be sent to target message queue failure when, by the information of the failure be sent to message transmission queue or
Target message queue.
Wherein, if network failure or transmission abnormality occurs in transmission process in message, it will lead to transmission failure；Generally
It is production message and consumption message respectively including two kinds of situations.
When failure, which appears in the producer, produces message, it is confirm mode, i.e. sender that proxy server, which can be set,
Affirmation mode, the benefit of the mode are that message has a unique ID, after message is assigned to corresponding queue, agency service
Device can return to one confirmation message of the producer, and whether the producer is succeeded by judging that the information produces come confirmation message, so
When message channel is there are when failure, the producer can be got by confirmation message and unsuccessfully be identified, and message can be sent again, thus
Guarantee that creation data is not lost.
It is according to queue current pointer when subscribing to message due to consumer when failure, which appears in consumer, subscribes to message
Position is come what is carried out, if breaking down at this time, proxy server does not receive the confirmation message of consumer's transmission, proxy server
Automatically current consumer can be disconnected, and message is sent to the consumer of another subscription, while current consumption can be returned to
One error message of person informs consumer's mistake producing cause, to guarantee that consumer can get new information in time.
The present invention provides a kind of data processing equipments based on message queue, by using message queue technology as data
Efficient data convergence is realized, by separate sources so that communicating decoupling between data source library and data warehouse in the channel of convergence
Data integrate, the communication specification of uniform data allows data to play bigger benefit.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician
Each specific application can be used different methods to achieve the described function, but this realization is it is not considered that exceed
The scope of the present invention.
According to another aspect of the present invention, a kind of data processing system based on message queue is provided, including memory, with
And processor；
The memory storage is for realizing in the data processing method according to an embodiment of the present invention based on message queue
Corresponding steps program code；
The processor is implemented with executing above according to the present invention for running the program code stored in the memory
The corresponding steps of the data processing method based on message queue of example.
In one embodiment, it executes when said program code is run by the processor and implements above according to the present invention
The corresponding steps of the aforementioned data processing method based on message queue of example.
In addition, according to another aspect of the present invention, additionally providing a kind of computer readable storage medium, it is situated between in the storage
Program instruction is stored in matter, when described program instruction is run by computer or processor for executing the embodiment of the present invention
The corresponding steps of data processing method based on message queue, and message team is based on for realizing according to an embodiment of the present invention
The data processing system of column.
Illustratively, the computer readable storage medium can be appointing for one or more computer readable storage mediums
Meaning combination.
In one embodiment, the computer program instructions may be implemented real according to the present invention when being run by computer
Apply the aforementioned data processing method based on message queue of example.
Although describing example embodiment by reference to attached drawing here, it should be understood that above example embodiment are only exemplary
, and be not intended to limit the scope of the invention to this.Those of ordinary skill in the art can carry out various changes wherein
And modification, it is made without departing from the scope of the present invention and spiritual.All such changes and modifications are intended to be included in appended claims
Within required the scope of the present invention.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician
Each specific application can be used different methods to achieve the described function, but this realization is it is not considered that exceed
The scope of the present invention.
It will be understood to those skilled in the art that any combination pair can be used other than mutually exclusive between feature
All features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed any method
Or all process or units of equipment are combined.Unless expressly stated otherwise, this specification (is wanted including adjoint right
Ask, make a summary and attached drawing) disclosed in each feature can be replaced with an alternative feature that provides the same, equivalent, or similar purpose.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
The above description is merely a specific embodiment or to the explanation of specific embodiment, protection of the invention
Range is not limited thereto, and anyone skilled in the art in the technical scope disclosed by the present invention, can be easily
Expect change or replacement, should be covered by the protection scope of the present invention.Protection scope of the present invention should be with claim
Subject to protection scope.
Claims (10)
1. a kind of data processing method based on message queue, which is characterized in that the described method includes:
Obtain the source data of at least one source database；
The source data is stored to message and transmits queue, and message transmission queue is sent to target by message channel
Message queue；
Corresponding source data is obtained based on the target message queue, corresponding source data is stored to corresponding number of targets
According to library.
2. the method as described in claim 1, which is characterized in that the source data for obtaining at least one source database, comprising:
It is adapted to using ORM frame or DBHelper class at least one described source database；
The source data is read according to preset data reading mode.
3. method according to claim 2, which is characterized in that using DBHelper class and at least one described source database into
Row is adapted to
It is adapted at least one described source database based on first interface and/or second interface, the first interface is used for specification institute
DBHelper class is stated, the second interface is for extending the DBHelper class for needing to obtain the source database information.
4. the method as described in claim 1, which is characterized in that store the source data to message and transmit queue, and by institute
It states message transmission queue and target message queue is sent to by message channel, comprising:
The source data is stored to the message based on scheduled message format and transmits queue；
When a triggered event is detected, the transmission message queue is sent to the target message team by the information channel
Column.
5. the method as described in claim 1, which is characterized in that the message channel of the message transmission queue includes message transmission
Queue sendaisle and message transmit queue receiving channel, the method also includes: setting message transmits queue sendaisle
Messaging parameter and the messaging parameter of message transmission queue receiving channel match to establish the message channel.
6. the method as described in claim 1, which is characterized in that the source data or described for obtaining at least one source database
Corresponding source data is stored to corresponding target database and is included in line or offline mode.
7. the method as described in claim 1, which is characterized in that the method also includes: configure the KeepAlive ginseng of TCP/IP
Number is to realize that suspension resumes.
8. a kind of data processing equipment based on message queue, which is characterized in that described device includes:
Data acquisition module, for obtaining the source data of at least one source database；
Message queue module transmits queue for storing the source data to message, and message transmission queue is passed through
Message channel is sent to target message queue；
Data write. module obtains corresponding source data based on the target message queue, corresponding source data is stored
To corresponding target database.
9. a kind of data processing system based on message queue, including memory, processor and be stored on the memory and
The computer program run on the processor, which is characterized in that the processor is realized when executing the computer program
The step of any one of claims 1 to 7 the method.
10. a kind of computer storage medium, is stored thereon with computer program, which is characterized in that the computer program is counted
The step of calculation machine realizes any one of claims 1 to 7 the method when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910407305.7A CN110209507A (en) | 2019-05-16 | 2019-05-16 | Data processing method, device, system and storage medium based on message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910407305.7A CN110209507A (en) | 2019-05-16 | 2019-05-16 | Data processing method, device, system and storage medium based on message queue |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110209507A true CN110209507A (en) | 2019-09-06 |
Family
ID=67787475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910407305.7A Pending CN110209507A (en) | 2019-05-16 | 2019-05-16 | Data processing method, device, system and storage medium based on message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209507A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795257A (en) * | 2019-09-19 | 2020-02-14 | 平安科技（深圳）有限公司 | Method, device and equipment for processing multi-cluster operation records and storage medium |
| CN110837573A (en) * | 2019-11-08 | 2020-02-25 | 苏州思必驰信息科技有限公司 | Distributed audio file storage and reading method and system |
| CN110855792A (en) * | 2019-11-19 | 2020-02-28 | 南京领行科技股份有限公司 | Message pushing method, device, equipment and medium |
| CN111008254A (en) * | 2019-12-03 | 2020-04-14 | 北京锐安科技有限公司 | Object creating method and device, computer equipment and storage medium |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111209299A (en) * | 2020-04-20 | 2020-05-29 | 四川新网银行股份有限公司 | Real-time judgment method for anti-fraud of finance |
| CN112860412A (en) * | 2021-03-12 | 2021-05-28 | 网易（杭州）网络有限公司 | Service data processing method and device, electronic equipment and storage medium |
| WO2021135255A1 (en) * | 2019-12-31 | 2021-07-08 | 深圳云天励飞技术股份有限公司 | Message queue-based method for managing row and column permissions, and related device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488690A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Data integrating system and data integrating method |
| CN106452819A (en) * | 2015-08-13 | 2017-02-22 | 腾讯科技（深圳）有限公司 | Data acquisition system and data acquisition method |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108762738A (en) * | 2018-04-23 | 2018-11-06 | 北京兰光创新科技有限公司 | The data collecting system of hardware and software platform |
| CN109284334A (en) * | 2018-09-05 | 2019-01-29 | 拉扎斯网络科技（上海）有限公司 | Real-time database synchronization method and device, electronic equipment and storage medium |
-
2019
- 2019-05-16 CN CN201910407305.7A patent/CN110209507A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488690A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Data integrating system and data integrating method |
| CN106452819A (en) * | 2015-08-13 | 2017-02-22 | 腾讯科技（深圳）有限公司 | Data acquisition system and data acquisition method |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108762738A (en) * | 2018-04-23 | 2018-11-06 | 北京兰光创新科技有限公司 | The data collecting system of hardware and software platform |
| CN109284334A (en) * | 2018-09-05 | 2019-01-29 | 拉扎斯网络科技（上海）有限公司 | Real-time database synchronization method and device, electronic equipment and storage medium |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795257B (en) * | 2019-09-19 | 2023-06-16 | 平安科技（深圳）有限公司 | Method, device, equipment and storage medium for processing multi-cluster job record |
| CN110795257A (en) * | 2019-09-19 | 2020-02-14 | 平安科技（深圳）有限公司 | Method, device and equipment for processing multi-cluster operation records and storage medium |
| CN110837573B (en) * | 2019-11-08 | 2022-07-08 | 思必驰科技股份有限公司 | Distributed audio file storage and reading method and system |
| CN110837573A (en) * | 2019-11-08 | 2020-02-25 | 苏州思必驰信息科技有限公司 | Distributed audio file storage and reading method and system |
| CN110855792A (en) * | 2019-11-19 | 2020-02-28 | 南京领行科技股份有限公司 | Message pushing method, device, equipment and medium |
| CN110855792B (en) * | 2019-11-19 | 2023-04-18 | 南京领行科技股份有限公司 | Message pushing method, device, equipment and medium |
| CN111008254A (en) * | 2019-12-03 | 2020-04-14 | 北京锐安科技有限公司 | Object creating method and device, computer equipment and storage medium |
| CN111008254B (en) * | 2019-12-03 | 2023-08-29 | 北京锐安科技有限公司 | Object creation method, device, computer equipment and storage medium |
| CN111200637B (en) * | 2019-12-20 | 2022-07-08 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| WO2021135255A1 (en) * | 2019-12-31 | 2021-07-08 | 深圳云天励飞技术股份有限公司 | Message queue-based method for managing row and column permissions, and related device |
| CN111209299A (en) * | 2020-04-20 | 2020-05-29 | 四川新网银行股份有限公司 | Real-time judgment method for anti-fraud of finance |
| CN112860412A (en) * | 2021-03-12 | 2021-05-28 | 网易（杭州）网络有限公司 | Service data processing method and device, electronic equipment and storage medium |
| CN112860412B (en) * | 2021-03-12 | 2023-10-20 | 网易（杭州）网络有限公司 | Service data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209507A (en) | Data processing method, device, system and storage medium based on message queue | |
| US10642829B2 (en) | Distributed and optimized garbage collection of exported data objects | |
| US9514208B2 (en) | Method and system of stateless data replication in a distributed database system | |
| US10402115B2 (en) | State machine abstraction for log-based consensus protocols | |
| US8983902B2 (en) | Transparent caching of configuration data | |
| US8108623B2 (en) | Poll based cache event notifications in a distributed cache | |
| US10261872B2 (en) | Multilevel disaster recovery | |
| EP3617884B1 (en) | Adapter extension for inbound messages from robotic automation platforms to unified automation platform | |
| CN109522287A (en) | Monitoring method, system, equipment and the medium of distributed document storage cluster | |
| US9753966B1 (en) | Providing a distributed transaction information storage service | |
| CN102880475A (en) | Real-time event handling system and method based on cloud computing in computer software system | |
| US9152441B2 (en) | Systems and methods involving virtual machine host isolation over a network via a federated downstream cluster | |
| WO2004040420A2 (en) | Generic framework for applying object-oriented models to multi-tiered enterprise applications | |
| CN114253748B (en) | Message processing system and message processing method | |
| CN110071873A (en) | A kind of method, apparatus and relevant device sending data | |
| US20150356161A1 (en) | Transport layer abstraction for clustering implementation | |
| CN108874567A (en) | A kind of service processing method and system | |
| CN113630310B (en) | Distributed high-availability gateway system | |
| CN109639773A (en) | A kind of the distributed data cluster control system and its method of dynamic construction | |
| CN115115329A (en) | Manufacturing middleware and cloud manufacturing framework for intelligent production line | |
| CN109144748A (en) | A kind of server, distributed server cluster and its state-driven method | |
| CN115185705A (en) | Message notification method, device, medium and equipment | |
| US11637737B2 (en) | Network data management framework | |
| CN118193238A (en) | Service information processing method, device, equipment and storage medium | |
| JP2009515474A (en) | Independent message store and message transport agent |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190906 |
|
| RJ01 | Rejection of invention patent application after publication |