CN106293960B - A kind of method and system for realizing data conversion based on strom and memory grid - Google Patents

A kind of method and system for realizing data conversion based on strom and memory grid Download PDF

Info

Publication number
CN106293960B
CN106293960B CN201610599068.5A CN201610599068A CN106293960B CN 106293960 B CN106293960 B CN 106293960B CN 201610599068 A CN201610599068 A CN 201610599068A CN 106293960 B CN106293960 B CN 106293960B
Authority
CN
China
Prior art keywords
data
node
memory grid
processing
transmitting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610599068.5A
Other languages
Chinese (zh)
Other versions
CN106293960A (en
Inventor
连城
郭海涛
孙而焓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Electric Fufu Mdt Infotech Ltd
Original Assignee
China Electric Fufu Mdt Infotech Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Electric Fufu Mdt Infotech Ltd filed Critical China Electric Fufu Mdt Infotech Ltd
Priority to CN201610599068.5A priority Critical patent/CN106293960B/en
Publication of CN106293960A publication Critical patent/CN106293960A/en
Application granted granted Critical
Publication of CN106293960B publication Critical patent/CN106293960B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

The present invention provides a kind of method for realizing data conversion based on strom and memory grid, a plurality of transmitting nodes, processing node, memory grid node and a preheating that storm need to be provided and more new node, and each processing node corresponds to a transmitting node and a memory grid node, which comprises acquisition data are into message queue;One processing node is bound with a memory grid node, realizes shared memory space and synchronous averaging, closing;After preheating receives load request of data with more new node, all processing nodes is called to obtain the first data deposit memory grid node to be matched from associated data source;Transmitting node reads the second data to be matched in message queue to processing node;Processing node obtains the first data from memory grid node again and is matched, and the data after matching are sent to corresponding output node.The present invention also provides a kind of systems for realizing data conversion based on strom and memory grid, improve data-handling efficiency.

Description

A kind of method and system for realizing data conversion based on strom and memory grid
Technical field
The present invention relates to technical field of data processing, more particularly to one kind to realize that data turn based on strom and memory grid The method and system changed.
Background technique
As data to be treated in the unit time are more and more, be treated as in real time many mechanisms face pair it is primary Challenge, and storm is as a kind of low delay, stream calculation frame highly fault tolerant is widely used in more and more projects and software In system, strom reads data from message queue, is sent to processing node one by one by transmitting node and carries out data conversion behaviour Make, data transformation operations mainly carry out corresponding operating according to preset business, operation, the warp such as generally convert or abandon Data after crossing data transformation operations are retransmited to output node.When storm cooperates other traditional matched data sources (such as redis) When carrying out data transformation operations, since storm processing speed is fast, when carrying out Data Matching, traditional data in data transformation operations Source is difficult to meet the high speed processing ability of storm low delay, will cause the excessively high then initiation data product of storm processing node delay The problem of pressure, and when storm cooperates other traditional matched data sources (such as redis) to carry out data conversions, the change of matched data It is more very complicated, substantially reduce data-handling efficiency.
Summary of the invention
One of the technical problem to be solved in the present invention is to provide a kind of based on strom and memory grid realization data turn The method changed reduces data and overstocks, improves data-handling efficiency.
One of the technical problem to be solved in the present invention is achieved in that a kind of based on strom and memory grid realization number According to the method for conversion, need to provide a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes of storm with And a preheating and more new node, and an each processing node correspondence one transmitting node and memory grid node, the side Method includes the following steps:
Step 10, acquisition data, and collected data are entered into message team in the form of data flow in order one by one In column;
Step 20 binds each processing node and a memory grid node, the processing node after binding with it is interior Grid node shared memory space is deposited, and realizes synchronous averaging and close synchronously;
Step 30, before carrying out conversion operation, after the preheating receives load request of data with more new node, call all Processing node obtains the first data to be matched from associated data source, and first data are stored in the memory grid respectively bound In node;
Step 40 reads the second data to be matched by the transmitting node from message queue, and with data flow Form is emitted to processing node one by one;
It is obtained in step 50, the memory grid node bound by the second data that the processing node will receive with itself The first data got are matched, and the data that matching is completed are sent to corresponding output node and are further processed.
Further, the method also includes step 60, when the associated data source change need to update when, pause Storm topology, transmitting node stop transmitting data, after waiting pending datas to update, then step 30 are executed, by updated first data After being stored in memory grid node, service topology is restarted, completes online updating operation.
Further, " in the memory grid node for respectively binding first data deposit " tool in the step 30 Body is, the first data that each processing node will acquire are stored in the memory grid node of itself binding, and with HashMap<key, value>form stored.
Further, the step 50 specifically: by it is described processing node receive transmitting node transmitting come second After data, handled one by one, obtained from the memory grid node of the processing node binding HashMap of storage < Key, value > format data are matched, and matched data are sent to corresponding output node one by one and are carried out in next step Processing.
Further, the associated data source includes structured document, relevant database, non-relational database or disappears Cease queue.
The second technical problem to be solved by the present invention is to provide a kind of based on strom and memory grid realization data turn The system changed reduces data and overstocks, improves data-handling efficiency.
The second technical problem to be solved by the present invention is achieved in that a kind of based on strom and memory grid realization number According to the system of conversion, need to provide a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes of storm with And a preheating and more new node, and an each processing node correspondence one transmitting node and memory grid node, the system System includes data acquisition module, node binding module, data acquisition module, data transmission module and data conversion module:
The data acquisition module, for acquiring data, and by collected data one by one in order with the shape of data flow Formula is entered into message queue;
The node binding module, for binding each processing node and a memory grid node, after binding Processing node and memory grid nodes sharing memory headroom, and realize synchronous averaging and close synchronously;
The data acquisition module, for before carrying out conversion operation, the preheating to receive load data with more new node After request, all processing nodes is called to obtain the first data to be matched from associated data source, first data is stored in each From in the memory grid node of binding;
The data transmission module, for reading the second number to be matched from message queue by the transmitting node According to, and it is emitted to processing node one by one in the form of data flow;
The data conversion module, the second data for will be received from data transmission module by the processing node It is matched with the first data for getting in the memory grid node of itself binding, and data that matching is completed is sent to pair The output node answered is further processed.
Further, the system also includes an online updating modules, need for changing in the associated data source When updating, pause storm topology, transmitting node stops transmitting data, after waiting pending datas to update, then executes data acquisition mould Block after updated first data are stored in memory grid node, restarts service topology, completes online updating operation.
Further, " first data are stored in the memory grid section respectively bound in the data acquisition module In point " specifically, the first data that each processing node will acquire are stored in the memory grid node of itself binding, and With HashMap<key, value>form stored.
Further, the data conversion module specifically: for receiving transmitting node hair by the processing node It after penetrating the second data come, is handled one by one, obtains storage from the memory grid node of the processing node binding HashMap<key, value>format data are matched, and by matched data be sent to one by one corresponding output node into Row is handled in next step.
Further, the associated data source includes structured document, relevant database, non-relational database or disappears Cease queue.
The present invention has the advantage that
1, storm and memory grid are bound, and associated data can be stored in memory grid in advance, reduced because from outside The time of associated data waste is obtained, the delay of Data Matching operation is reduced, data is not will cause and overstocks, improves data processing effect Rate;
2, the feature fast using memory grid node read or write speed, gives full play to the high speed processing ability of storm low delay, Improve data conversion efficiency;
3, the present invention utilizes the high fault tolerance of storm, guarantees the high availability of storm cluster and memory grid cluster;
4, the present invention will need matched data to be stored in memory grid, can lead in not traffic affecting It crosses preheating and more new node carries out online updating.
Detailed description of the invention
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is a kind of method flow diagram that data conversion is realized based on strom and memory grid of the present invention.
Fig. 2 is a kind of system logic block diagram that data conversion is realized based on strom and memory grid of the present invention.
Fig. 3 is a kind of schematic illustration that data conversion is realized based on strom and memory grid of the present invention.
Specific embodiment
As shown in figures 1 and 3, a method of data conversion is realized based on strom and memory grid, storm need to be provided A plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and it is every The one corresponding transmitting node of processing node and a memory grid node, described method includes following steps:
Step 10, acquisition data, and collected data are entered into message team in the form of data flow in order one by one In column;
Step 20 binds each processing node and a memory grid node, the processing node after binding with it is interior Grid node shared memory space is deposited, and realizes synchronous averaging and close synchronously;
Step 30, before carrying out conversion operation, after the preheating receives load request of data with more new node, call all Processing node obtains the first data to be matched from associated data source, and first data are stored in the memory grid respectively bound In node, the associated data source includes structured document, relevant database, non-relational database or message queue, institute It states " in the memory grid node for respectively binding first data deposit " specifically, each processing node will acquire The first data be stored in the memory grid node of itself binding, and with HashMap<key, value>form stored;
Step 40 reads the second data to be matched by the transmitting node from message queue, and with data flow Form is emitted to processing node one by one;
It is obtained in step 50, the memory grid node bound by the second data that the processing node will receive with itself The first data got are matched, and the data that matching is completed are sent to corresponding output node and are further processed, The step 50 specifically: by it is described processing node receive transmitting node transmitting come the second data after, located one by one Reason obtains HashMap<key of storage, value>format data from the memory grid node of the processing node binding It is matched, and matched data is sent to corresponding output node one by one and are further processed.
The method also includes step 60, when the associated data source change need to update when, pause storm open up It flutters, transmitting node stops transmitting data, after waiting pending datas to update, then executes step 30, will be in the deposit of updated first data After depositing grid node, service topology is restarted, completes online updating operation.
As shown in Figures 2 and 3, a kind of system for realizing data conversion based on strom and memory grid, need to provide storm A plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and it is every The one corresponding transmitting node of processing node and a memory grid node, the system comprises data acquisition module, nodes to tie up Cover half block, data acquisition module, data transmission module and data conversion module:
The data acquisition module, for acquiring data, and by collected data one by one in order with the shape of data flow Formula is entered into message queue;
The node binding module, for binding each processing node and a memory grid node, after binding Processing node and memory grid nodes sharing memory headroom, and realize synchronous averaging and close synchronously;
The data acquisition module, for before carrying out conversion operation, the preheating to receive load data with more new node After request, all processing nodes is called to obtain the first data to be matched from associated data source, first data is stored in each It is described specific " in the memory grid node for respectively binding first data deposit " from the memory grid node of binding For the first data that each processing node will acquire are stored in the memory grid node of itself binding, and with HashMap <key, value>form stored;
The data transmission module, for reading the second number to be matched from message queue by the transmitting node According to, and it is emitted to processing node one by one in the form of data flow;
The data conversion module, the second data for will be received from data transmission module by the processing node It is matched with the first data for getting in the memory grid node of itself binding, and data that matching is completed is sent to pair The output node answered is further processed;The data conversion module specifically: for being received by the processing node It after the second data that transmitting node transmitting comes, is handled one by one, from the memory grid node of the processing node binding HashMap<key of storage is obtained, value>format data are matched, and matched data are sent to one by one corresponding Output node is further processed.
The system also includes an online updating module, for the associated data source change need to update when, Suspend storm topology, transmitting node stops transmitting data, after waiting pending datas to update, then data acquisition module executed, after update The first data deposit memory grid node after, restart service topology, complete online updating operation.
Below with reference to a specific embodiment, the present invention will be further described:
It include a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory networks in storm cluster referring again to Fig. 3 Lattice node, a plurality of output nodes and a preheating and more new node, and each processing node corresponding transmitting node, one A memory grid node and an output node;
Before data transformation operations, the second data to be converted are stored in message queue in the form of data flow in order one by one In, it is deposited in message queue, waits the data in transmitting node transmitting queue;Each processing node and a memory grid node It is bound, is shared to realize with block memory headroom, while starting, close and restarting;Preheating receives load with more new node After matched data request, all processing node is called to obtain associated data source (such as structured document, relevant database, non- Relevant database or message queue etc.) in the first data to be matched be deposited into the memory grid node with oneself binding, And with HashMap<key, value>form storage;
When carrying out conversion operation, transmitting node is read from message queue needs matched second data, one by one with data The form of stream is emitted to processing node, after the second data flow into processing node, handles node from the memory network bound with itself According to HashMap<key of storage in lattice, value>formatted data is matched, and obtains value result according to predetermined The business of different demands is completed the operation such as to convert or abandon, and the data for matching completion flow into corresponding output node one by one and carry out down The processing of one step.
When occur matched data variation need to update when, pause storm topology, transmitting node stop transmitting data, wait After data update, transmitting node is read from message queue again needs matched data to give processing node, preheating and more new node Updated first data deposit memory grid node is obtained again, restarts service topology, completes online updating operation.
Associated data is stored in memory grid section in advance by binding storm and memory grid node by the present invention In point, node to be processed is waited to call directly, reduced because reducing Data Matching operation from the external time for obtaining associated data and wasting Delay, not will cause data overstock, improve data-handling efficiency;The high fault tolerance for utilizing storm simultaneously, guarantees storm collection The high availability of group and memory grid cluster;Furthermore it is possible to matched data will be needed to be stored in memory grid, so that not In the case of traffic affecting, online updating is carried out by preheating and more new node.
Although specific embodiments of the present invention have been described above, those familiar with the art should be managed Solution, we are merely exemplary described specific embodiment, rather than for the restriction to the scope of the present invention, it is familiar with this The technical staff in field should be covered of the invention according to modification and variation equivalent made by spirit of the invention In scope of the claimed protection.

Claims (8)

1. a kind of method for realizing data conversion based on strom and memory grid, it is characterised in that: the plural number of storm need to be provided A transmitting node, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and each processing The corresponding transmitting node of node and a memory grid node, described method includes following steps:
Step 10, acquisition data, and collected data are entered into message queue in the form of data flow in order one by one;
Step 20 binds each processing node and a memory grid node, processing node and memory network after binding Lattice nodes sharing memory headroom, and realize synchronous averaging and close synchronously;
Step 30, before carrying out conversion operation, it is described preheating with more new node receive load request of data after, call all processing Node obtains the first data to be matched from associated data source, and first data are stored in the memory grid node respectively bound In;
Step 40 reads the second data to be matched by the transmitting node from message queue, and in the form of data flow It is emitted to processing node one by one;
It is got in step 50, the memory grid node bound by the second data that the processing node will receive with itself The first data matched, and the data that matching is completed are sent to corresponding output node and are further processed;
The step 50 specifically: by it is described processing node receive transmitting node transmitting come the second data after, one by one into Row processing obtains HashMap<key of storage, value>format from the memory grid node of the processing node binding Data are matched, and matched data are sent to corresponding output node one by one and are further processed.
2. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist In: the method also includes step 60, when the associated data source change need to update when, pause storm topology, transmitting Node stops transmitting data, after waiting pending datas to update, then executes step 30, and updated first data are stored in memory grid section After point, service topology is restarted, completes online updating operation.
3. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist In: " in the memory grid node for respectively binding first data deposit " in the step 30 is specifically, each described The first data that will acquire of processing node are stored in the memory grid node of itself binding, and with HashMap<key, and value> Form stored.
4. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist In: the associated data source includes structured document, relevant database, non-relational database or message queue.
5. a kind of system for realizing data conversion based on strom and memory grid, it is characterised in that: the plural number of storm need to be provided A transmitting node, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and each processing The corresponding transmitting node of node and a memory grid node, the system comprises data acquisition module, node binding module, Data acquisition module, data transmission module and data conversion module:
The data acquisition module is recorded in the form of data flow in order one by one for acquiring data, and by collected data Enter into message queue;
The node binding module, for each processing node and a memory grid node to be bound, the place after binding Node and memory grid nodes sharing memory headroom are managed, and realizes synchronous averaging and close synchronously;
The data acquisition module, for before carrying out conversion operation, the preheating to receive load request of data with more new node Afterwards, it calls all processing nodes to obtain the first data to be matched from associated data source, first data deposit is respectively tied up In fixed memory grid node;
The data transmission module, for the second data to be matched to be read from message queue by the transmitting node, and It is emitted to processing node one by one in the form of data flow;
The data conversion module, for by second data that will be received from data transmission module of processing node with from The first data for getting are matched in the memory grid node of body binding, and will data that matching is completed be sent to it is corresponding Output node is further processed;
The data conversion module specifically: for by it is described processing node receive transmitting node transmitting come the second data Afterwards, it is handled one by one, HashMap < key of storage is obtained from the memory grid node of the processing node binding, Value > format data are matched, and matched data are sent to corresponding output node one by one and are carried out at next step Reason.
6. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist In: the system also includes an online updating module, for the associated data source change need to update when, pause Storm topology, transmitting node stop transmitting data and then execute data acquisition module after waiting pending datas to update, by updated the After one data are stored in memory grid node, service topology is restarted, completes online updating operation.
7. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist In: " in the memory grid node for respectively binding first data deposit " in the data acquisition module is specifically, every One first data that will acquire of processing node are stored in the memory grid node of itself binding, and with HashMap < key, Value > form stored.
8. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist In: the associated data source includes structured document, relevant database, non-relational database or message queue.
CN201610599068.5A 2016-07-27 2016-07-27 A kind of method and system for realizing data conversion based on strom and memory grid Active CN106293960B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610599068.5A CN106293960B (en) 2016-07-27 2016-07-27 A kind of method and system for realizing data conversion based on strom and memory grid

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610599068.5A CN106293960B (en) 2016-07-27 2016-07-27 A kind of method and system for realizing data conversion based on strom and memory grid

Publications (2)

Publication Number Publication Date
CN106293960A CN106293960A (en) 2017-01-04
CN106293960B true CN106293960B (en) 2019-10-25

Family

ID=57652542

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610599068.5A Active CN106293960B (en) 2016-07-27 2016-07-27 A kind of method and system for realizing data conversion based on strom and memory grid

Country Status (1)

Country Link
CN (1) CN106293960B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106897131B (en) * 2017-02-22 2020-05-29 浪潮(北京)电子信息产业有限公司 Parallel computing method and device for astronomical software Griding
CN111931066B (en) * 2020-09-11 2021-09-07 四川新网银行股份有限公司 Real-time recommendation system design method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103201740A (en) * 2010-11-15 2013-07-10 萨思学会有限公司 Grid computing system alongside a distributed database architecture
CN103473318A (en) * 2013-09-12 2013-12-25 中国科学院软件研究所 Distributed transaction security method for memory data grid
CN103488699A (en) * 2013-09-04 2014-01-01 用友软件股份有限公司 Data processing device and method based on stored data grids

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2416878B (en) * 2004-08-06 2008-05-14 Univ Surrey Resource management in grid computing

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103201740A (en) * 2010-11-15 2013-07-10 萨思学会有限公司 Grid computing system alongside a distributed database architecture
CN103488699A (en) * 2013-09-04 2014-01-01 用友软件股份有限公司 Data processing device and method based on stored data grids
CN103473318A (en) * 2013-09-12 2013-12-25 中国科学院软件研究所 Distributed transaction security method for memory data grid

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
内存网格调度方法分析;卢俊文 等;《厦门理工学院学报》;20150630;第23卷(第3期);第75-79页 *

Also Published As

Publication number Publication date
CN106293960A (en) 2017-01-04

Similar Documents

Publication Publication Date Title
CN100531055C (en) Data synchronous system and its method
CN112148455B (en) Task processing method, device and medium
CN105190543A (en) Reachability-based coordination for cyclic dataflow
CN109450936A (en) A kind of adaptation method and device of the hetero-com-munication agreement based on Kafka
CN102346775A (en) Method for synchronizing multiple heterogeneous source databases based on log
CN105574643A (en) Real-time data center and big data platform fusion method for power grid
CN106293960B (en) A kind of method and system for realizing data conversion based on strom and memory grid
CN111445236A (en) Block chain transaction verification method, equipment and storage medium
CN106325984A (en) Big data task scheduling apparatus
CN104618433A (en) ECA (Event-Condition-Action) rule based service combination method
CN109347884A (en) A kind of real-time ethernet is to the conversion method and device of fieldbus, storage medium
CN106375480A (en) Electric energy data real-time acquisition system and method based on distributed system
CN104683472A (en) Data transmission method capable of supporting large data volume
CN106027534A (en) System for implementing financial message processing based on Netty
WO2012152110A1 (en) Splitting rule generation method and device for clearing and settlement subsystem
CN110226159A (en) Best-effort traffic library facility
Liu et al. Online job scheduling for distributed machine learning in optical circuit switch networks
CN107451211B (en) A kind of download system based on RabbitMQ and MongoDB
CN112688802B (en) High-performance exchange middleware based on API gateway
CN102710749A (en) Message queue control method and server
CN107169100A (en) A kind of electric automobile long-distance controlling service system and method
CN102629352B (en) Towards compound event method for mode matching and the system of real-time perception environment
CN106354493A (en) Development mode implementation method for solving traditional software development pain points
CN101504613B (en) Distributed dynamic binary translation method used for limited system
CN103532758B (en) Be applicable to the configuration processing method of transmission of future generation, data equipment fusion

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: Wufeng Street software Avenue Gulou District of Fuzhou city in Fujian province 350000 No. 89 Building No. 12

Applicant after: China Electric fufu Mdt InfoTech Ltd

Address before: Wufeng Street software Avenue Gulou District of Fuzhou city in Fujian province 350000 No. 89 Building No. 12

Applicant before: Fujian Fushitong Information Software Co.,Ltd.

GR01 Patent grant
GR01 Patent grant