CN109918216A - A kind of data processing method and system based on pipeline - Google Patents
A kind of data processing method and system based on pipeline Download PDFInfo
- Publication number
- CN109918216A CN109918216A CN201910171727.9A CN201910171727A CN109918216A CN 109918216 A CN109918216 A CN 109918216A CN 201910171727 A CN201910171727 A CN 201910171727A CN 109918216 A CN109918216 A CN 109918216A
- Authority
- CN
- China
- Prior art keywords
- data
- middleware
- pipe
- pipeline
- data pipe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims abstract description 49
- 238000000926 separation method Methods 0.000 claims abstract description 4
- 238000012360 testing method Methods 0.000 claims description 8
- 238000012986 modification Methods 0.000 abstract description 4
- 230000004048 modification Effects 0.000 abstract description 4
- 238000004883 computer application Methods 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
Landscapes
- Stored Programmes (AREA)
Abstract
The invention discloses a kind of data processing method and system based on pipeline, belong to computer application and technical field of data processing, the Logic Separation of data processing is the middleware in data pipe by this method, data to be processed flow into data pipe, pass through the processing of each middleware respectively, final outflow data processed result, realizes the flexibility and changeability of data process method and the scalability of custom logic.A kind of data processing system based on pipeline, including the data pipe being composed of middleware, the middleware is data process method, and multiple middlewares are arranged in order in data pipe；Pending data is flowed into data pipe, passes through the processing of each middleware respectively, finally flows out data processed result.The present invention can carry out specially treated to data under the premise of it is not necessary to modify data processing method, realize that the procedure processing of data reduces the difficulty and cost of modification data process method to improve the flexibility of program processing data.
Description
Technical field
The present invention relates to computer application and technical field of data processing, at specifically a kind of data based on pipeline
Manage method and system.
Background technique
It is frequently encountered in program development process and inevitable problem is exactly data processing, traditional data processing side
Formula is to write a method or realize a class, and data to be processed are passed to this method or class in the form of parameter
In some method, the processing result of method is exactly the processing result to the data, and there are one for the method for this data processing
A disadvantage needs to rewrite data processing method, to increase exactly when needing to add specially treated to data in a certain situation
The big difficulty and cost of data processing.
Summary of the invention
Technical assignment of the invention is place against the above deficiency, provides a kind of data processing method based on pipeline and is
System can carry out specially treated to data under the premise of it is not necessary to modify data processing method, realize the procedure processing of data, from
And improve the flexibility of program processing data.
The technical solution adopted by the present invention to solve the technical problems is:
The Logic Separation of data processing is one in data pipe by a kind of data processing method based on pipeline, this method
A sequential middleware, data to be processed flow into data pipe, pass through the processing of each middleware respectively, finally flow out number
According to processing result, the flexibility and changeability of data process method and the scalability of custom logic are realized.
Specifically, middleware different in the data pipe is responsible for handling part different in data, at data
Reason needs to adjust sequence of the middleware in data pipe, so that the business for changing entire data pipe processing data is patrolled
Volume, it is not necessary to modify the realization programs of other middlewares, realize the flexibility and changeability of data process method.
Specifically, adding customized middleware in data pipe, data pipe is made to realize the specially treated to data, and
Customized middleware does not influence the processing logic of other middlewares in data pipe, realizes the scalability of custom logic.
Preferably, the data pipe is composed of middleware, data pipe include conduit running main program module and
Part module among data processing.Conduit running main program module for realizing data pipe creation, management and operation, in data
Between part module be used for according to data processing needs, adjustment processing logic realizes special data processing.
The specific embodiment of this method is as follows:
1) data pipe creation module, is write；
2) data pipe, is created, and the addition test middleware into data pipe；
3), start test program, verifying pipeline data processing.
Specifically, writing data pipe creation module and including:
1.1) data medium, is created；
1.2), request to create is entrusted；
1.3) middleware interface, is created；
1.4), creation pipeline realizes class.
Preferably, the mode of the creation data medium is creation MyContext class, carries pending data.
public class MyContext
{
public Object Value{get；set；}
}
Further, data pipe is created, and the addition test middleware into data pipe, i.e. instantiation data pipeline are real
Example, the implementation to data pipe addition middleware are as follows:
2.1) middleware class, is created；
2.2) it, creates pipeline and adds middleware.
The invention also discloses a kind of data processing systems based on pipeline, including the data pipe being composed of middleware
Road, the middleware are data process method, and multiple middlewares are arranged in order in data pipe；Pending data is flowed into
Data pipe passes through the processing of each middleware respectively, finally flows out data processed result；
Customized centre is inserted into corresponding position by adjusting the position of middleware in data pipe, or in data pipe
Part realizes the specially treated to data.It is not necessary to modify the realization program of other middlewares, other middleware logarithms will not be influenced
According to processing logic, thus reduce modification data process method difficulty and cost.
Further, the data pipe includes conduit running main program module and data processing middleware module,
Conduit running main program module is created for pipeline, realizes data medium, request commission, middleware interface and pipe
The operation of road realization class；
Part module is used for the creation and addition of middleware class among data processing, realizes the data processing of data pipe.
The specific implementation of the system is as follows:
1, data pipe creation module is write；
1.1, data medium is created,
1.2, request to create is entrusted,
1.3, middleware interface is created,
1.4, creation pipeline realizes class.
2, data pipe is created, and the addition test middleware into data pipe；
Pipeline example is instantiated, and adds the method for middleware into pipeline are as follows:
2.1, middleware class is created,
2.2, it creates pipeline and adds middleware.
3, start test program, verifying pipeline data processing.
Compared to the prior art a kind of data processing method and system based on pipeline of the invention, has below beneficial to effect
Fruit:
This method solve needing to rewrite data processing method when adding specially treated to data, so that it is special to increase data
The problem of difficulty and cost of processing, when to carry out specially treated to data, need to only adjust pipe by the design of middleware
The position of middleware in road, or customized middleware is inserted into corresponding position in the duct, without modifying other middlewares
It realizes program, other middlewares will not be influenced to the processing logic of data, to reduce the difficulty of modification data process method
Degree and cost.
Detailed description of the invention
Fig. 1 is startup program in embodiment, the screenshot of data pipe processing result.
Specific embodiment
The Logic Separation of data processing is one in data pipe by a kind of data processing method based on pipeline, this method
A sequential middleware, data to be processed flow into data pipe, pass through the processing of each middleware respectively, finally reserve number
According to processing result, the flexibility and changeability of data process method and the scalability of custom logic are realized.
Different middlewares is responsible for handling part different in data in data pipe, needs to adjust according to data processing
Sequence of the middleware in data pipe, thus change the service logic of entire data pipe processing data, it is not necessary to modify
The realization program of other middlewares, realizes the flexibility and changeability of data process method.
Customized middleware is added in data pipe, so that data pipe is realized the specially treated to data, and customized
Middleware does not influence the processing logic of other middlewares in data pipe, realizes the scalability of custom logic.
The data pipe is composed of middleware, and data pipe includes conduit running main program module and data processing
Middleware module.Creation, management and operation of the conduit running main program module for realizing data pipe, data middleware module
For according to data processing needs, adjustment processing logic to realize special data processing.
The specific embodiment of this method is as follows:
1, data pipe creation module is write；
2, data pipe is created, and the addition test middleware into data pipe；
3, start test program, verifying pipeline data processing.
Wherein, writing data pipe creation module includes:
1.1, data medium is created；
MyContext class is created, pending data is carried.
1.2, request to create is entrusted；
public delegate Task RequestDelegate(MyContext context)
1.3, middleware interface is created；
1.4, creation pipeline realizes class.
2, data pipe is created, and addition test middleware, i.e. instantiation data pipeline example into data pipe, to number
Implementation according to pipeline addition middleware is as follows:
2.1, middleware class is created；
2.2, it creates pipeline and adds middleware.
SingleChannel pip=new SingleChannel ()；
pip.Use(new MyMiddleware())；
pip.Use(new MyMiddleware1())；
MyContext mycontext=new MyContext ()；
Task task=pip.Build () (mycontext)；
task.Wait()；
Thread.Sleep(3000)；
Console.WriteLine (" data processed result "+mycontext.Value)；
Console.ReadKey()；
3, startup program, data pipe processing result are as shown in Figure 1.
As another embodiment of the presently claimed invention, a kind of data processing system based on pipeline, including by middleware group
Data pipe made of conjunction, the middleware are data process method, and multiple middlewares are arranged in order in data pipe；It will
Pending data flows into data pipe, passes through the processing of each middleware respectively, finally flows out data processed result.
Customized centre is inserted into corresponding position by adjusting the position of middleware in data pipe, or in data pipe
Part realizes the specially treated to data.It is not necessary to modify the realization program of other middlewares, other middleware logarithms will not be influenced
According to processing logic, thus reduce modification data process method difficulty and cost.
In the data processing system, data pipe includes conduit running main program module and data processing middleware module,
Conduit running main program module is created for pipeline, realizes that data medium, request commission, middleware interface and pipeline realize class
Operation；Part module is used for the creation and addition of middleware class among data processing, realizes the data processing of data pipe.
The implementation of the system is as follows:
1, data pipe creation module is write, comprising:
1.1, data medium is created；
MyContext class is created, pending data is carried.
1.2, request to create is entrusted；
public delegate Task RequestDelegate(MyContext context)
1.3, middleware interface is created；
1.4, creation pipeline realizes class.
2, data pipe is created, and the addition test middleware into data pipe；
Instantiation data pipeline example, the implementation to data pipe addition middleware are as follows:
2.1, middleware class is created；
2.2, it creates pipeline and adds middleware.
SingleChannel pip=new SingleChannel ()；
pip.Use(new MyMiddleware())；
pip.Use(new MyMiddleware1())；
MyContext mycontext=new MyContext ()；
Task task=pip.Build () (mycontext)；
task.Wait()；
Thread.Sleep(3000)；
Console.WriteLine (" data processed result "+mycontext.Value)；
Console.ReadKey()；
3, start test program, verifying pipeline data processing.
The technical personnel in the technical field can readily realize the present invention with the above specific embodiments,.But it answers
Work as understanding, the present invention is not limited to above-mentioned specific embodiments.On the basis of the disclosed embodiments, the technical field
Technical staff can arbitrarily combine different technical features, to realize different technical solutions.
Except for the technical features described in the specification, it all is technically known to those skilled in the art.
Claims (10)
1. a kind of data processing method based on pipeline, it is characterised in that the Logic Separation of data processing is data pipe by this method
Middleware in road, data to be processed flow into data pipe, pass through the processing of each middleware respectively, final to flow out at data
Reason is as a result, realize the flexibility and changeability of data process method and the scalability of custom logic.
2. a kind of data processing method based on pipeline according to claim 1, it is characterised in that in the data pipe
Different middlewares is responsible for handling part different in data, needs to adjust the middleware in data pipe according to data processing
Sequence in road realizes the flexibility and changeability of data process method to change the service logic of entire data pipe processing data
Property.
3. a kind of data processing method based on pipeline according to claim 1 or 2, it is characterised in that in data pipe
Customized middleware is added, so that data pipe is realized the specially treated to data, and customized middleware does not influence data pipe
In other middlewares processing logic, realize the scalability of custom logic.
4. a kind of data processing method based on pipeline according to claim 1, it is characterised in that the data pipe by
Middleware is composed, and data pipe includes conduit running main program module and data processing middleware module.
5. a kind of data processing method based on pipeline according to claim 1, it is characterised in that the specific reality of this method
It is as follows to apply mode:
1) data pipe creation module, is write；
2) data pipe, is created, and the addition test middleware into data pipe；
3), start test program, verifying pipeline data processing.
6. a kind of data processing method based on pipeline according to claim 5, it is characterised in that write data pipe wound
Modeling block includes:
1.1) data medium, is created；
1.2), request to create is entrusted；
1.3) middleware interface, is created；
1.4), creation pipeline realizes class.
7. a kind of data processing method based on pipeline according to claim 6, it is characterised in that the creation data carry
The mode of body is creation MyContext class, carries pending data.
8. a kind of data processing method based on pipeline according to claim 5, it is characterised in that creation data pipe, and
Addition test middleware, i.e. instantiation data pipeline example into data pipe, to the realization side of data pipe addition middleware
Formula is as follows:
2.1) middleware class, is created；
2.2) it, creates pipeline and adds middleware.
9. a kind of data processing system based on pipeline, it is characterised in that including the data pipe being composed of middleware, institute
Stating middleware is data process method, and multiple middlewares are arranged in order in data pipe；Pending data is flowed into data
Pipeline passes through the processing of each middleware respectively, finally flows out data processed result；
It is real that customized middleware is inserted into corresponding position by adjusting the position of middleware in data pipe, or in data pipe
Now to the specially treated of data.
10. a kind of data processing system based on pipeline according to claim 9, it is characterised in that the data pipe packet
Conduit running main program module and data processing middleware module are included,
Conduit running main program module is created for pipeline, realizes that data medium, request commission, middleware interface and pipeline are real
The operation of existing class；
Part module is used for the creation and addition of middleware class among data processing, realizes the data processing of data pipe.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910171727.9A CN109918216A (en) | 2019-03-07 | 2019-03-07 | A kind of data processing method and system based on pipeline |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910171727.9A CN109918216A (en) | 2019-03-07 | 2019-03-07 | A kind of data processing method and system based on pipeline |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109918216A true CN109918216A (en) | 2019-06-21 |
Family
ID=66963814
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910171727.9A Pending CN109918216A (en) | 2019-03-07 | 2019-03-07 | A kind of data processing method and system based on pipeline |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109918216A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928578A (en) * | 2019-12-05 | 2020-03-27 | 潍柴动力股份有限公司 | Data processing method, device and system |
| CN111142925A (en) * | 2019-12-23 | 2020-05-12 | 山东浪潮通软信息科技有限公司 | Pipeline type data processing method, equipment and storage medium |
| CN111932234A (en) * | 2020-09-24 | 2020-11-13 | 国网电子商务有限公司 | Transaction data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783019A (en) * | 2004-12-03 | 2006-06-07 | 微软公司 | Interface infrastructure for creating and interacting with web services |
| US20110258414A1 (en) * | 2008-12-12 | 2011-10-20 | Bae Systems Plc | Apparatus and method for processing data streams |
| CN103324479A (en) * | 2013-06-13 | 2013-09-25 | 南京南自信息技术有限公司 | Distributed big-data computation middleware system framework in loose environment |
| US20140297620A1 (en) * | 2013-03-29 | 2014-10-02 | International Business Machines Corporation | Analytics based on pipes programming model |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
-
2019
- 2019-03-07 CN CN201910171727.9A patent/CN109918216A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783019A (en) * | 2004-12-03 | 2006-06-07 | 微软公司 | Interface infrastructure for creating and interacting with web services |
| US20110258414A1 (en) * | 2008-12-12 | 2011-10-20 | Bae Systems Plc | Apparatus and method for processing data streams |
| US20140297620A1 (en) * | 2013-03-29 | 2014-10-02 | International Business Machines Corporation | Analytics based on pipes programming model |
| CN103324479A (en) * | 2013-06-13 | 2013-09-25 | 南京南自信息技术有限公司 | Distributed big-data computation middleware system framework in loose environment |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 雨夜朦胧: "ASP.NET Core运行原理解剖：Middleware请求管道的构成", 《HTTPS://WWW.CNBLOGS.COM/RAININGNIGHT/P/M/MIDDLEWARE-IN-ASP-NET-COER.HTML》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928578A (en) * | 2019-12-05 | 2020-03-27 | 潍柴动力股份有限公司 | Data processing method, device and system |
| CN111142925A (en) * | 2019-12-23 | 2020-05-12 | 山东浪潮通软信息科技有限公司 | Pipeline type data processing method, equipment and storage medium |
| CN111932234A (en) * | 2020-09-24 | 2020-11-13 | 国网电子商务有限公司 | Transaction data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109918216A (en) | A kind of data processing method and system based on pipeline | |
| US9164807B2 (en) | Statically allocating set of tasks to particular cluster and dynamically allocating each task to different processing units within cluster | |
| CN104714849B (en) | System and method for realizing optimum performance in synthetic workload environment | |
| CN103761139B (en) | General purpose computation virtualization implementation method based on dynamic library interception | |
| CN107357661A (en) | A kind of fine granularity GPU resource management method for mixed load | |
| CN109154924A (en) | Multiple uplink side jaws equipments | |
| CN102955737B (en) | The program debugging method of heterogeneous processor system and system | |
| CN103605576A (en) | Multithreading-based MapReduce execution system | |
| KR101516055B1 (en) | Map-reduce workflow processing device and method, and storage media storing the same | |
| CN102203718B (en) | Memory transfer processing method, device and system thereof | |
| CN107292741A (en) | A kind of resource allocation methods and device | |
| CN104583896A (en) | Distributing power to heterogenous compute elements of a processor | |
| CN104038444B (en) | A kind of method of resource allocation, equipment and system | |
| CN106407231A (en) | A data multi-thread export method and system | |
| CN103823709B (en) | System of virtual cluster, resource allocation methods and management node | |
| CN101216780B (en) | Method and apparatus for accomplishing multi-instance and thread communication under SMP system | |
| CN109902038A (en) | A kind of PCIe bus address space distribution method and device | |
| DE102020106002A1 (en) | DYNAMIC LOAD BALANCING OF COMPUTER SYSTEMS UNDER DIFFERENT COMPUTING CONTEXTS | |
| CN111163149B (en) | Intelligent contract platform method based on block chain | |
| CN104731572B (en) | Fulfil the method and system of the hardware right of hardware thread | |
| WO2022134538A1 (en) | Micro-operating system bytecode reduced instruction set and resource restriction apparatus | |
| CN103440159B (en) | Method and system for scheduling processes | |
| CN102521043B (en) | A kind of task processing method and application system | |
| CN105988952B (en) | The method and apparatus for distributing hardware-accelerated instruction for Memory Controller Hub | |
| CN102497410A (en) | Method for dynamically partitioning computing resources of cloud computing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190621 |