CN109412970A - Stream compression system, stream compression method, electronic equipment and storage medium - Google Patents
Stream compression system, stream compression method, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109412970A CN109412970A CN201811158015.5A CN201811158015A CN109412970A CN 109412970 A CN109412970 A CN 109412970A CN 201811158015 A CN201811158015 A CN 201811158015A CN 109412970 A CN109412970 A CN 109412970A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- layer
- interface
- processing module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2408—Traffic characterised by specific attributes, e.g. priority or QoS for supporting different services, e.g. a differentiated services [DiffServ] type of service
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Abstract
The present invention provides a kind of stream compression system, stream compression method, electronic equipment and storage medium, and wherein system includes: expansible data processing module, acquisition and crosses filter data, and exports filtered data；Configurable scheduler module is scheduled the data of each layer of data processing module；Configurable flow control module is managed the data traffic of each layer of data processing module；Interface layer provides the expansion interface at least for data processing module extension；And configuration module, the configuration interface for being at least scheduled for module and the configuration of flow control module is provided.The present invention is extended data processing module by expansion interface, facilitates docking different data sources；Scheduler module and flow control module are configured by configuring interface, realize the dynamic management to stream compression；By interface communication between modules, independently of each other, the circulation processing of mass data is realized flexibly and efficiently.
Description
Technical field
The present invention relates to technical field of data processing, specifically, being related to a kind of stream compression system, stream compression side
Method, electronic equipment and storage medium.
Background technique
Large scale system always has mass data and needs to collect, filter and be sent to downstream clothes in distributed system
Business processing.The problem of how effectively collecting data and be efficiently conveyed to downstream, must be taken into consideration when being processing mass data.
Some more or less defects of existing stream compression tool.Still target is single for some excellent performances, such as mainly
It is to solve the synchronization between database data.Also some targets are extensive, but because of design reasons, it is sacrificial to performance
Domestic animal.
Such as typically, daily record data is often magnanimity in large scale system.For the stream compression that daily record data is common
Tool has Flume and Logstash.Flume provides log synchronous service and has done to performance very big in order to avoid data exception
It sacrifices；If it is desired to which improving performance just needs to start a large amount of Flume nodes, virtually increase O&M cost and management cost,
Also carry out new problem to the log storage tape of rear end.And Logstash is due to characteristic of speech sounds, when extension, JVM fine tuning all
Face various difficulties.
It should be noted that information is only used for reinforcing the reason to background of the invention disclosed in above-mentioned background technology part
Solution, therefore may include the information not constituted to the prior art known to persons of ordinary skill in the art.
Summary of the invention
In view of this, the present invention provides a kind of stream compression system, stream compression method, electronic equipment and storage medium,
It is poor to solve stream compression Tool extensibility in the prior art, it can not effective control node number, the scheduling in stream compression and pipe
Inflexible problem in reason.
According to an aspect of the present invention, a kind of stream compression system is provided, comprising: expansible data processing module,
The data processing module acquires and crosses filter data, and exports filtered data；Configurable scheduler module, at the data
Reason module is communicated by interface with the scheduler module, and the scheduler module carries out the data of each layer of the data processing module
Scheduling；Configurable flow control module, the data processing module are communicated by interface with the flow control module, the flow control module
The data traffic of each layer of the data processing module is managed；Interface layer provides and at least expands for the data processing module
The expansion interface of exhibition；And configuration module, the configuration interface at least configured for the scheduler module and flow control module is provided.
Preferably, in above-mentioned stream compression system, the data processing module includes: data collection layer, including one
Or multiple collectors, for acquiring data；Filter layer, including one or more filters are filtered the data of acquisition；Number
According to write-in layer, including one or more output interfaces, for exporting filtered data；Each layer of the data processing module it
Between pass through abstraction interface communicate.
Preferably, in above-mentioned stream compression system, the data processing module further include: one or more buffer layers,
Between the data collection layer and the filter layer, and/or, it is located between the filter layer and data write-in layer；
The asynchronous circulation of data is carried out between each layer of data processing module by the buffer layer.
Preferably, in above-mentioned stream compression system, the data collection layer has acquisition progress feedback interface, for extremely
It is few to submit data to acquire progress to the scheduler module and the flow control module.
Preferably, in above-mentioned stream compression system, the acquisition progress feedback interface includes synchronous feedback interface and different
Walk feedback interface.
Preferably, in above-mentioned stream compression system, the scheduler module is executed: one or more for dispatching the filter layer
A filter, control data flow separately through scheduled filter from the data collection layer；And the filtered data of control
From data write-in layer output.
Preferably, in above-mentioned stream compression system, the scheduler module also executes: to circulation with identical behavior
The unique mark ID of data addition.
Preferably, in above-mentioned stream compression system, data of the flow control module to each layer of the data processing module
It reads quantity and the reading data frequency is managed.
Preferably, above-mentioned stream compression system further include: configurable abnormality detection module, for monitoring the number of circulation
According to.
According to another aspect of the present invention, a kind of stream compression method is provided, the stream compression method is by above-mentioned
Stream compression system executes, comprising: the data processing module acquires and crosses filter data, and exports filtered data；It is described
Scheduler module is communicated by interface with the data processing module, is scheduled to the data of each layer of the data processing module；
The flow control module is communicated with the data processing module by interface, to the data traffic of each layer of the data processing module into
Row management；The data processing module passes through expansion interface extension process one or more data source；The scheduler module passes through
Configure the scheduling method of interface configuration data；And the flow control module is by configuring the management mould of interface configuration data flow
Formula.
According to an aspect of the present invention, a kind of electronic equipment is provided, comprising: processor；And memory, for storing
Executable instruction；Wherein, the processor is configured to execute above-mentioned stream compression side via the executable instruction is executed
The step of method.
According to an aspect of the present invention, a kind of computer readable storage medium is provided, computer program is stored thereon with,
The computer program realizes the step of above-mentioned stream compression method when being executed by processor.
The beneficial effect of the present invention compared with prior art is:
Data processing module can be extended by expansion interface, facilitate docking different data sources；Scheduler module can pass through
Configuration interface is configured, and is dynamically come into force, is facilitated integration and schedules system resources；Flow control module can be matched by configuring interface
It sets, dynamically comes into force, flow-control administration is carried out to data in real time data circulation；Whole system has considerable flexibility,
It is easy to use；
Also, data processing module, scheduler module, flow control module, by interface communication, all modules are mutually indepedent, can be with
Effective control node number enables whole system efficiently to circulate mass data.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
It can the limitation present invention.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows and meets implementation of the invention
Example, and be used to explain the principle of the present invention together with specification.It should be evident that the accompanying drawings in the following description is only the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 shows a kind of module diagram of stream compression system in the embodiment of the present invention；
Fig. 2 shows the configuration diagrams of stream compression system a kind of in the embodiment of the present invention；
Fig. 3 shows a kind of framework block schematic diagram of data processing module in the embodiment of the present invention；
Fig. 4 shows a kind of step schematic diagram of stream compression method in the embodiment of the present invention；
Fig. 5 shows the schematic diagram of a kind of electronic equipment in the embodiment of the present invention；
Fig. 6 shows a kind of schematic diagram of computer readable storage medium in the embodiment of the present invention.
Specific embodiment
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be with a variety of shapes
Formula is implemented, and is not understood as limited to embodiment set forth herein.On the contrary, thesing embodiments are provided so that the present invention will
Fully and completely, and by the design of example embodiment comprehensively it is communicated to those skilled in the art.It is identical attached in figure
Icon note indicates same or similar structure, thus will omit repetition thereof.
Stream compression system of the invention can efficiently circulate and handle mass data, and whole system has well flexibly
Property, it is highly convenient for user's use.Referring to figs. 1 and 2, in some embodiments of the invention, stream compression system 1 includes
But it is not limited to lower module:
Expansible data processing module 11, data processing module 11 acquires and crosses filter data, and exports filtered number
According to.
In a preferred embodiment, data processing module 11 further comprises:
Data collection layer 111, including one or more collectors, for acquiring data.As shown in Fig. 2, data collection layer
111 may include that (Kafka collector, Kafka are that a kind of distributed post subscription of high-throughput disappears to Kafka Collector
Breath system), Log File Collector (journal file collector) and other expansible collectors, can according to need into
Row extension, to dock different data sources.Each collector in data collection layer 111 acquires initial data, and cooperation frame connects
Mouthful, it is abstracted output initial data.Each collector in data collection layer 111 passes through interface and is communicated, each collector
It is independent mutually, it can according to need the specific collector operation of control.
Data collection layer 11 also has acquisition progress feedback interface, at least mentioning to scheduler module 12 and flow control module 13
Intersection number is according to acquisition progress.Preferably, the acquisition progress feedback interface of data collection layer 11 includes synchronous feedback interface, for synchronizing
Data acquisition progress and asynchronous feedback interface are submitted, acquires progress for asynchronous submission data.Realize that grasping data in real time adopts
The case where data for collecting each collector in layer 11 acquire progress, avoid Data duplication acquisition and loss of data.
Filter layer 112, including one or more filters, for being filtered to the data of acquisition.Filter layer 112 can be with
Including Filter (filter) 1121, Filter (filter) 1122, Filter (filter) 1123 and other expansible mistakes
Filter.Each filter in filter layer 112 passes through interface and is communicated, and each filter is independent mutually, can be according to need
Want layout and plug filter.
Layer 113, including one or more output interfaces is written in data, for exporting filtered data.Layer is written in data
113 may include DB (Database, database) interface, a kind of Elasticsearch (log analysis platform) interface and Ta Ke
The output interface of extension can according to need extension and be docked to different systems, realize the specific aim output of data.
Further, data processing module 11 may also include one or more buffer layers 114, be located at data collection layer 111
Between filter layer 112, and/or, it is located between filter layer 112 and data write-in layer 113, between each layer of data processing module 11
The asynchronous circulation of data is carried out by buffer layer 114.Specifically, if not configuring buffer layer 114, scheduler module 12 will
The stream compression between 11 upper and lower level of data processing module is controlled by synchronous mode；Buffer layer 114 can allow data to acquire
Layer 111 carries out the acquisition of data next time as early as possible, and the data that can also come in multiple circulation are packaged, processing next time
Device does not need to carry out lock competition as far as possible.
In buffer layer 114 include multiple buffering area, ring buffer as shown in Figure 2, blocking queue and its
His expansible buffer area can according to need and extend different buffer areas, realizes the caching process of data.
In a preferred embodiment, it is communicated between each layer of data processing module 11 by abstraction interface, so that at data
It manages between each layer of module 11 independently of each other, can operate independently from and be applied in other environment.
Configurable scheduler module 12, data processing module 11 are communicated by interface with scheduler module 12, scheduler module 12
The data of each layer of data processing module 11 are scheduled.
Specifically, in some embodiments, scheduler module 12 executes: first control one of data collection layer 111 or
Multiple collectors acquire data.Then the one or more filters of filter layer 112 are dispatched, control data flow separately through scheduled
Filter be filtered；Wherein scheduler module 12 can carry out layout to the filter of filter layer 112, and data can be with the shape of stream
Formula flows through all filters (as shown by the arrows in Figure 2, data flow to Filter 1123 from Filter 1121), filter it
Between can arbitrarily be plugged by interface, depending on filtering need depending on.Then, scheduler module 12 can control filtered data from number
According to one or more output interfaces output of write-in layer 113.
Scheduler module 12 can be with the resource of each layer of integral data processing module 11, due to passing through interface between each layer
Communication, can according to need and build different configuration of data processing module.Furthermore scheduler module 12 can also dispatch data processing
11 each layers of internal thread of module, allow one process to use system resource as much as possible.
In a preferred embodiment, scheduler module 12 can also carry out: add only to the data with identical behavior of circulation
One mark ID.It specifically, is that convection current turns data abstraction id field, identical ID indicates identical behavior, is written in this way in data
Layer 113 is easier to ensure that data idempotent.
Further, the configuration interface that scheduler module 12 can be provided by configuration module 15 is configured, such as by clear
Device of looking at is configured, to adapt to different stream compression dispatching requirements.
Configurable flow control module 13, data processing module 11 are communicated by interface with flow control module 13, flow control module 13
The data traffic of each layer of data processing module 11 is managed.
By taking daily record data circulation processing as an example, since data volume is big, possible excessive data will affect entire physical machine
Network interface card, or even in peak period may result in situations such as memory overflows since buffer layer 114 configures improper.Pass through flow control module
The parameters such as the reading data quantity of 13 pairs of data processing modules 11, the frequency, interval configure, and dynamically come into force, and realize number in real time
In the case of circulation, flow-control administration is carried out to data by flow control module 13.
The configuration interface that flow control module 13 can be provided by configuration module 15 is configured, and is such as matched by browser
It sets, manages demand for control to adapt to different data traffics.
Interface layer 14 provides the expansion interface at least extended for data processing module 11, realizes real time data circulation situation
Under, data processing module 11 can facilitate docking different data sources with flexible expansion.
Interface layer 14 is also provided for above-mentioned modules (including data processing module 11, scheduler module 12, flow control module 13
Deng) intercommunication abstraction interface.By being abstracted respectively to the main several nodes of stream compression, realization acquisition,
Filtering sends, scheduling, manages independently of each other.By realize abstraction interface so that the core of stream compression system need to only pay close attention to it is logical
With logic, and extension side pays close attention to application specific logic.Modules are mutually indepedent, and combination is as needed to adapt to different data streams
Turn demand.
Further, in a preferred embodiment, stream compression system 1 further include: configurable abnormality detection module 16,
For monitoring the data of circulation.Due to the diversity of data, may cause since data exception makes a part of filter be in deadlock
Death situation state or current processor data in EMS memory have excessively had little time the problems such as processing, can by abnormality detection module 16
With monitoring data, alarm is issued when noting abnormalities.The configuration interface that abnormality detection module 16 can also be provided by configuration module 15
It is configured, is such as configured by browser, monitor demand to adapt to different data exceptions.
Stream compression system in above-described embodiment is extended data processing module by expansion interface, facilitates docking
Different data sources；The management modules such as scheduler module, flow control module, abnormality detection module are configured by configuring interface, are moved
State comes into force, and convenient to be managed in real time data circulation to data, whole system has considerable flexibility, conveniently makes
With.By interface communication between all modules, independently of each other, can effective control node number, enable whole system efficiently
Circulate mass data.And due to acquisition, filter, send, scheduling, management etc. is mutually indepedent, can be for few when problem occurs
It measures module and carries out problem investigation, save computing resource, improve resource utilization.
Referring to shown in Fig. 3, in a specific application example, using the data processing module 11 of above-mentioned any embodiment
Daily record data is acquired, filter and is exported.
In the present embodiment data processing module 11 include data collection layer 111, filter layer 112, data write-in layer 113, with
And two buffer layers 114, it is located between data collection layer 111 and filter layer 112 and layer is written in filter layer 112 and data
Between 113.
Wherein, data collection layer 111 by Kafka collector to daily record data, including business diary, gateway log,
The daily record datas such as Metrics data, Agent log are acquired.
Filter layer 112 is filtered by daily record data of multiple filters to acquisition, including format filtering, business mistake
Filter, gateway-filter etc..
Data are written layer 113 and Elasticsearch are written in filtered daily record data by Elasticsearch interface
Cluster.
Buffer layer 114 realizes multithreading shared data by blocking queue.It is effectively reduced by bulk queue
To the request number of times of Elasticsearch, one process can handle a variety of stream compression demands.
By above-mentioned data processing module 11,30000 daily record datas of processing per second are may be implemented simultaneously in peak period single machine
Elasticsearch cluster is written.Even if the failure of some collector is restarted, daily record data can also be effectively ensured and do not repeat,
It does not lose.
The embodiment of the present invention also provides a kind of stream compression method, and stream compression method is retouched by above-mentioned any embodiment
The stream compression system stated executes.Referring to shown in Fig. 4, in some embodiments, stream compression method includes but is not limited to following
Step:
S10, data processing module acquire and cross filter data, and export filtered data.
S20, scheduler module are communicated by interface with data processing module, are adjusted to the data of each layer of data processing module
Degree.
S30, flow control module are communicated with data processing module by interface, to the data traffic of each layer of data processing module into
Row management.
S40, data processing module pass through expansion interface extension process one or more data source.
S50, scheduler module are by configuring the scheduling method of interface configuration data.
S60, flow control module are by configuring the management mode of interface configuration data flow.
Wherein, the label before step is only used for distinguishing different steps, is not intended to limit between each step and executes sequence.
Further, stream compression method further includes each mould of stream compression system described in above-mentioned any embodiment
The step of submodule of block executes, is not repeated to describe herein.
The embodiment of the present invention also provides a kind of electronic equipment, including processor and memory, and being stored in memory can hold
Row instruction, processor are configured as executing the step of the stream compression method in above-described embodiment via executable instruction is executed
Suddenly.
As described above, electronic equipment of the invention can be extended data processing module by expansion interface, it is convenient
Dock different data sources；Each management module is configured by configuring interface, realizes the dynamic management to stream compression；Institute
Have by interface communication between module, independently of each other, effectively control node number, saving computing resource can improve the utilization of resources
Rate.
Fig. 5 is the structural schematic diagram of electronic equipment in the embodiment of the present invention, it should be appreciated that Fig. 5 is only schematic
Ground shows modules, these modules can be virtual software module or actual hardware module, the merging of these modules,
Fractionation and its increase of complementary modul block are all within protection scope of the present invention.
Person of ordinary skill in the field it is understood that various aspects of the invention can be implemented as system, method or
Program product.Therefore, various aspects of the invention can be embodied in the following forms, it may be assumed that complete hardware embodiment, complete
The embodiment combined in terms of full Software Implementation (including firmware, microcode etc.) or hardware and software, can unite here
Referred to as " circuit ", " module " or " platform ".
The electronic equipment 400 of this embodiment according to the present invention is described referring to Fig. 5.The electronics that Fig. 4 is shown
Equipment 400 is only an example, should not function to the embodiment of the present invention and use scope bring any restrictions.
As shown in figure 4, electronic equipment 400 is showed in the form of universal computing device.The component of electronic equipment 400 can wrap
Include but be not limited to: at least one processing unit 410, at least one storage unit 420, connection different platform component (including storage
Unit 420 and processing unit 410) bus 430, display unit 440 etc..
Wherein, storage unit is stored with program code, and program code can be executed with unit 410 processed, so that processing is single
Member 410 executes the step of the illustrative embodiments various according to the present invention of this specification above-mentioned stream compression method part description
Suddenly.For example, processing unit 410 can execute step as shown in Figure 4 respectively.
Storage unit 420 may include the readable medium of volatile memory cell form, such as Random Access Storage Unit
(RAM) 4201 and/or cache memory unit 4202, it can further include read-only memory unit (ROM) 4203.
Storage unit 420 can also include program/utility with one group of (at least one) program module 4205
4204, such program module 4205 includes but is not limited to: operating system, one or more application program, other program moulds
It may include the realization of network environment in block and program data, each of these examples or certain combination.
Bus 430 can be to indicate one of a few class bus structures or a variety of, including storage unit bus or storage
Cell controller, peripheral bus, graphics acceleration port, processing unit use any bus structures in a variety of bus structures
Local bus.
Electronic equipment 400 can also be with one or more external equipments 500 (such as keyboard, sensing equipment, bluetooth equipment
Deng) communication, can also be enabled a user to one or more equipment interact with the electronic equipment 400 communicate, and/or with make
Any equipment (such as the router, modulation /demodulation that the electronic equipment 400 can be communicated with one or more of the other calculating equipment
Device etc.) communication.This communication can be carried out by input/output (I/O) interface 450.Also, electronic equipment 400 can be with
By network adapter 460 and one or more network (such as local area network (LAN), wide area network (WAN) and/or public network,
Such as internet) communication.Network adapter 460 can be communicated by bus 430 with other modules of electronic equipment 400.It should
Understand, although not shown in the drawings, other hardware and/or software module can be used in conjunction with electronic equipment 400, including but unlimited
In: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive and number
According to backup storage platform etc..
The embodiment of the present invention also provides a kind of computer readable storage medium, for storing computer program, computer journey
Sequence is performed the step of stream compression method for realizing above-described embodiment.In some possible embodiments, of the invention
Various aspects are also implemented as a kind of form of program product comprising program code, when program product on the terminal device
When operation, program code is for making terminal device execute the description of this specification above-mentioned stream compression method part according to the present invention
The step of various illustrative embodiments.
As described above, computer readable storage medium of the invention can carry out data processing module by expansion interface
Extension, facilitates docking different data sources；Each management module is configured by configuring interface, stream compression is moved in realization
State management；By interface communication between all modules, independently of each other, can effectively control node number, saving computing resource improve
Resource utilization.
Fig. 6 is the structural schematic diagram of computer readable storage medium of the invention.Refering to what is shown in Fig. 6, describing according to this
The program product 600 for realizing the above method of the embodiment of invention can use the read-only storage of portable compact disc
Device (CD-ROM) and including program code, and can be run on terminal device, such as PC.However, journey of the invention
Sequence product is without being limited thereto, and in this document, readable storage medium storing program for executing can be any tangible medium for including or store program, the journey
Sequence can be commanded execution system, device or device use or in connection.
Program product can be using any combination of one or more readable mediums.Readable medium can be readable signal Jie
Matter or readable storage medium storing program for executing.Readable storage medium storing program for executing for example can be but be not limited to electricity, magnetic, optical, electromagnetic, infrared ray or partly lead
System, device or the device of body, or any above combination.More specific example (the non exhaustive column of readable storage medium storing program for executing
Table) it include: the electrical connection with one or more conducting wires, portable disc, hard disk, random access memory (RAM), read-only storage
Device (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read only memory (CD-
ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.
Computer readable storage medium may include in a base band or as carrier wave a part propagate data-signal,
In carry readable program code.The data-signal of this propagation can take various forms, including but not limited to electromagnetic signal,
Optical signal or above-mentioned any appropriate combination.Readable storage medium storing program for executing can also be any readable Jie other than readable storage medium storing program for executing
Matter, the readable medium can send, propagate or transmit for by instruction execution system, device or device use or and its
The program of combined use.The program code for including on readable storage medium storing program for executing can transmit with any suitable medium, including but not
It is limited to wireless, wired, optical cable, RF etc. or above-mentioned any appropriate combination.
The program for executing operation of the present invention can be write with any combination of one or more programming languages
Code, programming language include object oriented program language-Java, C++ etc., further include conventional process
Formula programming language-such as " C " language or similar programming language.Program code can be calculated fully in user
It executes in equipment, partly execute on a user device, executing, as an independent software package partially in user calculating equipment
Upper part executes on a remote computing or executes in remote computing device or server completely.It is being related to remotely counting
In the situation for calculating equipment, remote computing device can pass through the network of any kind, including local area network (LAN) or wide area network
(WAN) it is connected to user calculating equipment, or, it may be connected to external computing device (such as utilize ISP
To be connected by internet).
The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be said that
Specific implementation of the invention is only limited to these instructions.For those of ordinary skill in the art to which the present invention belongs, exist
Under the premise of not departing from present inventive concept, a number of simple deductions or replacements can also be made, all shall be regarded as belonging to of the invention
Protection scope.
Claims (12)
1. a kind of stream compression system characterized by comprising
Expansible data processing module, the data processing module acquires and crosses filter data, and exports filtered data；
Configurable scheduler module, the data processing module are communicated by interface with the scheduler module, the scheduler module
The data of each layer of the data processing module are scheduled；
Configurable flow control module, the data processing module are communicated by interface with the flow control module, the flow control module
The data traffic of each layer of the data processing module is managed；
Interface layer provides the expansion interface at least for data processing module extension；And
Configuration module provides the configuration interface at least configured for the scheduler module and flow control module.
2. stream compression system as described in claim 1, which is characterized in that the data processing module includes:
Data collection layer, including one or more collectors, for acquiring data；
Filter layer, including one or more filters are filtered the data of acquisition；
Layer, including one or more output interfaces is written in data, for exporting filtered data；
It is communicated between each layer of the data processing module by abstraction interface.
3. stream compression system as claimed in claim 2, which is characterized in that the data processing module further include:
One or more buffer layers, between the data collection layer and the filter layer, and/or, it is located at the filter layer
Between data write-in layer；
The asynchronous circulation of data is carried out between each layer of data processing module by the buffer layer.
4. stream compression system as claimed in claim 2, which is characterized in that the data collection layer is fed back with acquisition progress
Interface, at least submitting data to acquire progress to the scheduler module and the flow control module.
5. stream compression system as claimed in claim 4, which is characterized in that the acquisition progress feedback interface includes synchronizing instead
Present interface and asynchronous feedback interface.
6. stream compression system as claimed in claim 2, which is characterized in that the scheduler module executes:
The one or more filters of the filter layer are dispatched, control data flow separately through scheduled from the data collection layer
Filter；And
Filtered data are controlled from data write-in layer output.
7. stream compression system as claimed in claim 6, which is characterized in that the scheduler module also executes:
To the unique mark ID of data addition with identical behavior of circulation.
8. stream compression system as described in claim 1, which is characterized in that the flow control module is to the data processing module
The reading data quantity and the reading data frequency of each layer are managed.
9. stream compression system as described in claim 1, which is characterized in that further include:
Configurable abnormality detection module, for monitoring the data of circulation.
10. a kind of stream compression method, which is characterized in that the stream compression method is described in any item by claim 1-9
Stream compression system executes, comprising:
The data processing module acquires and crosses filter data, and exports filtered data；
The scheduler module is communicated with the data processing module by interface, to the data of each layer of the data processing module into
Row scheduling；
The flow control module is communicated by interface with the data processing module, to the data flow of each layer of the data processing module
Amount is managed；
The data processing module passes through expansion interface extension process one or more data source；
The scheduler module is by configuring the scheduling method of interface configuration data；And
The flow control module is by configuring the management mode of interface configuration data flow.
11. a kind of electronic equipment characterized by comprising
Processor；And
Memory, for storing executable instruction；
Wherein, the processor is configured to carry out stream compression described in perform claim requirement 10 via the execution executable instruction
The step of method.
12. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of stream compression method described in any one of claim 10 is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811158015.5A CN109412970B (en) | 2018-09-30 | 2018-09-30 | Data transfer system, data transfer method, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811158015.5A CN109412970B (en) | 2018-09-30 | 2018-09-30 | Data transfer system, data transfer method, electronic device, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109412970A true CN109412970A (en) | 2019-03-01 |
| CN109412970B CN109412970B (en) | 2022-07-05 |
Family
ID=65466667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811158015.5A Active CN109412970B (en) | 2018-09-30 | 2018-09-30 | Data transfer system, data transfer method, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109412970B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947837A (en) * | 2019-03-21 | 2019-06-28 | 上海富数科技有限公司 | Realize the system and method for intelligent scheduling data grabber function |
| CN110071873A (en) * | 2019-04-09 | 2019-07-30 | 华为技术有限公司 | A kind of method, apparatus and relevant device sending data |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6047323A (en) * | 1995-10-19 | 2000-04-04 | Hewlett-Packard Company | Creation and migration of distributed streams in clusters of networked computers |
| CN101567845A (en) * | 2009-04-27 | 2009-10-28 | 成都科来软件有限公司 | Real-time processing system and method of universal network data packet |
| CN102685801A (en) * | 2012-05-11 | 2012-09-19 | 中国联合网络通信集团有限公司 | Network management equipment and method |
| CN104090823A (en) * | 2014-06-09 | 2014-10-08 | 中国建设银行股份有限公司 | Flow control method and device for computer system |
| CN105978960A (en) * | 2016-05-06 | 2016-09-28 | 武汉烽火众智数字技术有限责任公司 | Cloud scheduling system and method based on mass video structured processing |
| CN106060149A (en) * | 2016-06-24 | 2016-10-26 | 北京交通大学 | Mobile internet mass data analysis and audit technical architecture |
| CN106354795A (en) * | 2016-08-26 | 2017-01-25 | 南威软件股份有限公司 | Distributed data exchanging system |
| CN107317724A (en) * | 2017-06-06 | 2017-11-03 | 中证信用增进股份有限公司 | Data collecting system and method based on cloud computing technology |
-
2018
- 2018-09-30 CN CN201811158015.5A patent/CN109412970B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6047323A (en) * | 1995-10-19 | 2000-04-04 | Hewlett-Packard Company | Creation and migration of distributed streams in clusters of networked computers |
| CN101567845A (en) * | 2009-04-27 | 2009-10-28 | 成都科来软件有限公司 | Real-time processing system and method of universal network data packet |
| CN102685801A (en) * | 2012-05-11 | 2012-09-19 | 中国联合网络通信集团有限公司 | Network management equipment and method |
| CN104090823A (en) * | 2014-06-09 | 2014-10-08 | 中国建设银行股份有限公司 | Flow control method and device for computer system |
| CN105978960A (en) * | 2016-05-06 | 2016-09-28 | 武汉烽火众智数字技术有限责任公司 | Cloud scheduling system and method based on mass video structured processing |
| CN106060149A (en) * | 2016-06-24 | 2016-10-26 | 北京交通大学 | Mobile internet mass data analysis and audit technical architecture |
| CN106354795A (en) * | 2016-08-26 | 2017-01-25 | 南威软件股份有限公司 | Distributed data exchanging system |
| CN107317724A (en) * | 2017-06-06 | 2017-11-03 | 中证信用增进股份有限公司 | Data collecting system and method based on cloud computing technology |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947837A (en) * | 2019-03-21 | 2019-06-28 | 上海富数科技有限公司 | Realize the system and method for intelligent scheduling data grabber function |
| CN110071873A (en) * | 2019-04-09 | 2019-07-30 | 华为技术有限公司 | A kind of method, apparatus and relevant device sending data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109412970B (en) | 2022-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10621009B2 (en) | System and method for topology-aware job scheduling and backfilling in an HPC environment | |
| CN104618693B (en) | A kind of monitor video based on cloud computing handles task management method and system online | |
| US9037833B2 (en) | High performance computing (HPC) node having a plurality of switch coupled processors | |
| CN101084488B (en) | Method and system for debugging multithreading procedure in a multicore architecture | |
| CN107918600A (en) | report development system and method, storage medium and electronic equipment | |
| CN104335137B (en) | Manage the power consumption and performance of computing system | |
| CN103729329B (en) | Intercore communication device and method | |
| CN108228354A (en) | Dispatching method, system, computer equipment and medium | |
| CN108595316B (en) | Lifecycle management method, manager, device, and medium for distributed application | |
| CN109783472A (en) | Moving method, device, computer equipment and the storage medium of table data | |
| US10225142B2 (en) | Method and system for communication between a management-server and remote host systems | |
| CN102043837A (en) | Data integration system and method | |
| CN103617067A (en) | Electric power software simulation system based on cloud computing | |
| CN102736595A (en) | Unified platform of intelligent power distribution terminal based on 32 bit microprocessor and real time operating system (RTOS) | |
| WO2024012221A1 (en) | Method for creating virtual service shared pool on basis of formatted data stream file base template | |
| CN109412970A (en) | Stream compression system, stream compression method, electronic equipment and storage medium | |
| CN109857550A (en) | Resource allocation method, device, equipment and storage medium based on machine learning | |
| CN116302306A (en) | Matching-based enhanced debugging for micro-service architecture | |
| CN104391990A (en) | Multi-task type collecting and harvesting method based on vertical industry | |
| Nikitenko et al. | Making large-scale systems observable-another inescapable step towards exascale | |
| CN103380608A (en) | Method for gathering queue information and job information in computation environment | |
| CN108334550A (en) | A kind of data processing system based on Human Resource Management System | |
| US9710349B2 (en) | Storing first computer trace information in memory of second computer | |
| CN105471986B (en) | A kind of Constructing data center Scale Revenue Ratio method and device | |
| Kidane et al. | NoC based virtualized FPGA as cloud Services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20190301 Assignee: Nanjing Manyun Cold Chain Technology Co.,Ltd. Assignor: JIANGSU MANYUN SOFTWARE TECHNOLOGY Co.,Ltd. Contract record no.: X2023980038397 Denomination of invention: Data flow system, data flow methods, electronic devices, and storage media Granted publication date: 20220705 License type: Common License Record date: 20230724 |