CN110245158A - A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology - Google Patents
A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology Download PDFInfo
- Publication number
- CN110245158A CN110245158A CN201910495241.0A CN201910495241A CN110245158A CN 110245158 A CN110245158 A CN 110245158A CN 201910495241 A CN201910495241 A CN 201910495241A CN 110245158 A CN110245158 A CN 110245158A
- Authority
- CN
- China
- Prior art keywords
- data
- flink
- kafka
- source
- mode
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 45
- 238000005516 engineering process Methods 0.000 title claims abstract description 45
- 238000000034 method Methods 0.000 title abstract description 18
- 230000011218 segmentation Effects 0.000 claims abstract description 47
- 238000012545 processing Methods 0.000 claims abstract description 28
- 238000004140 cleaning Methods 0.000 claims abstract description 22
- 238000013480 data collection Methods 0.000 claims abstract description 16
- 230000001747 exhibiting effect Effects 0.000 claims abstract description 4
- 238000005520 cutting process Methods 0.000 claims description 34
- 238000003860 storage Methods 0.000 claims description 17
- 230000002776 aggregation Effects 0.000 claims description 9
- 238000004220 aggregation Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 9
- 238000006116 polymerization reaction Methods 0.000 claims description 9
- 238000003672 processing method Methods 0.000 claims description 5
- 238000012423 maintenance Methods 0.000 abstract description 6
- 238000003745 diagnosis Methods 0.000 abstract description 5
- 239000013589 supplement Substances 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 9
- 238000003780 insertion Methods 0.000 description 8
- 230000037431 insertion Effects 0.000 description 8
- 239000012141 concentrate Substances 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
Abstract
The invention discloses a kind of multi-source heterogeneous generating date system and methods based on Flink stream calculation technology, the system includes: data acquisition side, the isomeric data being dispersed in multiple system components is obtained by log mode and/or SDK mode and/or MQ mode, is sent to Kafka in a manner of continuous flow after preliminary treatment；Task management platform side, configuration data Source Type, configure isomeric data cleaning and segmentation rules and configuration data collection dimension and index, and Flink stream calculation technology log-on data real time processing tasks are based on after the completion of all configurations, it is stored after data calculate in real time according to data set definition；Data exhibiting and outlet side, obtain the result output of data set, the present invention can analyze the data of a variety of source different structures exported in existing business system, find the correlation between log event and business, to help operation maintenance personnel to improve efficiency, supplement is provided for existing business diagnosis system.
Description
Technical field
The present invention relates to a kind of multi-source heterogeneous generating date system and methods, are based on Flink more particularly to one kind
The multi-source heterogeneous generating date system and method for stream calculation technology.
Background technique
In internet+epoch, for the demands such as the quick exploitation, the elastic telescopic that adapt to business, the IT system framework of enterprise
Positive Docker container cluster and the evolution of micro services direction, this framework improves resource utilization, it is bigger flexible to bring
Property, support high concurrent scene.
But with the increase of call relation complexity between the expansion of business scale, service, log output quantity is increasingly
More, when facing failure and performance issue, the difficulty of analysis is bigger, therefore, how to divide the mass data of system output
Wherein valuable information is found out in analysis, helps operation maintenance personnel to improve efficiency, providing supplement for existing business diagnosis system is urgently
Problem to be solved, multi-source heterogeneous generating date technology is come into being as a result,.
Currently, multi-source heterogeneous generating date technology is mainly by establishing self-defining data to solve current problem
The data that real time processing tasks export a variety of sources such as file journalization, Agent output, message queue carry out cleaning cutting, spirit
The dimension and index of ground living configuration data collection generate time series data, the data in graphic exhibition data set, according to configured
Alarm rule notifies contact person, mitigates the pressure of operation maintenance personnel, finds valuable information relevant with business.
However, for this problem, the solution of current multi-source heterogeneous generating date technology there is also it is some not
Foot place:
1, the visualized graph interface of data cleansing and segmentation rules can be constructed without providing, but passes through configuration file
It realizes.
2, daily record data is not converted into the time series data of structuring, so that index value can not be carried out according to time, dimension
Grouping calculates.
3, show interface without providing configurable data interaction chart.
Flink is the distributed data processing platform efficiently calculated based on memory, is the top project of Apache
One of.Its core is a streamed data stream engine (Streaming dataflow engine), provides point of data flow
Cloth data distribution, communication and fault tolerance have the characteristics such as efficient, reliable, expansible, and have with the Hadoop ecosystem
Well compatibility.Flink describes the data set of parallel computation using DataSet, and provides to corresponding data set
Such as map, reduce, join, group etc data-processing interface abundant.However, currently, not occurring also flowing based on Flink
The multi-source heterogeneous generating date technology of computing technique.
Summary of the invention
In order to overcome the deficiencies of the above existing technologies, purpose of the present invention is to provide one kind to be based on Flink stream calculation
The multi-source heterogeneous generating date system and method for technology, by a variety of source difference knots exported in existing business system
The mass data of structure is analyzed, and is handled node by all systems that key information match business is passed through, is found log event
Correlation between business helps operation maintenance personnel to improve efficiency, provides supplement for existing business diagnosis system.
In order to achieve the above object, the present invention proposes a kind of multi-source heterogeneous generating date based on Flink stream calculation technology
System, comprising:
Data acquire side, multiple for being dispersed in by log mode and/or the acquisition simultaneously of SDK mode and/or MQ mode
Isomeric data in system component, is sent to Kafka after preliminary treatment in a manner of continuous flow；
Task management platform side for configuration data Source Type, the cleaning of configuration isomeric data and segmentation rules and is matched
It sets the dimension and index of data set, and is based on after the completion of all configurations Flink stream calculation technology log-on data and handle in real time times
Business, and storage unit is stored according to data set definition after data calculate in real time；
Data exhibiting and outlet side show with chart mode or pass through interface mode for obtaining the result in data set
Output.
Preferably, the log mode is to read to specify the newly-increased interior of journal file in real time using log data acquisition device
Hold, be sent to log and collect module, the data of acquisition are sent into Kafka after modular filtration is collected in log；The SDK mode is
Support that insertion Agent uploads data as data source in application or container, Agent uploads data to background service, and data are passed through
Enter Kafka after background service processing；Or Kafka is directly sent data to by Agent as data source；The MQ mode is
Support Kafka message queue as data source, data are transmitted directly to Kafka.
Preferably, the task management platform side includes:
Configuration unit, for configuration data Source Type, the cleaning of configuration isomeric data and segmentation rules and configuration data
The dimension and index of collection；
Data processing unit is handled in real time for being based on Flink stream calculation technology log-on data after the completion of all configurations
Task, and storage unit, each generating date task corresponding one are stored according to data set definition after data calculate in real time
A Flink data segmentation task can have multiple data sets, the corresponding Flink data set of each data set in one task
Calculating task.
Preferably, when configuration data Source Type, if selecting daily record data as data source, input journal road is needed
Diameter needs to input the AccessKeys of SDK Agent if selecting insertion SDK Agent reported data as data source, if choosing
MQ is selected as data source, then needs to input the Topic of Kafka.
Preferably, it when configuring the cleaning and segmentation rules of isomeric data, is constructed in such a way that figure pulls building block
Data segmentation rules, and after the real time data for obtaining crawl, data cutting preview is carried out according to the data segmentation rules of definition
Trial cut point.
Preferably, it when the dimension and index of configuration data collection, according to the data definition data set after cutting, and needs to input
The parameters such as filter condition, polymerization dimension, statistical indicator, time field.
Preferably, the data processing unit further comprises:
Flink cleaning and cutting unit to data cutting and are patrolled for consuming the data in Kafka according to segmentation rules
Processing is collected, and cutting data are placed again into Kafka；
Flink computing unit calculates after Kafka consumption data according to time, dimension real time aggregation, and will be after calculating
As a result it is stored in storage unit；
Storage unit, including ElasticSearch search server and InfluxDb time series database, it is described
For ElasticSearch search server for storing initial data, the InfluxDb time series database is described for saving
Time series data after the polymerization calculating of Flink computing unit.
Preferably, the configuration unit is also used to configure customized alarm rule, and the data processing unit further includes
Flink alert process unit, for after consumption data in the Topic of Kafka, according to the alarm rule real-time judge whether
It needs to alarm, generates alarm logging, and notify contact person.
Preferably, the task management platform side further includes query unit, every for being inquired according to the input condition of acquisition
A data concentrate the data for having calculated completion.
In order to achieve the above objectives, the present invention also provides a kind of multi-source heterogeneous data based on Flink stream calculation technology are real-time
Processing method includes the following steps:
Step S1 acquires side in data, is obtained simultaneously by log mode or SDK mode or MQ mode and is dispersed in multiple systems
Isomeric data in system component, is sent to Kafka after preliminary treatment in a manner of continuous flow；
Step S2, in task management platform side, configuration data Source Type, configure cleaning and the segmentation rules of isomeric data with
And the dimension and index of configuration data collection, and be based on Flink stream calculation technology log-on data after the completion of all configurations and locate in real time
Reason task, and time series database is stored according to data set definition after data are calculated in real time；
Step S3, the result obtained in the task management platform side data set show with chart mode or pass through interface
Mode exports.
Compared with prior art, a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology of the present invention
System and method are analyzed by the mass data to a variety of source different structures exported in existing business system, pass through key
Information (such as TraceID, order number) matches all systems that business is passed through and handles node, finds between log event and business
Correlation, help operation maintenance personnel improve efficiency, provide supplement for existing business diagnosis system.
Detailed description of the invention
Fig. 1 is that a kind of structure of the multi-source heterogeneous generating date system based on Flink stream calculation technology of the present invention is shown
It is intended to；
The step of Fig. 2 is a kind of multi-source heterogeneous Real-time Data Processing Method based on Flink stream calculation technology of the invention is flowed
Cheng Tu；
Fig. 3 is the multi-source heterogeneous generating date system based on Flink stream calculation technology in the specific embodiment of the invention
Functional block diagram；
Fig. 4 is the multi-source heterogeneous generating date system based on Flink stream calculation technology in the specific embodiment of the invention
Logical architecture figure.
Specific embodiment
Below by way of specific specific example and embodiments of the present invention are described with reference to the drawings, those skilled in the art can
Understand further advantage and effect of the invention easily by content disclosed in the present specification.The present invention can also pass through other differences
Specific example implemented or applied, details in this specification can also be based on different perspectives and applications, without departing substantially from
Various modifications and change are carried out under spirit of the invention.
Before introducing the present invention, several open source components according to the present invention are first introduced:
1、Elasticsearch
Storage assembly as ultimate log, and can, it can be achieved that distributed storage, in real time search and mass data analysis
Log is persisted to disk.And the RESTful api interface for providing open source realizes being introduced directly into for log.
2、Logstash
The data in various formats and source can be collected, can storage format as needed write parsing script, realize data
The unitized output of format.
3、Filebeat
It for light-weighted log collection component, can install on the server, realize persistent collection and the transmission of log.
4、Kafka
Message system is subscribed to for a kind of distributed post of high-throughput, is capable of the storage log of persistence and fault-tolerance
Stream, can solve the speed of log collection and the speed inconsistence problems of processing.
Fig. 1 is that a kind of structure of the multi-source heterogeneous generating date system based on Flink stream calculation technology of the present invention is shown
It is intended to.As shown in Figure 1, a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology of the present invention, comprising:
Data acquire side 10, for passing through log mode or SDK (Software Development Kit, software tool
Packet) mode or MQ (Message Queue, message queue are also message-oriented middleware) mode while acquisition be dispersed in multiple systems
Isomeric data in component, is sent to Kafka after preliminary treatment in a manner of continuous flow.In the specific embodiment of the invention
In, data acquisition the collected isomeric data in side 10 for example in addition to having time stamp, call D-chain trace TraceId, error stack, answer
With service etc. outside system informations, there are also and the information such as the relevant order number of business, cell-phone number, commodity sign, the present invention is not with this
It is limited.
The log mode refers to the new content for reading specified journal file in real time using log data acquisition device, hair
It gives log and collects module, the data of acquisition are sent into Kafka after modular filtration is collected in log, in the specific embodiment of the invention
In, Yu Suoshu installs FileBeat light-type log collector in the server of data acquisition side 10, it can read and turn
Log lines are sent out, can also be restarted from the position of interruption, data is read and uploads LogStach, and match in the LogStach
The filtering rule of data is set, the data that multiple Filebeat are uploaded enter Kafka after passing through LogStach filtration treatment, that is,
It says, data acquire side 10 and support journal file as data source, and it is newly-increased interior to read specified journal file in real time using FileBeat
Hold, be sent to LogStach, data enter Kafka after LogStach is filtered；
The SDK mode refers to supporting that insertion Agent uploads data as data source, on Agent in application or container
Data are passed to background service, data enter Kafka after background service is handled, and certain Agent can also directly send out data
Kafka is given as data source；
The MQ mode refers to support Kafka message queue as data source, and data are transmitted directly to Kafka.
Task management platform side 20, be used for configuration data Source Type, configure isomeric data cleaning and segmentation rules and
The dimension and index of configuration data collection, and Flink stream calculation log-on data real time processing tasks are based on after the completion of all configurations,
And storage unit is stored according to data set definition after data calculate in real time.
Specifically, task management platform side 20 includes:
Configuration unit 201 needs to input day for configuration data Source Type if selecting daily record data as data source
Will path needs to input the AccessKeys of SDK Agent if selecting insertion SDK Agent reported data as data source,
If selecting MQ as data source, need to input the Topic of Kafka, in the specific embodiment of the invention, configuration unit 201
The a variety of data sources of configuration are held, and try the data that crawl uploads, examine the data source of configuration whether correct；Configuration unit 201
It is also used to configure the cleaning and segmentation rules of isomeric data, it, can be by web administration console in the specific embodiment of the invention
Segmentation rules are defined by graphical building blocks block mode in interface, support single, more separator sheers and simple logic
Operation, the cleaning of input test data preview and cutting are as a result, specifically, configuration unit 20 can pass through web administration console interface
Data segmentation rules are constructed in such a way that figure pulls building block, and after the real time data for obtaining crawl, data cutting is pre-
It lookes at and carries out trial cut point according to the data segmentation rules of definition, to help user judges the whether correct of segmentation rules configuration；Match
Dimension and index that unit 20 is also used to configuration data collection is set to need to input that is, according to the data definition data set after cutting
Parameters, a real time data calculating tasks such as filter condition, polymerization dimension, statistical indicator, time field can define multiple data
Collection, data set are calculated in real time based on the data after cutting.
Data processing unit 202 is based on Flink stream calculation technology log-on data after the completion of all configurations and handle in real time times
Business, and time series database is stored according to data set definition after data calculate in real time.In the present invention, each generating date
Task corresponds to a Flink data segmentation task, can there is multiple data sets in a task, and each data set is one corresponding
Flink data set calculating task, after task start, the corresponding segmentation task of task and affiliated data set calculating task are all opened
Dynamic, the newly-increased data in data source finally can all enter Kafka, and Flink data segmentation task consumes the data in Kafka, root
According to segmentation rules to data cutting and logical process, cutting processing result is placed again into Kafka, Flink data set calculating task
Time series database is stored in after calculating after Kafka consumption data according to time, dimension real time aggregation.
Specifically, data processing unit 202 further comprises:
Flink cleaning and cutting unit 2021, for consuming the data in Kafka, according to segmentation rules to data cutting
And logical process, and cutting data are placed again into Kafka；
Flink computing unit 2022 calculates after Kafka consumption data according to time, dimension real time aggregation, and will calculate
Result afterwards is stored in storage unit 2023, specifically, the consumption data from the Topic of Kafka of Flink computing unit 2022
Afterwards, then according to data set definition according to time, dimension packet aggregation parameter, composite index value, timing is stored into after calculating
Database.It should be noted that each data set has corresponding Flink Job task, the same Kafka can be consumed
Cutting data in Topic, generate different data sets.
Storage unit 2023, for storing related data.In the specific embodiment of the invention, storage unit 2023 includes
ElasticSearch search server and InfluxDb time series database, wherein ElasticSearch search server is used for
Initial data is stored, InfluxDb time series database is used to save the time series data after the polymerization of Flink computing unit 2022 calculates.
In the specific embodiment of the invention, data are acquired in the collected isomeric data in side 10 in addition to having time stamp, calling
Outside the system informations such as D-chain trace TraceId, error stack, application service, there are also and the relevant order number of business, cell-phone number, quotient
The information such as product mark, in task management platform side 20 according to the cleaning segmentation rules for configuring isomeric data, aggregated data collection
Dimension and index, order number, timestamp, call D-chain trace TraceId data to be stored in data to concentrate, starting
After the real-time calculating task of Flink, the data of business and system bi-directional association will enter data set, can thus pass through key
All systems that information (such as system tracks TraceID, order number) matching business is passed through handle node.
Preferably, configuration unit 201 can also configure customized alarm rule, and it is fixed that each newly-built alarm needs to input previous step
Justice data set, alert notice mode and notice object, alarm rule, in the specific embodiment of the invention, every alarm rule
Need to input following parameter:
Nearest a few minutes
Index in data set
Average value, total, maximum value, minimum value
Be more than or equal to, be less than or equal to, ring it is rise/fall % more year-on-year than rise/fall %, yesterday
Threshold values
A plurality of alarm rule can be defined in one alarm task.
Correspondingly, data processing unit 202 further includes Flink alert process unit, for disappearing from the Topic of Kafka
After taking data, whether need to alarm according to alarm rule real-time judge, generates alarm logging, and notify contact person, for example, by using
Short message or lettergram mode notify contact person.
Preferably, after the definition of the completion task of task management platform side 20 and alarm rule, it can star and stop
Task, i.e., the described task management platform side 20 can star, stop real time data processing Flink task, Flink cleaning and cutting
The data segmentation task of unit 2021, the data calculating task (Flink computing unit 2022) of each data set, alarm task
(Flink alert process unit) is all as an individual Flink Job operation.Each data set calculates the data completed and is put into
Time series database influxDb.If meeting alarm rule, alarm task generates alarm logging and notifies contact person.
Preferably, the task management platform side 20 further includes query unit, by inquiring in each data set based on
Calculate the data completed.Specifically, the task management platform side 20 can obtain input time model by web administration console interface
It encloses, time interval inquires the data that completion has been calculated in each data set.That is, the task management platform side 20 is also
The external inquiry function of data intensive data is provided, the Http inquiry request of external belt parameter is sent to background service, inquiry knot
Fruit is placed in Response to be returned in the form of JSON string.
Preferably, the task management platform side 20 can also be by defining in data set in web administration console interface
The chart ways of presentation of data, input data set number and its need show index, select a chart type and its configuration item after,
Can be according to configured item with the data in chart mode set of displayable data, in the specific embodiment of the invention, the web administration control
The task configuration data in Postgresql is read at platform interface processed using Ant Design React.js front end frame, is used
BizCharts.js shows the time series data being stored in Influxdb data set with chart mode.
Data exhibiting and outlet side 30 are showed or by interface side for obtaining the result in data set with chart mode
Formula output, that is to say, that the present invention provides external interface and exports the time series data being stored in data set, after user obtains data
Designed, designed chart shows data.
The step of Fig. 2 is a kind of multi-source heterogeneous Real-time Data Processing Method based on Flink stream calculation technology of the invention is flowed
Cheng Tu.As shown in Fig. 2, a kind of multi-source heterogeneous Real-time Data Processing Method based on Flink stream calculation technology of the present invention, including such as
Lower step:
Step S1 acquires side in data, is obtained simultaneously by log mode or SDK mode or MQ mode and is dispersed in multiple systems
Isomeric data in system component, is sent to Kafka after preliminary treatment in a manner of continuous flow.
The log mode refers to the new content for reading specified journal file in real time using log data acquisition device, hair
It gives log and collects module, the data of acquisition are sent into Kafka after modular filtration is collected in log, in the specific embodiment of the invention
In, Yu Suoshu installs FileBeat light-type log collector in the server of data acquisition side 10, it can read and turn
Log lines are sent out, can also be restarted from the position of interruption, data is read and uploads LogStach, and match in the LogStach
The filtering rule of data is set, the data that multiple Filebeat are uploaded enter Kafka after passing through LogStach filtration treatment, that is,
It says, data acquire side 10 and support journal file as data source, and it is newly-increased interior to read specified journal file in real time using FileBeat
Hold, be sent to LogStach, data enter Kafka after LogStach is filtered；
The SDK mode refers to supporting that insertion Agent uploads data as data source, on Agent in application or container
Data are passed to background service, data enter Kafka after background service is handled, and certain Agent can also directly send out data
Kafka is given as data source；
The MQ mode refers to support Kafka message queue as data source, and data are transmitted directly to Kafka.
Step S2, in task management platform side, configuration data Source Type, configure cleaning and the segmentation rules of isomeric data with
And the dimension and index of configuration data collection, and be based on Flink stream calculation technology log-on data after the completion of all configurations and locate in real time
Reason task, and read in real time that is, after the completion of configuration after data are calculated in real time according to data set definition deposit time series database
Data in Kafka are placed again into after extracting the data needed according to the real-time cutting of segmentation rules based on Flink stream calculation
Kafka carries out polymerization calculating according to time, dimension according to data set definition based on Flink stream calculation, and in time series database
Result after storage calculates.
In the specific embodiment of the invention, for configuration data Source Type, if selecting daily record data as data source, need
Input journal path is wanted, if selecting insertion SDK Agent reported data as data source, needs to input SDK Agent's
AccessKeys needs to input the Topic of Kafka if selecting MQ as data source, that is to say, that the present invention supports configuration
A variety of data sources, and the data that crawl uploads are tried, examine the data source of configuration whether correct；For configuration isomeric data
Cleaning and segmentation rules can be by passing through graphical building blocks in the specific embodiment of the invention in web administration console interface
Block mode is defined segmentation rules, supports single, more separator sheers and simple logical operation, input test data preview
Cleaning and cutting result；For the dimension and index of configuration data collection, the data after cutting are according to time, dimension packet aggregation meter
Index value is calculated, specifically, data cutting rule can be constructed in such a way that web administration console interface pulls building block using figure
Then, and after the real time data for obtaining crawl, data cutting preview carries out trial cut point according to the data segmentation rules of definition, with side
Help user judges the whether correct of segmentation rules configuration；For the dimension and index of configuration data collection, i.e., according to cutting after
Data definition data set needs to input the parameters such as filter condition, polymerization dimension, statistical indicator, time field, a real time data
Calculating task can define multiple data sets, and data set is calculated in real time based on the data after cutting.
The log-on data real time processing tasks after the completion of all configurations, and according to data set definition after data are calculated in real time
It is stored in time series database.In the present invention, the corresponding Flink data segmentation task of each generating date task, one
There can be multiple data sets in task, the corresponding Flink data set calculating task of each data set, after task start, task
Corresponding segmentation task and affiliated data set calculating task all start, and the newly-increased data in data source finally can all enter
Kafka, Flink data segmentation task consume the data in Kafka, according to segmentation rules to data cutting and logical process, knot
Fruit is placed again into Kafka, and Flink data set calculating task calculates after Kafka consumption data according to time, dimension real time aggregation
After be stored in time series database.
Preferably, in step S2, customized alarm rule is also configured, after consumption data in the Topic from Kafka,
Whether need to alarm according to alarm rule real-time judge, alarm logging is generated, and notify contact person, for example, by using short message or postal
Part mode notifies contact person.
Step S3, the result obtained in the task management platform side data set show with chart mode or pass through interface
Mode exports, that is to say, that the present invention also provides external interfaces to export the time series data being stored in data set, and user obtains number
Show data according to rear designed, designed chart.
It is below that the multi-source heterogeneous data based on Flink stream calculation technology for illustrating the present invention by specific embodiment are real
When processing system treatment process.Fig. 3 is the multi-source heterogeneous data based on Flink stream calculation technology in the specific embodiment of the invention
The functional block diagram of real time processing system, Fig. 4 are in the specific embodiment of the invention based on the multi-source heterogeneous of Flink stream calculation technology
The logical architecture figure of generating date system.It as shown in Figures 3 and 4, should the multi-source heterogeneous number based on Flink stream calculation technology
It is as follows according to the treatment process of real time processing system:
Data acquire side, support daily record data as data source, and it is newly-increased interior that FileBeat reads specified journal file in real time
Hold, be sent to LogStach, data enter Kafka after LogStach is filtered；Support that insertion SDK is uploaded in application or container
Data are as data source, and SDK uploads data to background service, and data enter Kafka after treatment, support Kafka message team
Column are used as data source, and data are transmitted directly to Kafka.
Console side is managed, configuration data Source Type is supported to need to input day if selecting daily record data as data source
Will path needs to input the AccessKeys of SDK Agent if selecting insertion SDK Agent reported data as data source,
If selecting MQ as data source, need to input the Topic of Kafka；Also support the clear of configuration isomeric data in management console side
It washes and segmentation rules, segmentation rules is defined by graphical building blocks block mode in interface, input test data preview is clear
It washes and cutting result；The dimension and index for supporting configuration data collection, the data after cutting are calculated according to time, dimension packet aggregation
Index value, in embodiments of the present invention, management console side includes:
Data receiver layer acquires the data that side uploads for receiving data；
Data analysis layer, the log-on data real time processing tasks after the completion of all configurations, each generating date task
A Flink data segmentation task is corresponded to, can there is multiple data sets, the corresponding Flink of each data set in a task
Data set calculating task.After task start, the corresponding segmentation task of task and affiliated data set calculating task all start.Number
It finally can all enter Kafka according to the newly-increased data in source, Flink task consumes the data in Kafka, according to segmentation rules logarithm
According to cutting and logical process, be as a result placed again into Kafka, Flink data set calculating task after Kafka consumption data according to when
Between, dimension real time aggregation calculate after be stored in accumulation layer, wherein accumulation layer include ElasticSearch search server and
InfluxDb time series database, wherein ElasticSearch search server is for storing initial data, ordinal number when InfluxDb
It is used to save the time series data after polymerization calculates according to library.
Input time range, time granularity is also supported to inquire the data in data set in the management console side.Support is matched
Set subtype and its option display data intensive data.If chart, which shows interface, is not able to satisfy user demand, support to pass through
HTTP interface externally provides the service of inquiry data intensive data, and input time range, time granularity, index and dimension list are made
The a plurality of time series data concentrated for querying condition, returned data.
In the present embodiment, management console side uses Ant Design React or Bizchart front end frame chart exhibition
Time series data in existing database.
Console, for providing the definition of task data source, the definition of task segmentation rules, the definition of task data collection, task pipe
Manage (starting/stopping), external interface service, data set inquiry, interaction chart, alerting service.
In conclusion the present invention is a kind of based on the multi-source heterogeneous generating date system of Flink stream calculation technology and side
Method is analyzed by the mass data to a variety of source different structures exported in existing business system, since data acquire side
In addition to system informations such as having time stamp, calling D-chain trace TraceId, error stack, application services in collected isomeric data
Outside, there are also and the information such as the relevant order number of business, cell-phone number, commodity sign, the configuration according to task management platform side
Cleaning segmentation rules, the dimension and index of aggregated data collection of isomeric data order number, timestamp, call D-chain trace
TraceId data are stored in a data and concentrate, after starting the real-time calculating task of Flink, the number of business and system bi-directional association
According to data set will be entered, it can thus pass through key message (such as system tracks TraceID, order number) matching business warp
All systems processing node crossed, finds the correlation between log event and business, helps operation maintenance personnel to improve efficiency, be existing
Some business diagnosis systems provide supplement.
The above-described embodiments merely illustrate the principles and effects of the present invention, and is not intended to limit the present invention.Any
Without departing from the spirit and scope of the present invention, modifications and changes are made to the above embodiments by field technical staff.Therefore,
The scope of the present invention, should be as listed in the claims.
Claims (10)
1. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology, comprising:
Data acquire side, are dispersed in multiple systems for obtaining simultaneously by log mode and/or SDK mode and/or MQ mode
Isomeric data in component, is sent to Kafka after preliminary treatment in a manner of continuous flow；
Task management platform side, for configuration data Source Type, the cleaning of configuration isomeric data and segmentation rules and configuration number
Flink stream calculation technology log-on data real time processing tasks are based on according to the dimension and index of collection, and after the completion of all configurations, and
Storage unit is stored according to data set definition after data calculate in real time；
Data exhibiting and outlet side show for obtaining the result in data set with chart mode or defeated by interface mode
Out.
2. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as described in claim 1, special
Sign is: the log mode is to read the new content of specified journal file in real time using log data acquisition device, is sent to
Module is collected in log, and the data of acquisition are sent into Kafka after modular filtration is collected in log；The SDK mode is to support application
Or be embedded in Agent in container and upload data as data source, Agent uploads data to background service, and data pass through background service
Enter Kafka after processing；Or Kafka is directly sent data to by Agent as data source；The MQ mode is to support
Kafka message queue is transmitted directly to Kafka as data source, data.
3. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 2, special
Sign is that the task management platform side includes:
Configuration unit for configuration data Source Type, configures cleaning and segmentation rules and the configuration data collection of isomeric data
Dimension and index；
Data processing unit, for being based on Flink stream calculation technology log-on data real time processing tasks after the completion of all configurations,
And storage unit is stored according to data set definition after data calculate in real time, each generating date task is one corresponding
Flink data segmentation task can have multiple data sets, the corresponding Flink data set meter of each data set in one task
Calculation task.
4. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 3, special
Sign is: when configuration data Source Type, if selecting daily record data as data source, input journal path is needed, if selection
Be embedded in SDK Agent reported data as data source, then need to input the AccessKeys of SDK Agent, if select MQ as
Data source then needs to input the Topic of Kafka.
5. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 3, special
Sign is: when configuring the cleaning and segmentation rules of isomeric data, constructing data cutting in such a way that figure pulls building block
Rule, and after the real time data for obtaining crawl, data cutting preview carries out trial cut point according to the data segmentation rules of definition.
6. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 3, special
Sign is: when the dimension and index of configuration data collection, according to the data definition data set after cutting, and need to input filtering rod
The parameters such as part, polymerization dimension, statistical indicator, time field.
7. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 3, special
Sign is that the data processing unit further comprises:
Flink cleaning and cutting unit, for consuming the data in Kafka, according to segmentation rules to data cutting and logic at
Reason, and cutting data are placed again into Kafka；
Flink computing unit calculates after Kafka consumption data according to time, dimension real time aggregation, and by the result after calculating
It is stored in storage unit；
Storage unit, including ElasticSearch search server and InfluxDb time series database, it is described
For ElasticSearch search server for storing initial data, the InfluxDb time series database is described for saving
Time series data after the polymerization calculating of Flink computing unit.
8. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 7, special
Sign is that the configuration unit is also used to configure customized alarm rule, and the data processing unit further includes at Flink alarm
Unit is managed, it is raw for whether needing to alarm according to the alarm rule real-time judge after consumption data in the Topic of Kafka
At alarm logging, and notify contact person.
9. a kind of multi-source heterogeneous generating date system based on Flink stream calculation technology as claimed in claim 7, special
Sign is: the task management platform side further includes query unit, for inquiring each data set according to the input condition of acquisition
In calculated the data of completion.
10. a kind of multi-source heterogeneous Real-time Data Processing Method based on Flink stream calculation technology, includes the following steps:
Step S1 acquires side in data, is obtained simultaneously by log mode or SDK mode or MQ mode and is dispersed in multiple system groups
Isomeric data in part, is sent to Kafka after preliminary treatment in a manner of continuous flow；
Step S2, in task management platform side, configuration data Source Type configures the cleaning of isomeric data and segmentation rules and matches
It sets the dimension and index of data set, and is based on after the completion of all configurations Flink stream calculation technology log-on data and handle in real time times
Business, and according to data set definition deposit time series database after data are calculated in real time；
Step S3, the result obtained in the task management platform side data set show with chart mode or pass through interface mode
Output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910495241.0A CN110245158A (en) | 2019-06-10 | 2019-06-10 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910495241.0A CN110245158A (en) | 2019-06-10 | 2019-06-10 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110245158A true CN110245158A (en) | 2019-09-17 |
Family
ID=67886263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910495241.0A Pending CN110245158A (en) | 2019-06-10 | 2019-06-10 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245158A (en) |
Cited By (46)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110688383A (en) * | 2019-09-26 | 2020-01-14 | 中国银行股份有限公司 | Data acquisition method and system |
| CN110691124A (en) * | 2019-09-24 | 2020-01-14 | 华润网络(深圳)有限公司 | Data processing method and system under remote multi-active architecture |
| CN110705891A (en) * | 2019-10-11 | 2020-01-17 | 集奥聚合(北京)人工智能科技有限公司 | Data processing method based on high-allocable changeability |
| CN110727700A (en) * | 2019-10-22 | 2020-01-24 | 中信银行股份有限公司 | Method and system for integrating multi-source streaming data into transaction type streaming data |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN111177193A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Flink-based log streaming processing method and system |
| CN111221831A (en) * | 2019-12-26 | 2020-06-02 | 杭州顺网科技股份有限公司 | Computing system for real-time processing of advertisement effect data |
| CN111241044A (en) * | 2020-01-08 | 2020-06-05 | 中国联合网络通信集团有限公司 | Method, device and equipment for building heterogeneous database and readable storage medium |
| CN111339052A (en) * | 2020-02-28 | 2020-06-26 | 中国银联股份有限公司 | Unstructured log data processing method and device |
| CN111523003A (en) * | 2020-04-27 | 2020-08-11 | 北京图特摩斯科技有限公司 | Data application method and platform with time sequence dynamic map as core |
| CN111657891A (en) * | 2020-06-23 | 2020-09-15 | 北京理工大学 | Method for monitoring health condition of old people based on edge computing platform |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112069264A (en) * | 2020-09-10 | 2020-12-11 | 阳光保险集团股份有限公司 | Heterogeneous data source acquisition method and device, electronic equipment and storage medium |
| CN112148578A (en) * | 2020-10-12 | 2020-12-29 | 贵州电网有限责任公司 | IT fault defect prediction method based on machine learning |
| CN112202818A (en) * | 2020-12-01 | 2021-01-08 | 南京中孚信息技术有限公司 | Network traffic intrusion detection method and system fusing threat information |
| CN112214866A (en) * | 2020-08-25 | 2021-01-12 | 武汉船用机械有限责任公司 | Health condition judgment method and device for marine equipment and computer equipment |
| CN112286757A (en) * | 2020-10-12 | 2021-01-29 | 浙江深大智能科技有限公司 | Data synchronization monitoring method and device, electronic equipment and storage medium |
| CN112307057A (en) * | 2020-10-27 | 2021-02-02 | 北京健康之家科技有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN112330147A (en) * | 2020-11-04 | 2021-02-05 | 北京思特奇信息技术股份有限公司 | Service acceptance information monitoring method and device and storage medium |
| CN112422445A (en) * | 2020-10-10 | 2021-02-26 | 四川新网银行股份有限公司 | Kafka-based real-time acquisition, calculation and storage method for buried point data |
| CN112506743A (en) * | 2020-12-09 | 2021-03-16 | 天津狮拓信息技术有限公司 | Log monitoring method and device and server |
| CN112527385A (en) * | 2021-02-18 | 2021-03-19 | 成都新希望金融信息有限公司 | Data processing method, device, working node and storage medium |
| CN112543127A (en) * | 2019-09-23 | 2021-03-23 | 北京轻享科技有限公司 | Monitoring method and device of micro-service architecture |
| CN112596997A (en) * | 2020-12-29 | 2021-04-02 | 科技谷（厦门）信息技术有限公司 | Automatic flow control method based on Flink real-time calculation |
| CN112667683A (en) * | 2020-12-25 | 2021-04-16 | 平安科技（深圳）有限公司 | Stream computing system, electronic device and storage medium therefor |
| CN112948455A (en) * | 2021-01-08 | 2021-06-11 | 四川新网银行股份有限公司 | Real-time analysis and calculation method based on Apache drive |
| CN113010483A (en) * | 2020-11-20 | 2021-06-22 | 云智慧（北京）科技有限公司 | Mass log management method and system |
| CN113094250A (en) * | 2021-05-12 | 2021-07-09 | 成都新希望金融信息有限公司 | Log early warning method and device, electronic equipment and storage medium |
| CN113111652A (en) * | 2020-01-13 | 2021-07-13 | 阿里巴巴集团控股有限公司 | Data processing method and device and computing equipment |
| CN113268503A (en) * | 2021-04-02 | 2021-08-17 | 北京比格大数据有限公司 | Information aggregation method, storage medium, and computer device |
| CN113515500A (en) * | 2021-05-24 | 2021-10-19 | 苏州维众数据技术有限公司 | Visual data processing system and processing method |
| CN113518365A (en) * | 2021-04-29 | 2021-10-19 | 北京红山信息科技研究院有限公司 | Data association method, device, server and storage medium |
| CN113568761A (en) * | 2020-04-28 | 2021-10-29 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN113626447A (en) * | 2021-10-12 | 2021-11-09 | 民航成都信息技术有限公司 | Civil aviation data management platform and method |
| CN113641301A (en) * | 2021-02-19 | 2021-11-12 | 中国建设银行股份有限公司 | Data management method and device |
| CN113656264A (en) * | 2021-09-08 | 2021-11-16 | 上海童渠信息技术有限公司 | Real-time alarm service platform system |
| CN113824601A (en) * | 2021-11-24 | 2021-12-21 | 国网江苏省电力有限公司营销服务中心 | Electric power marketing monitored control system based on service log |
| CN113844976A (en) * | 2021-09-10 | 2021-12-28 | 北京声智科技有限公司 | Alarm data processing method and device, computer equipment and storage medium |
| CN114116842A (en) * | 2021-11-25 | 2022-03-01 | 上海柯林布瑞信息技术有限公司 | Multi-dimensional medical data real-time acquisition method and device, electronic equipment and storage medium |
| CN114297189A (en) * | 2022-01-10 | 2022-04-08 | 成都国铁电气设备有限公司 | Method for cleaning geometric detection data of subway track based on Flink stream processing |
| CN114371884A (en) * | 2021-12-31 | 2022-04-19 | 南京星云数字技术有限公司 | Method, device, equipment and storage medium for processing Flink calculation task |
| CN114629929A (en) * | 2022-03-16 | 2022-06-14 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114979186A (en) * | 2022-05-16 | 2022-08-30 | 浪潮云信息技术股份公司 | Flow link analysis method and system based on Flink component |
| CN115168474A (en) * | 2022-07-26 | 2022-10-11 | 成都智元汇信息技术股份有限公司 | Internet of things center station system building method based on big data model |
| CN115499303A (en) * | 2022-08-29 | 2022-12-20 | 浪潮软件科技有限公司 | Log analysis tool based on Flink |
| CN115952200A (en) * | 2023-01-17 | 2023-04-11 | 安芯网盾(北京)科技有限公司 | Multi-source heterogeneous data aggregation query method and device based on MPP (maximum power point tracking) architecture |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN107545014A (en) * | 2016-06-28 | 2018-01-05 | 国网天津市电力公司 | Stream calculation instant disposal system for treating based on Storm |
| CN107577805A (en) * | 2017-09-26 | 2018-01-12 | 华南理工大学 | A kind of business service system towards the analysis of daily record big data |
| CN107908690A (en) * | 2017-11-01 | 2018-04-13 | 南京欣网互联网络科技有限公司 | A kind of data processing method based on big data OA operation analysis |
| CN109271412A (en) * | 2018-09-28 | 2019-01-25 | 中国-东盟信息港股份有限公司 | The real-time streaming data processing method and system of smart city |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| CN109710731A (en) * | 2018-11-19 | 2019-05-03 | 北京计算机技术及应用研究所 | A kind of multidirectional processing system of data flow based on Flink |
-
2019
- 2019-06-10 CN CN201910495241.0A patent/CN110245158A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN107545014A (en) * | 2016-06-28 | 2018-01-05 | 国网天津市电力公司 | Stream calculation instant disposal system for treating based on Storm |
| CN107577805A (en) * | 2017-09-26 | 2018-01-12 | 华南理工大学 | A kind of business service system towards the analysis of daily record big data |
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| CN107908690A (en) * | 2017-11-01 | 2018-04-13 | 南京欣网互联网络科技有限公司 | A kind of data processing method based on big data OA operation analysis |
| CN109271412A (en) * | 2018-09-28 | 2019-01-25 | 中国-东盟信息港股份有限公司 | The real-time streaming data processing method and system of smart city |
| CN109710731A (en) * | 2018-11-19 | 2019-05-03 | 北京计算机技术及应用研究所 | A kind of multidirectional processing system of data flow based on Flink |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Non-Patent Citations (3)
| Title |
|---|
| 从义昊: "基于数据流的分布式实时推荐算法的研究与实现" * |
| 李若鹏: "基于大数据的网络异常行为检测平台的设计与实现" * |
| 蔡鲲鹏: "基于Flink平台的应用研究" * |
Cited By (62)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112543127A (en) * | 2019-09-23 | 2021-03-23 | 北京轻享科技有限公司 | Monitoring method and device of micro-service architecture |
| CN110691124B (en) * | 2019-09-24 | 2020-11-24 | 华润网络(深圳)有限公司 | Data processing method and system under remote multi-active architecture |
| CN110691124A (en) * | 2019-09-24 | 2020-01-14 | 华润网络(深圳)有限公司 | Data processing method and system under remote multi-active architecture |
| CN110688383A (en) * | 2019-09-26 | 2020-01-14 | 中国银行股份有限公司 | Data acquisition method and system |
| CN110705891A (en) * | 2019-10-11 | 2020-01-17 | 集奥聚合(北京)人工智能科技有限公司 | Data processing method based on high-allocable changeability |
| CN110727700A (en) * | 2019-10-22 | 2020-01-24 | 中信银行股份有限公司 | Method and system for integrating multi-source streaming data into transaction type streaming data |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN110784419B (en) * | 2019-10-22 | 2023-02-28 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional railway electric service data |
| CN111177193A (en) * | 2019-12-13 | 2020-05-19 | 航天信息股份有限公司 | Flink-based log streaming processing method and system |
| CN111221831A (en) * | 2019-12-26 | 2020-06-02 | 杭州顺网科技股份有限公司 | Computing system for real-time processing of advertisement effect data |
| CN111221831B (en) * | 2019-12-26 | 2024-03-29 | 杭州顺网科技股份有限公司 | Computing system for processing advertisement effect data in real time |
| CN111241044A (en) * | 2020-01-08 | 2020-06-05 | 中国联合网络通信集团有限公司 | Method, device and equipment for building heterogeneous database and readable storage medium |
| CN111241044B (en) * | 2020-01-08 | 2023-09-19 | 中国联合网络通信集团有限公司 | Method, device, equipment and readable storage medium for constructing heterogeneous database |
| CN113111652A (en) * | 2020-01-13 | 2021-07-13 | 阿里巴巴集团控股有限公司 | Data processing method and device and computing equipment |
| CN113111652B (en) * | 2020-01-13 | 2024-02-13 | 阿里巴巴集团控股有限公司 | Data processing method and device and computing equipment |
| CN111339052A (en) * | 2020-02-28 | 2020-06-26 | 中国银联股份有限公司 | Unstructured log data processing method and device |
| CN111523003A (en) * | 2020-04-27 | 2020-08-11 | 北京图特摩斯科技有限公司 | Data application method and platform with time sequence dynamic map as core |
| CN113568761A (en) * | 2020-04-28 | 2021-10-29 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN113568761B (en) * | 2020-04-28 | 2023-06-27 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN111657891A (en) * | 2020-06-23 | 2020-09-15 | 北京理工大学 | Method for monitoring health condition of old people based on edge computing platform |
| CN112214866B (en) * | 2020-08-25 | 2023-11-17 | 武汉船用机械有限责任公司 | Marine equipment health condition judging method and device and computer equipment |
| CN112214866A (en) * | 2020-08-25 | 2021-01-12 | 武汉船用机械有限责任公司 | Health condition judgment method and device for marine equipment and computer equipment |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
| CN112069264A (en) * | 2020-09-10 | 2020-12-11 | 阳光保险集团股份有限公司 | Heterogeneous data source acquisition method and device, electronic equipment and storage medium |
| CN112422445A (en) * | 2020-10-10 | 2021-02-26 | 四川新网银行股份有限公司 | Kafka-based real-time acquisition, calculation and storage method for buried point data |
| CN112286757A (en) * | 2020-10-12 | 2021-01-29 | 浙江深大智能科技有限公司 | Data synchronization monitoring method and device, electronic equipment and storage medium |
| CN112148578A (en) * | 2020-10-12 | 2020-12-29 | 贵州电网有限责任公司 | IT fault defect prediction method based on machine learning |
| CN112307057A (en) * | 2020-10-27 | 2021-02-02 | 北京健康之家科技有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN112330147A (en) * | 2020-11-04 | 2021-02-05 | 北京思特奇信息技术股份有限公司 | Service acceptance information monitoring method and device and storage medium |
| CN113010483A (en) * | 2020-11-20 | 2021-06-22 | 云智慧（北京）科技有限公司 | Mass log management method and system |
| CN112202818A (en) * | 2020-12-01 | 2021-01-08 | 南京中孚信息技术有限公司 | Network traffic intrusion detection method and system fusing threat information |
| CN112506743A (en) * | 2020-12-09 | 2021-03-16 | 天津狮拓信息技术有限公司 | Log monitoring method and device and server |
| CN112667683A (en) * | 2020-12-25 | 2021-04-16 | 平安科技（深圳）有限公司 | Stream computing system, electronic device and storage medium therefor |
| CN112667683B (en) * | 2020-12-25 | 2023-05-26 | 平安科技（深圳）有限公司 | Stream computing system, electronic device thereof, and storage medium |
| CN112596997A (en) * | 2020-12-29 | 2021-04-02 | 科技谷（厦门）信息技术有限公司 | Automatic flow control method based on Flink real-time calculation |
| CN112948455A (en) * | 2021-01-08 | 2021-06-11 | 四川新网银行股份有限公司 | Real-time analysis and calculation method based on Apache drive |
| CN112527385A (en) * | 2021-02-18 | 2021-03-19 | 成都新希望金融信息有限公司 | Data processing method, device, working node and storage medium |
| CN112527385B (en) * | 2021-02-18 | 2021-11-30 | 成都新希望金融信息有限公司 | Data processing method, device, working node and storage medium |
| CN113641301A (en) * | 2021-02-19 | 2021-11-12 | 中国建设银行股份有限公司 | Data management method and device |
| CN113268503A (en) * | 2021-04-02 | 2021-08-17 | 北京比格大数据有限公司 | Information aggregation method, storage medium, and computer device |
| CN113268503B (en) * | 2021-04-02 | 2024-03-08 | 北京比格大数据有限公司 | Information aggregation method, storage medium, and computer device |
| CN113518365A (en) * | 2021-04-29 | 2021-10-19 | 北京红山信息科技研究院有限公司 | Data association method, device, server and storage medium |
| CN113518365B (en) * | 2021-04-29 | 2023-11-17 | 北京红山信息科技研究院有限公司 | Data association method, device, server and storage medium |
| CN113094250A (en) * | 2021-05-12 | 2021-07-09 | 成都新希望金融信息有限公司 | Log early warning method and device, electronic equipment and storage medium |
| CN113094250B (en) * | 2021-05-12 | 2023-08-18 | 成都新希望金融信息有限公司 | Log early warning method and device, electronic equipment and storage medium |
| CN113515500A (en) * | 2021-05-24 | 2021-10-19 | 苏州维众数据技术有限公司 | Visual data processing system and processing method |
| CN113515500B (en) * | 2021-05-24 | 2023-06-30 | 苏州维众数据技术有限公司 | Visual data processing system and processing method |
| CN113656264A (en) * | 2021-09-08 | 2021-11-16 | 上海童渠信息技术有限公司 | Real-time alarm service platform system |
| CN113844976A (en) * | 2021-09-10 | 2021-12-28 | 北京声智科技有限公司 | Alarm data processing method and device, computer equipment and storage medium |
| CN113844976B (en) * | 2021-09-10 | 2023-04-25 | 北京声智科技有限公司 | Alarm data processing method, device, computer equipment and storage medium |
| CN113626447A (en) * | 2021-10-12 | 2021-11-09 | 民航成都信息技术有限公司 | Civil aviation data management platform and method |
| CN113824601A (en) * | 2021-11-24 | 2021-12-21 | 国网江苏省电力有限公司营销服务中心 | Electric power marketing monitored control system based on service log |
| CN114116842B (en) * | 2021-11-25 | 2023-05-19 | 上海柯林布瑞信息技术有限公司 | Multidimensional medical data real-time acquisition method and device, electronic equipment and storage medium |
| CN114116842A (en) * | 2021-11-25 | 2022-03-01 | 上海柯林布瑞信息技术有限公司 | Multi-dimensional medical data real-time acquisition method and device, electronic equipment and storage medium |
| CN114371884A (en) * | 2021-12-31 | 2022-04-19 | 南京星云数字技术有限公司 | Method, device, equipment and storage medium for processing Flink calculation task |
| CN114297189A (en) * | 2022-01-10 | 2022-04-08 | 成都国铁电气设备有限公司 | Method for cleaning geometric detection data of subway track based on Flink stream processing |
| CN114629929A (en) * | 2022-03-16 | 2022-06-14 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114629929B (en) * | 2022-03-16 | 2024-03-08 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114979186A (en) * | 2022-05-16 | 2022-08-30 | 浪潮云信息技术股份公司 | Flow link analysis method and system based on Flink component |
| CN115168474A (en) * | 2022-07-26 | 2022-10-11 | 成都智元汇信息技术股份有限公司 | Internet of things center station system building method based on big data model |
| CN115499303A (en) * | 2022-08-29 | 2022-12-20 | 浪潮软件科技有限公司 | Log analysis tool based on Flink |
| CN115952200A (en) * | 2023-01-17 | 2023-04-11 | 安芯网盾(北京)科技有限公司 | Multi-source heterogeneous data aggregation query method and device based on MPP (maximum power point tracking) architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245158A (en) | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology | |
| CN107577805B (en) | Business service system for log big data analysis | |
| US11086687B2 (en) | Managing resource allocation in a stream processing framework | |
| CN108335075B (en) | Logistics big data oriented processing system and method | |
| CN104407964B (en) | A kind of centralized monitoring system and method based on data center | |
| CN109634818A (en) | Log analysis method, system, terminal and computer readable storage medium | |
| CN108365985A (en) | A kind of cluster management method, device, terminal device and storage medium | |
| US20100070981A1 (en) | System and Method for Performing Complex Event Processing | |
| CN103401698B (en) | For the monitoring system that server health is reported to the police in server set group operatione | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN107291594A (en) | The device and method that openstack platforms are monitored and managed to ceph | |
| CN111221831B (en) | Computing system for processing advertisement effect data in real time | |
| CN112506743A (en) | Log monitoring method and device and server | |
| CN113448812A (en) | Monitoring alarm method and device under micro-service scene | |
| CN114416684A (en) | Method for identifying part of executable logic, hardware storage device and computing system | |
| CN108595316A (en) | Life cycle management method, manager, equipment and the medium of Distributed Application | |
| CN112965874A (en) | Configurable monitoring alarm method and system | |
| CN109190025A (en) | information monitoring method, device, system and computer readable storage medium | |
| CN111177237B (en) | Data processing system, method and device | |
| US11288258B2 (en) | Dedicated audit port for implementing recoverability in outputting audit data | |
| CN111339052A (en) | Unstructured log data processing method and device | |
| CN103823743A (en) | Monitoring method and monitoring device of software system | |
| CN111263120B (en) | Video equipment online time counting method and device based on stream calculation and terminal | |
| CN116232844A (en) | System monitoring method based on distributed system | |
| CN114756301B (en) | Log processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190917 |
|
| RJ01 | Rejection of invention patent application after publication |