CN116841752A - Data analysis and calculation system based on distributed real-time calculation framework - Google Patents
Data analysis and calculation system based on distributed real-time calculation framework Download PDFInfo
- Publication number
- CN116841752A CN116841752A CN202311109401.6A CN202311109401A CN116841752A CN 116841752 A CN116841752 A CN 116841752A CN 202311109401 A CN202311109401 A CN 202311109401A CN 116841752 A CN116841752 A CN 116841752A
- Authority
- CN
- China
- Prior art keywords
- calculation
- data
- request
- unit
- distributed real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 116
- 238000007405 data analysis Methods 0.000 title claims abstract description 22
- 230000002776 aggregation Effects 0.000 claims abstract description 34
- 238000004220 aggregation Methods 0.000 claims abstract description 34
- 238000001914 filtration Methods 0.000 claims abstract description 14
- 238000000605 extraction Methods 0.000 claims abstract description 7
- 238000012216 screening Methods 0.000 claims description 13
- 238000004458 analytical method Methods 0.000 claims description 7
- 230000002123 temporal effect Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract description 4
- 238000000034 method Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 14
- 238000011161 development Methods 0.000 description 3
- 231100000279 safety data Toxicity 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 239000010419 fine particle Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data analysis and calculation system based on a distributed real-time calculation framework, which comprises: a distributed real-time computing framework and a business application, the distributed real-time computing framework comprising: the request module generates a writing request and a reading request according to the streaming data; the filtering module generates a writing instruction when the writing request meets a preset writing condition, and generates a reading instruction when the reading request meets a preset reading condition; the calculation module comprises: the extraction unit extracts a plurality of calculation parameters from the streaming data according to the type of the aggregation field; the first calculation unit substitutes each calculation parameter into a primary calculation formula in an operator template to obtain a plurality of primary data, and divides each primary data into a plurality of intermediate data according to fine granularity; and after receiving the reading instruction, the second calculation unit screens each intermediate data to obtain a plurality of screened data, and substitutes each screened data into a convergence formula to obtain final data. The invention reduces the calculation cost and improves the calculation speed.
Description
Technical Field
The invention relates to the technical field of business data processing, in particular to a data analysis and calculation system based on a distributed real-time calculation framework.
Background
At present, most of data analysis and calculation systems adopt distributed calculation frameworks such as Spark, flink and the like to complete tasks of data analysis and calculation. In the process of using the distributed computing framework, the data analysis computing system performs big data task development, writes the distributed computing task, executes the computing task in the distributed computing framework such as Spark or Flink, and writes the computing result into a database for the service system to use. For safety data analysis, the existing data analysis computing system needs to deploy a huge operation architecture, and meanwhile, the computation of real-time streaming data in service application needs to be actively initiated, so that additional dispatch service connection needs to be configured, and therefore, the operation cost and the computation amount in the data computation process are large.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention aims to provide a data analysis and calculation system based on a distributed real-time calculation frame, which is used for reducing calculation cost and improving calculation speed.
In order to achieve the above purpose, the present invention provides the following technical solutions: a data analysis computing system based on a distributed real-time computing framework, comprising a distributed real-time computing framework and a business application, the distributed real-time computing framework being connected to the business application and comprising:
the request module is used for generating a writing request and a reading request according to the streaming data of the business application;
the filtering module is connected with the request module and is used for generating a writing instruction when the writing request meets preset writing conditions and generating a reading instruction when the reading request meets preset reading conditions;
the calculation module is connected with the filtering module and comprises:
the extraction unit is used for extracting a plurality of calculation parameters from the streaming data according to the type of the aggregation field after receiving the writing instruction;
the first calculation unit is connected with the extraction unit and is used for substituting each calculation parameter into a primary calculation formula in the operator template to obtain a plurality of primary data, and dividing each primary data into a plurality of intermediate data according to fine granularity;
the second calculation unit is connected with the first calculation unit and is used for screening each intermediate data according to a preset screening condition in the operator template to obtain a plurality of screening data after receiving the reading instruction, and substituting each screening data into a preset convergence formula to obtain final data.
Further, the write request includes an aggregation field and a calculation target of the streaming data, the read request includes the aggregation field, the write condition includes a field condition and a limiting condition, and the read condition includes a field condition;
the filter module includes:
the analysis unit is used for analyzing the write-in request to obtain the aggregation field and the data content, and analyzing the read-out request to obtain the aggregation field;
and the generation unit is connected with the analysis unit and is used for generating the writing instruction when the aggregation field in the writing request meets the field condition and the calculation target meets the limiting condition, and generating the reading instruction when the aggregation field in the reading request meets the field condition.
Further, the distributed real-time computing framework and the business application are integrated to run on nodes, and a plurality of the nodes are connected with a node task balancing module, wherein the node task balancing module comprises: the node counting unit is used for counting the number of the nodes and generating a task adjustment instruction when the number of the nodes changes;
and the task balancing unit is connected with the node counting unit and is used for summarizing the tasks on the current nodes into the total tasks according to the task adjustment instructions and redistributing the total tasks according to the number of the current nodes.
Further, the computing module further includes:
the storage unit is used for storing a plurality of operator templates, and each operator template is matched with the corresponding type of the aggregation field;
and the matching unit is respectively connected with the storage unit and the first computing unit and is used for matching the types of the aggregation fields in the storage unit to obtain the corresponding operator templates.
Further, the fine granularity is a temporal granularity.
Further, a read interface and a write interface are configured on the distributed real-time computing framework, and the distributed real-time computing framework obtains the read request through the read interface and obtains the write request through the write interface.
The invention has the beneficial effects that:
the invention directly obtains the read request and the write request through the interface aiming at the streaming data generated by the business application in the safety data analysis scene, and does not need to be configured with additional dispatch service connection, thereby realizing analysis and calculation of the streaming data and reducing the use cost of distributed calculation; meanwhile, the streaming data is subjected to primary calculation by utilizing a primary calculation formula when being written, and is subjected to convergence calculation according to a convergence formula when the call is required to be read, so that the streaming data is different from calculation which is required to be actively initiated in the prior art, the calculation cost is reduced as a whole, the calculation cost and the storage cost of a large amount of real-time calculation are saved, the real-time calculation task can be rapidly and conveniently completed by service application, and the aim of the service application is fulfilled.
Drawings
FIG. 1 is a schematic diagram of a data analysis computing system based on a distributed real-time computing framework in accordance with the present invention.
Reference numerals: 1. a distributed real-time computing framework; 2. business application; 3. a request module; 4. a filtration module; 41. an analysis unit; 42. a generating unit; 5. a computing module; 51. an extraction unit; 52. a first calculation unit; 53. a second calculation unit; 54. a storage unit; 55. a matching unit; 6. a node task balancing module; 61. a node statistics unit; 62. and a task balancing unit.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It will be understood that when an element is referred to as being "fixed to" another element, it can be directly on the other element or intervening elements may also be present. When a component is considered to be "connected" to another component, it can be directly connected to the other component or intervening components may also be present. When an element is referred to as being "disposed on" another element, it can be directly on the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
Referring to fig. 1, the present embodiment provides a data analysis computing system based on a distributed real-time computing framework, which includes a distributed real-time computing framework 1 and a service application 2, wherein the distributed real-time computing framework 1 is connected with the service application 2, and includes:
a request module 3, configured to generate a write request and a read request according to the streaming data of the service application 2;
a filtering module 4, connected to the request module 3, configured to generate a write instruction when the write request satisfies a preset write condition, and generate a read instruction when the read request satisfies a preset read condition;
a calculation module 5, connected to the filtering module 4, comprising:
the extracting unit 51 is configured to extract a plurality of calculation parameters from the streaming data according to the type of the aggregation field after receiving the write command;
a first calculation unit 52, connected to the extraction unit 51, configured to substitute each calculation parameter into a primary calculation formula in the operator template to obtain a plurality of primary data, and divide each primary data into a plurality of intermediate data according to a fine granularity;
the second calculating unit 53 is connected to the first calculating unit 52, and is configured to, after receiving the reading instruction, screen each intermediate data according to a preset screening condition in the operator template to obtain a plurality of screening data, and then substitute each screening data into a preset convergence formula to obtain final data.
Specifically, in this embodiment, when service data needs to be written into or read from the distributed real-time computing framework 1, the request module 3 generates a write request or a read request according to the streaming data, and sends the write request or the read request to the filter module 4. The filtering module 4 is configured to determine whether the write request satisfies a write condition, and determine whether the read request satisfies a read condition, generate a write instruction when the write request satisfies the write condition, and generate a read instruction when the read instruction satisfies the read condition. The writing instruction and the reading instruction both comprise calculation tasks, the calculation tasks are matched with the operator templates, after the extracting unit 51 receives the writing instruction, the corresponding operator templates are obtained by matching according to the calculation tasks in the writing instruction, then the extracting unit 51 extracts a plurality of calculation parameters from the streaming data according to the type of the aggregation field, the calculation parameters are substituted into a primary calculation formula in the operator templates, primary data is obtained by calculation, and then the primary data is aggregated to a fine-grained intermediate result to form intermediate data. The second calculation unit 53 matches the calculation tasks included in the read instruction to obtain a corresponding operator template after receiving the read instruction, then the second calculation unit 53 firstly screens each intermediate data according to the screening conditions in the operator template, and then substitutes the screened data obtained after the screening into a convergence formula to perform convergence operation to obtain final data.
According to the technical scheme, aiming at the streaming data generated by the service application 2 in the safety data analysis scene, the reading request and the writing request are directly acquired through the interface, and additional dispatching service connection is not required to be configured, so that the analysis and calculation of the streaming data are realized, and the use cost of the distributed calculation is reduced; meanwhile, in the technical scheme, the streaming data is subjected to primary calculation by utilizing a primary calculation formula when being written, and is subjected to convergence calculation according to a convergence formula when being required to be read and called, so that the streaming data is different from calculation which is required to be actively initiated in the prior art, the calculation cost is reduced as a whole, a large amount of calculation cost and storage cost of real-time calculation are saved, the real-time calculation task can be rapidly and conveniently completed by the service application 2, and the aim of the service application 2 is fulfilled.
The functions realized by the invention also comprise:
1. a lightweight distributed real-time computing framework 1 is constructed:
the invention provides a computing framework formed by designing and packaging for meeting the scene requirement of real-time analysis service of time sequence stream data. Different from the traditional real-time computing framework in the market, which has huge functions and numerous modules, the distributed real-time computing framework 1 solves the problem of multi-value real-time computing of streaming data and adopts a fixed computing mode and flow. Therefore, the business personnel can quickly understand the distributed real-time computing framework 1, and can easily use the distributed real-time computing framework. The real-time computing developer can also rapidly complete design development work under the framework according to the limiting mode. Meanwhile, by adopting a fixed calculation mode and flow, the distributed real-time calculation framework 1 can perform deeper performance optimization and characteristic support, and the calculation mode reaches the advanced performance which the general real-time calculation module 5 cannot have.
2. Supporting operator template encapsulation
In the use of the invention, based on the industry experience of a data analyst, more than twenty real-time calculation operators are designed and packaged into operator templates, and only corresponding calculation parameters are required to be configured during the use. In one embodiment, a standard deviation calculation operator is used, when the operator is used, a latitude field A is configured and calculated, a field B (which is a digital value) for standard deviation calculation is needed, and additionally, a filtering condition can be configured by the filtering module 4 to screen a data set participating in calculation from streaming data. After the configuration is completed, the distributed real-time computing framework 1 divides stream data meeting the filtering condition according to the configuration, according to the dimension of the field A, then calculates variance according to the value accumulation of the field B, and stores the calculation result in the database in real time.
Because of the light weight and flexibility of the distributed real-time computing framework 1, users can conveniently and rapidly complete the secondary development, the general capability packaging and other works on the framework. The encapsulated operator can rapidly apply the algorithm mode to a plurality of business scenes through simple configuration. Through the operator template capability, the business purpose can be achieved in most scenes only through configuration, and the portability and flexibility of configuration are effectively improved.
The data analysis and calculation system can rapidly complete the application of real-time calculation, and can play a good role in small-scale service nodes or large-scale distributed clusters.
Small-scale service nodes behave as: under the 8-thread 16GB memory virtual machine, more than 15000 times per second of real-time computation can be completed under the scale of about 1000 ten thousand difference values, and 99% of computation response time is ensured to be completed within 50 milliseconds.
Large-scale distributed clusters behave as: under a 21-station 32-thread 64GB virtual machine, more than 20 ten thousand real-time calculations per second can be completed on the scale of about 4 hundred million difference values, and 99% of calculation response time is ensured to be completed within 50 milliseconds.
The initial calculation formula and the convergence formula referenced by each calculation task are different because the operator templates invoked each time are different.
Embodiment one:
the calculation task is configured as 'last 5 minutes login failure times', and the calculation process of the calculation task comprises the following steps:
the filtering module 4 screens the data in the writing instruction transmitted by the service application 2, if the field of the login success mark is not available or the value of the login success mark is failed, the calculation condition is not met, the process is terminated and the process returns. And otherwise, performing subsequent processes.
The extracting unit 51 extracts necessary writing data fields, such as a login account number, a login success flag, and a login time as calculation parameters, from the complete streaming data according to the writing instruction.
The first calculation unit 52 performs a process of reading the intermediate calculation data of the current minute according to the operator template of the number of times, and writing the result into the storage after adding up the current result, namely completing the writing calculation process, wherein the fine granularity is the minute in the process.
When reading, the second computing unit 53 queries the query condition generated according to the last 5 minutes in the computing task, queries all intermediate data in the last 5 minutes, and performs final aggregation and accumulation, and the final value is the configured value of the "last 5 minutes login failure times", namely the final data.
Embodiment two:
the calculation task is configured with a 'last 24 hours transaction amount variance', and the calculation process of the calculation task comprises the following steps:
the filtering module 4 filters the data in the writing instruction transmitted by the business application 2, if the data type is not the transaction type, or the transaction amount field is not available, or the value of the transaction amount is not the digital type, the calculation condition is not met, the process is terminated, and the process returns. And otherwise, performing subsequent processes.
The extracting unit 51 extracts "transaction type", "transaction amount" as a calculation parameter from the complete stream data according to the writing instruction.
The first calculation unit 52 performs the calculation using the incremental calculation mode to avoid the need to acquire all value calculations each time. And according to the variance and the average value of the current value and the previous value in each calculation, storing the accumulated variance result, replacing the accumulated variance result with the total variance result, wherein the variance result is intermediate data, and the fine granularity is hour in the flow.
In reading, the second calculation unit 53 inquires all variance result values within the last 24 hours, and then sums all variances according to a formula according to an incremental variance similar principle to obtain a total variance result.
In this embodiment, referring to fig. 1, the write request includes an aggregation field and a calculation target of the streaming data, the read request includes an aggregation field, the write condition includes a field condition and a constraint condition, and the read condition includes a field condition;
the filter module 4 includes:
a parsing unit 41, configured to parse the write request to obtain an aggregation field and data content, and parse the read request to obtain an aggregation field;
a generating unit 42, a connection parsing unit 41, for generating a write instruction when the aggregation field in the write request satisfies the field condition and the calculation target satisfies the constraint condition, and generating a read instruction when the aggregation field in the read request satisfies the field condition.
In the first embodiment, the aggregation condition is that the aggregation field includes a "login success flag" field, and the constraint is that the value of the "login success flag" is not "failure"; in the second embodiment, the aggregation condition is that the aggregation field includes a "transaction amount" field, the constraint is that the value of the "transaction amount" is a digital type and the data type is a "transaction" type.
The distributed real-time computing framework 1 and the business application 2 are integrated and run on nodes, a plurality of nodes are connected with a node task balancing module 6, and the node task balancing module 6 comprises:
a node statistics unit 61, configured to count the number of each node, and generate a task adjustment instruction when the number of nodes changes;
the task balancing unit 62 is connected to the node statistics unit 61, and is configured to aggregate the tasks on the current nodes into a task total according to the task adjustment instruction, and redistribute the task total according to the number of the current nodes.
Specifically, in the present embodiment, a lightweight centreless distributed cluster architecture is constructed. The centreless distributed cluster architecture may consist of several nodes, where the distributed real-time computing framework 1 and business applications 2 in this embodiment are running integrated on the nodes. There is no explicit central node in the distributed cluster architecture, and all nodes are equal. The node task balancing module 6 is utilized to realize the task allocation balance on each node, and the process is as follows: the node statistics unit 61 senses the change of the number of nodes in the distributed cluster architecture through network intercommunication, when the number of the nodes is increased, the node statistics unit 61 generates a task adjustment instruction and sends the task adjustment instruction to the task balancing unit 62, and the task balancing unit 62 gathers the tasks on the current nodes into the task total amount according to the task balancing instruction and redistributes the task total amount according to the number of the current nodes; similarly, when the number of nodes decreases, it indicates that there are nodes offline from the distributed cluster architecture, at this time, the node statistics unit 61 generates a task adjustment instruction, and sends the task adjustment instruction to the task balancing unit 62, and the task balancing unit 62 sums the tasks on each current node into a task total amount according to the task balancing instruction, and redistributes the task total amount according to the number of each current node.
In this embodiment, as shown with reference to fig. 1,
the calculation module 5 further includes:
the storage unit 54 stores a plurality of operator templates, and each operator template is matched with the type of the corresponding aggregation field;
the matching unit 55 is respectively connected with the storage unit 54 and the first computing unit 52, and is configured to match the types of the aggregation fields in the storage unit 54 to obtain corresponding operator templates.
Preferably, the fine particle size is a temporal particle size.
Preferably, the distributed real-time computing framework 1 is configured with a read interface and a write interface, and the distributed real-time computing framework 1 obtains a read request through the read interface and obtains a write request through the write interface.
Specifically, in this embodiment, the data interaction between the distributed real-time computing framework 1 and the service application 2 is directly implemented through the read interface and the write interface, so that no additional scheduling service is required to be configured, and the deployment cost of distributed computing is reduced.
The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the present invention may occur to one skilled in the art without departing from the principles of the present invention and are intended to be within the scope of the present invention.
Claims (6)
1. A data analysis computing system based on a distributed real-time computing framework, comprising a distributed real-time computing framework and a business application, wherein the distributed real-time computing framework is connected with the business application and comprises:
the request module is used for generating a writing request and a reading request according to the streaming data of the business application;
the filtering module is connected with the request module and is used for generating a writing instruction when the writing request meets preset writing conditions and generating a reading instruction when the reading request meets preset reading conditions;
the calculation module is connected with the filtering module and comprises:
the extraction unit is used for extracting a plurality of calculation parameters from the streaming data according to the type of the aggregation field after receiving the writing instruction;
the first calculation unit is connected with the extraction unit and is used for substituting each calculation parameter into a primary calculation formula in the operator template to obtain a plurality of primary data, and dividing each primary data into a plurality of intermediate data according to fine granularity;
the second calculation unit is connected with the first calculation unit and is used for screening each intermediate data according to a preset screening condition in the operator template to obtain a plurality of screening data after receiving the reading instruction, and substituting each screening data into a preset convergence formula to obtain final data.
2. The distributed real-time computing framework based data analysis computing system of claim 1, wherein: the write-in request comprises an aggregation field and a calculation target of the streaming data, the read-in request comprises the aggregation field, the write-in condition comprises a field condition and a limiting condition, and the read condition comprises a field condition;
the filter module includes:
the analysis unit is used for analyzing the write-in request to obtain the aggregation field and the data content, and analyzing the read-out request to obtain the aggregation field;
the generation unit is connected with the analysis unit and is used for generating the writing instruction when the aggregation field in the writing request meets the field condition and the calculation target meets the limiting condition, and generating the reading instruction when the aggregation field in the reading request meets the field condition.
3. The distributed real-time computing framework based data analysis computing system of claim 1, wherein: the distributed real-time computing framework and the business application are integrated and run on nodes, a plurality of the nodes are connected with a node task balancing module, and the node task balancing module comprises: the node counting unit is used for counting the number of the nodes and generating a task adjustment instruction when the number of the nodes changes;
and the task balancing unit is connected with the node counting unit and is used for summarizing the tasks on the current nodes into the total tasks according to the task adjustment instructions and redistributing the total tasks according to the number of the current nodes.
4. The distributed real-time computing framework based data analysis computing system of claim 1, wherein: the computing module further includes:
the storage unit is used for storing a plurality of operator templates, and each operator template is matched with the corresponding type of the aggregation field;
and the matching unit is respectively connected with the storage unit and the first computing unit and is used for matching the types of the aggregation fields in the storage unit to obtain the corresponding operator templates.
5. The distributed real-time computing framework based data analysis computing system of claim 1, wherein: the fine granularity is a temporal granularity.
6. The distributed real-time computing framework based data analysis computing system of claim 1, wherein: the distributed real-time computing framework is provided with a reading interface and a writing interface, and the distributed real-time computing framework obtains the reading request through the reading interface and obtains the writing request through the writing interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311109401.6A CN116841752B (en) | 2023-08-31 | 2023-08-31 | Data analysis and calculation system based on distributed real-time calculation framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311109401.6A CN116841752B (en) | 2023-08-31 | 2023-08-31 | Data analysis and calculation system based on distributed real-time calculation framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116841752A true CN116841752A (en) | 2023-10-03 |
| CN116841752B CN116841752B (en) | 2023-11-07 |
Family
ID=88174633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311109401.6A Active CN116841752B (en) | 2023-08-31 | 2023-08-31 | Data analysis and calculation system based on distributed real-time calculation framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116841752B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060002411A1 (en) * | 2001-02-21 | 2006-01-05 | Samrao Gurmohan S | Method and apparatus for transmitting streamed ingressing data through a switch fabric that provides read requests at an ingress independent request rate |
| US20060236395A1 (en) * | 2004-09-30 | 2006-10-19 | David Barker | System and method for conducting surveillance on a distributed network |
| CN107329982A (en) * | 2017-06-01 | 2017-11-07 | 华南理工大学 | A kind of big data parallel calculating method stored based on distributed column and system |
| CN108763527A (en) * | 2018-05-31 | 2018-11-06 | 北京明朝万达科技股份有限公司 | A kind of searching method and device of business datum |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN112328620A (en) * | 2020-11-05 | 2021-02-05 | 浪潮云信息技术股份公司 | Distributed database query acceleration method |
| CN115525192A (en) * | 2022-09-16 | 2022-12-27 | 平安健康保险股份有限公司 | User-oriented quotation charging method and device, computer equipment and storage medium |
| CN116150200A (en) * | 2022-11-16 | 2023-05-23 | 马上消费金融股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN116483319A (en) * | 2022-12-27 | 2023-07-25 | 清华大学 | Operator processing method, device, equipment and medium for software defined chip |
-
2023
- 2023-08-31 CN CN202311109401.6A patent/CN116841752B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060002411A1 (en) * | 2001-02-21 | 2006-01-05 | Samrao Gurmohan S | Method and apparatus for transmitting streamed ingressing data through a switch fabric that provides read requests at an ingress independent request rate |
| US20060236395A1 (en) * | 2004-09-30 | 2006-10-19 | David Barker | System and method for conducting surveillance on a distributed network |
| CN107329982A (en) * | 2017-06-01 | 2017-11-07 | 华南理工大学 | A kind of big data parallel calculating method stored based on distributed column and system |
| CN108763527A (en) * | 2018-05-31 | 2018-11-06 | 北京明朝万达科技股份有限公司 | A kind of searching method and device of business datum |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN112328620A (en) * | 2020-11-05 | 2021-02-05 | 浪潮云信息技术股份公司 | Distributed database query acceleration method |
| CN115525192A (en) * | 2022-09-16 | 2022-12-27 | 平安健康保险股份有限公司 | User-oriented quotation charging method and device, computer equipment and storage medium |
| CN116150200A (en) * | 2022-11-16 | 2023-05-23 | 马上消费金融股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN116483319A (en) * | 2022-12-27 | 2023-07-25 | 清华大学 | Operator processing method, device, equipment and medium for software defined chip |
Non-Patent Citations (2)
| Title |
|---|
| 梁毅;侯颖;陈诚;金翊;: "面向大数据流式计算的任务管理技术综述", 计算机工程与科学, no. 02 * |
| 樊春美;朱建生;单杏花;杨立鹏;李雯;: "基于Flink实时计算的自动化流控制算法", 计算机技术与发展, no. 08 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116841752B (en) | 2023-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108847989B (en) | Log processing method based on micro-service architecture, service system and electronic equipment | |
| CN105426292B (en) | A kind of games log real time processing system and method | |
| CN109725899B (en) | Data stream processing method and device | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| Logothetis et al. | In-situ {MapReduce} for Log Processing | |
| CN111355606B (en) | Web application-oriented container cluster self-adaptive expansion and contraction system and method | |
| CN109033109B (en) | Data processing method and system | |
| CN106982356B (en) | Distributed large-scale video stream processing system | |
| CN106354876A (en) | Data processing system and method | |
| CN103617287A (en) | Log management method and device in distributed environment | |
| CN111221831B (en) | Computing system for processing advertisement effect data in real time | |
| CN113448812A (en) | Monitoring alarm method and device under micro-service scene | |
| CN103927314A (en) | Data batch processing method and device | |
| US20200183913A1 (en) | Dedicated audit port for implementing recoverability in outputting audit data | |
| CN106326461A (en) | Real time processing guarantee method and system based on network signaling record | |
| CN107506381A (en) | A kind of big data distributed scheduling analysis method, system and device and storage medium | |
| CN112631754A (en) | Data processing method, data processing device, storage medium and electronic device | |
| CN108390793A (en) | A kind of method and device of analysis system stability | |
| CN113141410A (en) | Dynamically adjusted QPS control method, system, device and storage medium | |
| CN116841752B (en) | Data analysis and calculation system based on distributed real-time calculation framework | |
| CN106372171A (en) | Real-time data processing method of monitoring platform | |
| CN111124310B (en) | Storage system scheduling optimization method and related components | |
| CN113204464A (en) | Real-time service monitoring method, system, terminal and medium based on service scene | |
| CN112732242A (en) | Wide table processing script generation method and device | |
| CN111752541A (en) | Pay routing method based on Rete algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |