CN113239081A - Streaming data calculation method - Google Patents
Streaming data calculation method Download PDFInfo
- Publication number
- CN113239081A CN113239081A CN202110563865.9A CN202110563865A CN113239081A CN 113239081 A CN113239081 A CN 113239081A CN 202110563865 A CN202110563865 A CN 202110563865A CN 113239081 A CN113239081 A CN 113239081A
- Authority
- CN
- China
- Prior art keywords
- user
- keywords
- nodes
- directory list
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 66
- 230000000007 visual effect Effects 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims description 51
- 238000003860 storage Methods 0.000 claims description 37
- 238000000034 method Methods 0.000 claims description 18
- 238000012800 visualization Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000010223 real-time analysis Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
Abstract
The application provides a streaming data calculation method, which comprises the following steps: storing the title names corresponding to the data sources into root nodes according to Excel format detail data sources with specific attribute information, and storing specific index fields under each title name into corresponding sub-nodes; reading the title names stored in the root nodes to obtain various keywords in the title names; calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among a plurality of index fields; traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to the sub-nodes, and generating a directory list; editing files in various key words and index fields, and storing the edited latest directory list structure in Excel; and responding to a reading instruction issued by a user, acquiring the selected value of the target list, and displaying the selected value through the visual service terminal.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a streaming data calculation method.
Background
Real-time Streaming analysis (real time Streaming analysis), which is widely used in the field of big data in recent years, has three characteristics: infinite data, unbounded data processing, low latency, wherein infinite data refers to a growing, substantially infinite set of data; unbounded data processing refers to a continuous data processing mode, which can repeatedly process the above unbounded data through a processing engine, and can break through the bottleneck of a limited data processing engine; low latency means that the data is not well defined, the value of the data will decrease over time, and timeliness will be a problem to be solved continuously.
The streaming data is mainly applied to financial services, network monitoring, telecommunication data management, Web application, production manufacturing, sensing detection and the like, in a data flow model, because individual data are related tuples, such as data generated by network measurement, call recording, webpage access and the like, the data continuously reach in a large amount of rapid time-varying data flow forms, while the existing data flow model cannot meet the requirement of real-time calculation of the streaming data, and meanwhile, the persistent data flow model does not have the function of displaying a visual calculation result and cannot perform real-time analysis and traceability on the data flow in a targeted manner.
Disclosure of Invention
The invention aims to provide a streaming data calculation method, which realizes real-time calculation and visual display of streaming data through a visual service terminal.
The invention is realized by the following steps: a method of streaming data computation, the method comprising: processing the text data according to the data processing strategy corresponding to the processing strategy identifier selected by the user to obtain an Excel format detail data source with specific attribute information, wherein the specific attribute information comprises: title name and index fields;
storing the title names into a root node, and storing specific index fields under each title name into corresponding sub-nodes;
responding to a selection instruction issued by a user, reading the title names stored in the root nodes to obtain various keywords in the title names;
calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among the attribute values;
traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to the sub-nodes and generating a directory list, wherein the sub-nodes comprise: a left child node and a right child node;
editing files in the various key words and the index fields aiming at the directory list, and storing an edited latest directory list structure in an Excel format;
and responding to a reading instruction issued by a user, acquiring the selected value of the target list, and displaying the selected value in the visual service terminal in a tree structure form.
In a preferred technical solution of the present invention, processing text data according to a data processing policy corresponding to a processing policy identifier selected by a user to obtain an Excel format detail data source with specific attribute information includes:
responding to the processing strategy identifier of the ETL tool selected by a user, and executing a data processing strategy corresponding to the processing strategy identifier on the text data to obtain the Excel format detail data source with the title name and the index field.
In a preferred technical solution of the present invention, storing the title name in a root node, and storing a specific indicator field under each title name in a corresponding sub-node includes:
storing the title names in the root nodes to an inner layer in a group of storage units with continuous addresses according to the position of the sequence;
and sequentially storing the index fields corresponding to the sub-nodes in an outer layer connected by the storage unit from left to right, wherein the sub-nodes comprise a left sub-node and a right sub-node.
In a preferred technical solution of the present invention, reading the title name stored in the root node in response to a selection instruction issued by a user to obtain multiple types of keywords in the title name, includes:
randomly selecting at least one of the multiple kinds of keywords in the title name to generate a statistical path of the title name, wherein the multiple kinds of keywords comprise: time, attribute name, data value, region;
and establishing internal association for the selected multiple types of keywords through addresses to obtain multiple attribute values of the multiple types of keywords.
In a preferred technical solution of the present invention, calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among the plurality of attribute values includes:
establishing address association of the various keywords and the attribute values, and moving the address of the attribute value according to the latest mouse position issued by a user after the address association;
selecting a plurality of attribute values under the various types of keywords, and performing combined association on addresses of the attribute values;
and calculating the attribute values after the combination according to a function calculation formula selected by a user to obtain a calculation result among the index fields.
In a preferred embodiment of the present invention, traversing multiple types of keywords corresponding to the root node, and sequentially traversing index fields corresponding to multiple child nodes to generate a directory list, includes:
adding various key words corresponding to the traversed root nodes into a message queue according to a hierarchy according to a hierarchical tree structure;
and verifying whether the attribute value is a file or not aiming at the attribute value corresponding to each child node, if so, adding the attribute value to a message queue, and generating a directory list.
In a preferred embodiment of the present invention, the editing operation is performed on the files in the various kinds of keywords and the index fields with respect to the directory list, and the latest directory list structure edited is stored in an Excel format, including:
establishing a directory list according to a tree structure, and performing increasing operation on various key words and index fields stored in the target category;
and storing the added new directory lists of the root nodes corresponding to the various key words and the sub nodes corresponding to the index fields into a file in an Excel format.
In a preferred technical solution of the present invention, the method for obtaining the selected value of the directory list in response to a reading instruction issued by a user and displaying the value in the visual service terminal in a form of a tree structure includes:
aiming at the tree structure directory list, the visualization service terminal pushes visualization success information of a calculation result to a front-end access page;
and responding to a reading instruction sent by a user, acquiring the width and height selected values of the dynamic list, and displaying the values on a visual service terminal with a tree structure.
In a preferred embodiment of the present invention, an electronic device includes: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating via the bus when the electronic device is operating, the processor executing the machine-readable instructions to perform the steps of the streaming data calculation method according to any of the claims.
In a preferred embodiment of the present invention, a computer-readable storage medium has a computer program stored thereon, which, when executed by a processor, performs the steps of streaming data calculation according to any one of the claims.
The invention has the following beneficial effects: according to an Excel format detail data source with specific attribute information, storing title names corresponding to the data source into a root node, and storing specific index fields under each title name into corresponding sub-nodes; responding to a selection instruction issued by a user, reading the title names stored in the root nodes to obtain various keywords in the title names; calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among a plurality of index fields; traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to a plurality of sub-nodes and generating a directory list, wherein the sub-nodes comprise: a left child node and a right child node; editing files in various key words and index fields aiming at a directory list, and storing an edited latest directory list structure in an Excel format; responding to a reading instruction issued by a user, acquiring a selected value of a target list, and displaying the selected value on a visual service terminal in a tree structure form; specifically, specific attribute information corresponding to a detailed data source in an Excel format is read, the attribute information can be industrial data of financial services, network monitoring, telecommunication data management, Web application and production and manufacture, the industrial data is imported into a visual service terminal in the Excel format according to the industrial data stream report data, corresponding functions are called by using formulas for calculation aiming at various keywords corresponding to the specific attribute information and attribute values under the various keywords, calculation results among the attribute values are obtained, the calculation results are stored in the Excel format, a directory list in the Excel format with a hierarchical tree structure is displayed in a page browsing mode, the visual service terminal can instantly calculate streaming data, the visual service terminal is clear, quick and visual in hierarchy, the visual service terminal is a distributed cache Redis, in addition, the visual service terminal can gather valuable different kinds of data together, the data calculation can be carried out without a predefined model, the calculation architecture is extensible, real-time analysis can be provided for different clients, the application range is wide, and the use value is high.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a streaming data calculation method according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a statistical path of title names in step S301 of a streaming data calculation method according to an embodiment of the present application.
Fig. 3 is a schematic diagram of attribute values in step S302 of a streaming data calculation method according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a streaming data computing apparatus according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations, and therefore, the following detailed description of the embodiments of the present invention provided in the figures is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The streaming data at present is characterized in that the data can be continuously collected from various places, the sources are numerous, the format is complex, the data volume is huge, and the value of the streaming data is reduced along with the lapse of time, so that the streaming data can be processed as fast as possible after being generated, and the calculation result is responded in real time instead of being processed regularly after the data is accumulated.
In the prior art, the exogenous data flow-based calculation is that an analysis interface is called to retrieve an algorithm expression from an algorithm library file, then the algorithm is called through the algorithm expression, and finally the value of a corresponding data flow is calculated, so that the calculation efficiency is low.
In recent years, as streaming data is widely used, a persistent data flow model cannot meet data flow calculation of a large amount of data, an individual data flow unit is a related tuple, such as data generated by network measurement, call recording, webpage access and the like, the data continuously reaches in a large, quick and time-varying data flow form, a current-stage data flow model cannot meet the requirement of real-time calculation of the streaming data, and meanwhile, the persistent data flow model does not have a function of displaying a visual calculation result and cannot perform real-time analysis and traceability on the data flow in a targeted manner.
Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
In a possible implementation, fig. 1 is a schematic flow chart of a streaming data calculation method provided in an embodiment of the present invention; as shown in fig. 1, the data calculation in the method specifically includes the following steps:
step S10, processing the text data according to the data processing policy corresponding to the processing policy identifier selected by the user, to obtain an Excel format detail data source having specific attribute information, where the specific attribute information includes: title name and index fields.
Step S20, storing the title name into the root node, and storing the specific index field under each title name into the corresponding sub-node.
And step S30, responding to the selection instruction given by the user, reading the title name stored in the root node, and obtaining various keywords in the title name.
And step S40, calculating a plurality of attribute values under the various keywords selected by the user according to the formula selected by the user to obtain a calculation result among a plurality of index fields.
Step S50, traversing multiple kinds of keywords corresponding to the root node, and traversing index fields corresponding to multiple sub-nodes in sequence to generate a directory list, wherein the sub-nodes comprise: a left child node and a right child node.
And step S60, editing files in the various keywords and the index fields aiming at the directory list, and storing the latest edited directory list structure in an Excel format.
And step S70, responding to the reading instruction given by the user, acquiring the selected value of the target list, and displaying the value in the visual service terminal in a tree structure form.
Step S10, when implementing specifically, using an ETL (Extract-Transform-Load) data warehouse technology, a user selects a processing policy identifier in an ETL tool, generates a data processing policy for processing text data, and obtains an Excel format detail data source with a title name and an index field; wherein processing the policy identifier comprises: input and output, data cleaning, sorting, duplication removal, association combination and grouping statistics.
Step S20 is to establish a configuration file, create a storage unit of the title name corresponding to the Excel format detail data source through the configuration file, store the title name in the corresponding root node, store the index field in the corresponding child node, and enter the title name and the text data in the index field according to the Excel format specification.
Step S30, when the method is specifically implemented, reads the title name stored in the root node according to the configuration file logic, and obtains multiple types of keywords in the title name, where the multiple types of keywords include: time, developer, sale amount and region, and various keywords are used as new data sources.
In the specific implementation of step S40, a formula editor is used, a user selects a calculation formula, and a plurality of attribute values corresponding to a plurality of types of keywords selected by the user are calculated according to the calculation formula selected by the user, so as to obtain a calculation result of the attribute values.
In the specific implementation of step S50, traversing the depth, height, and width of the multiple keywords corresponding to the root node, calculating the coordinate value of the current root node, sequentially traversing the index fields corresponding to the multiple child nodes, where the multiple child nodes are nodes on the same layer, and sequentially storing from left to right to generate the target list.
For example: if the current child node and the previous child node belong to the same layer node and belong to the same father node, calculating the coordinate value of the current node; if the current child node and the previous child node belong to the same layer node but do not belong to the same father node as the previous child node, respectively calculating father node coordinate values of the current child node and the previous child node, and storing the father node coordinate values in the center position of the father node; if the current child node and the previous child node do not belong to the same layer node, calculating the coordinate value of the current child node and storing the coordinate value of the current child node in the central position of the previous father node.
In the specific implementation of step S60, according to the tree structure, multiple types of keywords and index subsections stored in the directory list are selected, and the multiple types of keywords and index subsections are subjected to addition, modification, and editing processing to obtain a new directory list, which is stored in an Excel format.
In the specific implementation of step S70, after receiving the reading instruction issued by the user, the visual service terminal obtains the width value and the height value of the latest target list, and displays the width value and the height value through the visual service terminal.
In a possible implementation scheme, in step S10, processing the text data according to the data processing policy corresponding to the processing policy identifier selected by the user to obtain an Excel format detail data source with specific attribute information includes:
step 101, responding to a processing strategy identifier of an ETL tool selected by a user, executing a data processing strategy corresponding to the processing strategy identifier on text data, and obtaining an Excel format detail data source with a title name and an index field.
In the specific implementation of step 101, a user uses an ETL tool to sequentially select and process input, sorting, duplicate removal, grouping statistics, association merging and output under a policy identifier, the ETL tool automatically performs a data processing policy on the acquired source data, and the detailed data source with a specific title name and an index field Excel format is obtained after processing.
In one possible implementation, the step S20 of storing the title names in the root node and storing the specific indicator field under each title name in the corresponding sub-node includes:
step 201, storing the title names in the root node into an inner layer of a group of storage units with continuous addresses according to the position of the sequence.
Step 202, index fields corresponding to the sub-nodes are sequentially stored in an outer layer connected by the storage units from left to right, and the sub-nodes comprise left sub-nodes and right sub-nodes.
In the specific implementation of step 201, according to the organization frame of the tree structure, the root node stores the storage units with consecutive addresses in a chain storage or sequential storage manner.
In specific implementation, in step 202, according to an organization structure of the tree structure, the left child node and the right child node in the child nodes are sequentially stored in the outer layer of the storage unit from left to right in a chain storage or sequential storage manner.
For example: a chain storage mode: the right data area of the root node and two addresses are formed, the title name corresponding to the root node is stored in the data area, and the two addresses are respectively used for storing the left child node and the right child node; a sequential storage mode: and storing the title names corresponding to the root nodes in a storage unit in an array form, wherein the root nodes are stored in a first position, the left child nodes are stored in a second position, and the right child nodes are stored in a third position.
In a possible implementation, fig. 2 is a schematic diagram of a statistical path of a title name of a streaming data calculation method according to an embodiment of the present invention.
In a possible implementation, in step S30, in response to the selection instruction issued by the user, the title names stored in the root node are read to obtain various keywords in the title names, including:
step S301, at least one multi-type keyword in the title name is randomly selected, and a statistical path of the title name is generated, wherein the multi-type keyword comprises: time, attribute name, data value, region.
Step S302, establishing internal association of the selected various keywords through addresses to obtain a plurality of attribute values of the various keywords.
In the specific implementation of step S301, according to a selection instruction issued by a user, the corresponding multiple types of keywords in the storage unit are read, and according to at least one of the randomly selected time, attribute name, data value, and area, a statistical path of the title name is generated by the randomly selected time, attribute name, data value, and corresponding address of the area.
As shown in fig. 2, the statistical path for randomly reading the title name includes:
statistical path 1 may be time.
Statistical path 2 may be a temporal, regional component.
The statistical path 3 may be time, attribute name, data value, area.
In a possible implementation, fig. 3 is a schematic diagram of attribute values of a streaming data calculation method according to an embodiment of the present invention.
In the specific implementation of step S302, index fields under the title name are randomly combined through address association index fields according to the statistical path, and a plurality of attribute values are obtained after the combination.
As shown in fig. 3, obtaining a plurality of attribute values of a plurality of types of keywords according to a random combination of statistical route 1, statistical route 2, or statistical route 3 includes:
the attribute value 1 may be a combination of statistical path 1 and statistical path 2.
The attribute value 2 may be a combination of attribute value 1 and statistical path 3.
The attribute value 3 may be a combination of the attribute value 2 and the attribute value 1.
In a possible implementation, in the step S40, calculating, according to the formula selected by the user, a plurality of attribute values under the plurality of types of keywords selected by the user to obtain a calculation result between the plurality of attribute values, includes:
step 401, establishing address association of various keywords and attribute values, and moving the address of the attribute value according to the latest mouse position issued by the user after the association.
Step 402, selecting a plurality of attribute values under the various keywords, and performing combined association on addresses of the attribute values.
And 403, calculating the multiple combined attribute values according to a function calculation formula selected by the user to obtain a calculation result among the index fields.
Step 401 is implemented specifically, the formula editor associates the data areas of the various keywords and the attribute values through addresses, captures the position of the mouse according to the selection instruction of the user, moves the address of the data area corresponding to the attribute value, and executes the dragging instruction issued by the user.
In step 402, when the method is implemented, the formula editor selects an attribute value from a plurality of keywords, and randomly combines and associates the plurality of attribute values through addresses.
In step 403, in a specific implementation, the combined attribute values are calculated according to a calculation formula selected by a user by using an operator built in a formula editor to call a corresponding function, so as to obtain a calculation result between the attribute values, and the calculation result is stored in an Excel format.
In a possible implementation scheme, in step S50, traversing multiple types of keywords corresponding to the root node, and sequentially traversing index fields corresponding to multiple child nodes, to generate a directory list, including:
step 501, adding various keywords corresponding to the traversed root node into a message queue according to the hierarchy.
Step 502, verifying whether the attribute value is a file or not according to the attribute value corresponding to each child node, and if so, adding the file to a message queue to generate a directory list.
Step 501, when the method is specifically implemented, a linked list is created by using a canvas, line data of various key words are read, added to a message queue according to a hierarchy, and displayed in a tree structure.
In the specific implementation of step 502, for each attribute value, the files stored in the sub-node corresponding to each attribute value are traversed one by one, and if the sub-node is not empty, the files stored in the sub-node are added to the message queue to generate the directory list.
For example: traversing the child nodes corresponding to each attribute value, and if the current child node and the previous child node belong to nodes on the same layer and belong to the same father node, calculating the coordinate value of the current node; if the current child node and the previous child node belong to the same layer node but do not belong to the same father node as the previous child node, respectively calculating father node coordinate values of the current child node and the previous child node, and storing the father node coordinate values in the center position of the father node; if the current child node and the previous child node do not belong to the same layer node, calculating the coordinate value of the current child node and storing the coordinate value of the current child node in the central position of the previous father node.
In one possible implementation, in step S60, editing files in various types of keywords and index fields for the directory list, and storing the latest directory list structure edited in an Excel format includes:
step 601, establishing a directory list according to a tree structure, and performing addition operation on multiple types of keywords and index fields stored in the target type.
Step 602, the root nodes corresponding to the added various keywords and the sub-nodes corresponding to the index fields are stored in an Excel format.
Step 601, during specific implementation, according to the tree structure, the incidence relation between the root node and the sub-node in the directory list is established, and the adding operation is executed on the files of the multiple types of keywords and the index fields stored in the target type.
In the specific implementation of step 602, various types of keywords and index fields added in the directory list are written back into the Excel format through the search box.
In a possible implementation, in step S70, in response to a reading instruction issued by a user, obtaining a selected value of the directory list, and displaying the value in the visualization service terminal in the form of a tree structure, the method includes:
step 701, aiming at the tree structure directory list, the visualization service terminal pushes visualization success information of the calculation result to the front-end access page.
And step 702, responding to a reading instruction sent by a user, acquiring a width value and a height value of the dynamic list, and displaying the width value and the height value on the visual service terminal.
In the specific implementation of step 701, after the root node and the child node corresponding to the directory list are calculated and updated according to the tree structure export function, the visualization service terminal pushes a success notification message.
In specific implementation, the step 702 responds to a reading instruction of a user, and converts a width value and a height value corresponding to a directory list into a graph by using a canvas function, wherein the graph is displayed by a visual service terminal and is used for displaying texts of various key words and index fields stored in the directory list.
Fig. 4 is a schematic structural diagram of a streaming data computing apparatus according to an embodiment of the present application, where the apparatus includes:
a data source module 801, configured to process text data according to a data processing policy corresponding to the processing policy identifier selected by the user, to obtain an Excel format detail data source with specific attribute information, where the specific attribute information includes: title name and index fields;
a storage module 802, configured to store the title names in the root node, and store the specific indicator field under each title name in the corresponding sub-node;
the reading module 803 is configured to respond to a selection instruction issued by a user, read the title names stored in the root nodes, and obtain multiple types of keywords in the title names;
the calculating module 804 is configured to calculate, according to the formula selected by the user, a plurality of attribute values under the plurality of types of keywords selected by the user to obtain a calculation result among the plurality of attribute values;
the list module 805 is configured to establish a directory list module, traverse multiple types of keywords corresponding to the root node, and sequentially traverse index fields corresponding to multiple child nodes to generate a directory list, where the child nodes include: a left child node and a right child node;
the editing module 806 is configured to edit files in the multiple types of keywords and the index fields for the directory list, and store an edited latest directory list structure in an Excel format;
the display module 807 is configured to respond to a reading instruction issued by the user, obtain a selected value of the target list, and display the selected value in the visual service terminal in a form of a tree structure.
In one possible implementation, the data source module 801 includes:
and the reading unit is used for responding to the processing strategy identifier of the ETL tool selected by the user, executing the data processing strategy corresponding to the processing strategy identifier on the text data, and obtaining the Excel format detail data source with the title name and the index field.
In one possible implementation, the storage module 802 includes:
the root node storage unit is used for storing the title names in the root nodes into an inner layer of a group of storage units with continuous addresses according to the position of the sequence;
and the subnode storage unit is used for sequentially storing the index fields corresponding to the subnodes from left to right and comprises a left subnode and a right subnode on the outer layer connected by the storage unit.
In one possible implementation, the reading module 803 includes:
a title name reading unit for randomly selecting at least one of multiple kinds of keywords in the title name, the multiple kinds of keywords including: time, attribute name, data value, region;
and the attribute value unit is used for establishing internal association of the selected multiple types of keywords through addresses to obtain multiple attribute values of the multiple types of keywords.
In one possible implementation, the calculation module 804 includes:
the attribute value address association unit is used for establishing address association of various keywords and attribute values and moving the address of the attribute value according to the latest mouse position issued by a user after the association;
the attribute value merging unit is used for selecting a plurality of attribute values under various keywords and carrying out combined association on addresses of the attribute values;
and the attribute value calculating unit is used for calculating the plurality of combined attribute values according to the function calculation formula selected by the user to obtain a calculation result among the index fields.
In one possible implementation, the list module 805 includes:
the adding queue unit is used for adding various key words corresponding to the root node into the message queue according to the hierarchy according to the hierarchical tree structure;
and the directory list generation unit is used for verifying whether the attribute value is a file or not according to the attribute value corresponding to each child node, and if so, adding the attribute value to the message queue to generate a directory list.
In one possible implementation, the editing module 806 includes:
the directory list adding unit is used for establishing a directory list according to a tree structure and adding various key words and index fields stored in the target list;
and the new directory list unit is used for storing the added new directory lists of the root nodes corresponding to the various key words and the sub nodes corresponding to the index fields into the file in the Excel format.
In one possible implementation, the display module 807 includes:
the visualization message unit is used for pushing visualization success messages of the calculation results to the front-end access page by the visualization service terminal aiming at the tree structure directory list;
and the visualization unit is used for responding to a reading instruction sent by a user, acquiring the width and height selected values of the dynamic list and displaying the values on the visualization service terminal.
Fig. 5 is a schematic structural diagram of an electronic device 90 according to an embodiment of the present application, including: a processor 901, a storage medium 902 and a bus 903, wherein the storage medium 902 stores machine readable instructions executable by the processor 901, when the electronic device executes the method for processing information, the processor 901 communicates with the storage medium 902 via the bus 903, and the processor 901 executes the machine readable instructions to execute the following steps:
processing the text data according to the data processing strategy corresponding to the processing strategy identifier selected by the user to obtain an Excel format detail data source with specific attribute information, wherein the specific attribute information comprises: title name and index fields;
storing the title names into a root node, and storing specific index fields under each title name into corresponding sub-nodes;
responding to a selection instruction issued by a user, reading the title names stored in the root nodes to obtain various keywords in the title names;
calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among the plurality of attribute values;
traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to a plurality of sub-nodes and generating a directory list, wherein the sub-nodes comprise: a left child node and a right child node;
editing files in various key words and index fields aiming at a directory list, and storing an edited latest directory list structure in an Excel format;
and responding to a reading instruction issued by a user, acquiring the selected value of the target list, and displaying the selected value in the visual service terminal in a tree structure form.
In the embodiment of the present application, the storage medium 902 may further execute other machine-readable instructions to perform other methods described in the present application, and for the specific steps and principles of the executed methods, reference is made to the above description, which is not repeated herein in detail.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program performs the following steps:
processing the text data according to the data processing strategy corresponding to the processing strategy identifier selected by the user to obtain an Excel format detail data source with specific attribute information, wherein the specific attribute information comprises: title name and index fields;
storing the title names into a root node, and storing specific index fields under each title name into corresponding sub-nodes;
responding to a selection instruction issued by a user, reading the title names stored in the root nodes to obtain various keywords in the title names;
calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among the plurality of attribute values;
traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to a plurality of sub-nodes and generating a directory list, wherein the sub-nodes comprise: a left child node and a right child node;
editing files in various key words and index fields aiming at a directory list, and storing an edited latest directory list structure in an Excel format;
and responding to a reading instruction issued by a user, acquiring the selected value of the target list, and displaying the selected value in the visual service terminal in a tree structure form.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of streaming data computation, the method comprising:
processing the text data according to the data processing strategy corresponding to the processing strategy identifier selected by the user to obtain an Excel format detail data source with specific attribute information, wherein the specific attribute information comprises: title name and index fields;
storing the title names into a root node, and storing specific index fields under each title name into corresponding sub-nodes;
responding to a selection instruction issued by a user, reading the title names stored in the root nodes to obtain various keywords in the title names;
calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result among the attribute values;
traversing various key words corresponding to the root node, sequentially traversing index fields corresponding to the sub-nodes and generating a directory list, wherein the sub-nodes comprise: a left child node and a right child node;
editing files in the various key words and the index fields aiming at the directory list, and storing an edited latest directory list structure in an Excel format;
and responding to a reading instruction issued by a user, acquiring the selected value of the directory list, and displaying the value at the visual service terminal in a tree structure form.
2. The streaming data calculation method according to claim 1, wherein the step of processing the text data according to the data processing policy corresponding to the processing policy identifier selected by the user to obtain an Excel format detail data source with specific attribute information comprises:
responding to the processing strategy identifier of the ETL tool selected by the user, and executing the data processing strategy corresponding to the processing strategy identifier on the text data to obtain the Excel format detail data source with the title name and the index field.
3. The streaming data calculation method of claim 1, wherein storing the title names in a root node and storing a specific indicator field under each title name in a corresponding sub-node comprises:
storing the title names in the root nodes into an inner layer of a group of storage units with continuous addresses according to the position of the sequence;
and sequentially storing the index fields corresponding to the sub-nodes in an outer layer connected by the storage unit from left to right, wherein the sub-nodes comprise a left sub-node and a right sub-node.
4. The streaming data calculating method according to claim 3, wherein reading the title names stored in the root node in response to a selection instruction issued by a user to obtain various keywords in the title names comprises:
randomly selecting at least one of the multiple kinds of keywords in the title name to generate a statistical path of the title name, wherein the multiple kinds of keywords comprise: time, attribute name, data value, region;
and establishing internal association for the selected multiple types of keywords through addresses to obtain multiple attribute values of the multiple types of keywords.
5. The streaming data calculation method according to claim 1, wherein calculating a plurality of attribute values under a plurality of types of keywords selected by a user according to a formula selected by the user to obtain a calculation result between the plurality of attribute values comprises:
establishing address association of the various keywords and the attribute values, and moving the address of the attribute value according to the latest mouse position issued by a user after the address association;
selecting a plurality of attribute values under the various types of keywords, and performing combined association on addresses of the attribute values;
and calculating the attribute values after the combination according to a function calculation formula selected by a user to obtain a calculation result among the index fields.
6. The streaming data calculation method according to claim 1, wherein traversing the plurality of types of keywords corresponding to the root node, and sequentially traversing the indicator fields corresponding to the plurality of sub-nodes to generate a directory list, comprises:
adding various key words corresponding to the root node into the message queue according to the hierarchy according to the hierarchical tree structure;
and verifying whether the attribute value is a file or not aiming at the attribute value corresponding to each child node, if so, adding the attribute value to a message queue, and generating a directory list.
7. The streaming data computing method according to claim 1, wherein, for the directory list, an editing operation is performed on the files in the multiple types of keywords and index fields, and an edited latest directory list structure is stored in an Excel format, and the method comprises:
establishing a directory list according to a tree structure, and performing increasing operation on various key words and index fields stored in the target category;
and storing the added new directory list of the root nodes corresponding to the various key words and the sub-nodes corresponding to the index fields into a file in an Excel format.
8. The streaming data calculation method according to claim 1, wherein the step of obtaining the selected value of the directory list in response to a reading command issued by a user and displaying the value in the form of a tree structure at the visual service terminal comprises:
aiming at the tree structure directory list, the visualization service terminal pushes visualization success information of a calculation result to a front-end access page;
and responding to a reading instruction sent by a user, acquiring the width and height selected values of the directory list, and displaying the values on a visual service terminal with a tree structure.
9. An electronic device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating via the bus when the electronic device is operating, the processor executing the machine-readable instructions to perform the steps of the streaming data calculation method according to any one of claims 1 to 8.
10. A computer-readable storage medium, having stored thereon a computer program for performing, when executed by a processor, the steps of the streaming data calculation method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110563865.9A CN113239081A (en) | 2021-05-21 | 2021-05-21 | Streaming data calculation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110563865.9A CN113239081A (en) | 2021-05-21 | 2021-05-21 | Streaming data calculation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113239081A true CN113239081A (en) | 2021-08-10 |
Family
ID=77138402
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110563865.9A Pending CN113239081A (en) | 2021-05-21 | 2021-05-21 | Streaming data calculation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113239081A (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100106710A1 (en) * | 2008-10-28 | 2010-04-29 | Hitachi, Ltd. | Stream data processing method and system |
| CN106250444A (en) * | 2016-07-27 | 2016-12-21 | 北京集奥聚合科技有限公司 | The real-time Input System of a kind of heterogeneous data source and method |
| CN106649773A (en) * | 2016-12-27 | 2017-05-10 | 北京大数有容科技有限公司 | Big data collaborative analysis tool platform |
| CN106778033A (en) * | 2017-01-10 | 2017-05-31 | 南京邮电大学 | A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms |
| US20170154088A1 (en) * | 2015-11-30 | 2017-06-01 | Tableau Software, Inc. | Systems and Methods for Interactive Visual Analysis Using A Specialized Virtual Machine |
| CN108170826A (en) * | 2018-01-08 | 2018-06-15 | 北京国信宏数科技有限责任公司 | A kind of macro economic analysis method and system based on internet big data |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN109522359A (en) * | 2018-11-02 | 2019-03-26 | 大连瀚闻资讯有限公司 | Visualization industrial analysis method based on big data |
| CN110119421A (en) * | 2019-04-03 | 2019-08-13 | 昆明理工大学 | A kind of electric power stealing user identification method based on Spark flow sorter |
| CN110704479A (en) * | 2019-09-12 | 2020-01-17 | 新华三大数据技术有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN111367989A (en) * | 2020-06-01 | 2020-07-03 | 北京江融信科技有限公司 | Real-time data index calculation system and method |
| CN111428458A (en) * | 2020-03-03 | 2020-07-17 | 中国平安人寿保险股份有限公司 | Universal report generation method and device and computer readable storage medium |
| CN112216347A (en) * | 2020-09-14 | 2021-01-12 | 苏州创腾软件有限公司 | Scientific data genome processing method, device and storage medium |
| US20210073279A1 (en) * | 2019-09-06 | 2021-03-11 | Tableau Software, Inc. | Using Natural Language Expressions to Define Data Visualization Calculations that Span Across Multiple Rows of Data from a Database |
| WO2021050182A1 (en) * | 2019-09-13 | 2021-03-18 | Tableau Software, Inc. | Utilizing appropriate measure aggregation for generating data visualizations of multi-fact datasets |
| CN112667683A (en) * | 2020-12-25 | 2021-04-16 | 平安科技（深圳）有限公司 | Stream computing system, electronic device and storage medium therefor |
-
2021
- 2021-05-21 CN CN202110563865.9A patent/CN113239081A/en active Pending
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100106710A1 (en) * | 2008-10-28 | 2010-04-29 | Hitachi, Ltd. | Stream data processing method and system |
| US20170154088A1 (en) * | 2015-11-30 | 2017-06-01 | Tableau Software, Inc. | Systems and Methods for Interactive Visual Analysis Using A Specialized Virtual Machine |
| CN106250444A (en) * | 2016-07-27 | 2016-12-21 | 北京集奥聚合科技有限公司 | The real-time Input System of a kind of heterogeneous data source and method |
| CN106649773A (en) * | 2016-12-27 | 2017-05-10 | 北京大数有容科技有限公司 | Big data collaborative analysis tool platform |
| CN106778033A (en) * | 2017-01-10 | 2017-05-31 | 南京邮电大学 | A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms |
| CN108170826A (en) * | 2018-01-08 | 2018-06-15 | 北京国信宏数科技有限责任公司 | A kind of macro economic analysis method and system based on internet big data |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN109522359A (en) * | 2018-11-02 | 2019-03-26 | 大连瀚闻资讯有限公司 | Visualization industrial analysis method based on big data |
| CN110119421A (en) * | 2019-04-03 | 2019-08-13 | 昆明理工大学 | A kind of electric power stealing user identification method based on Spark flow sorter |
| US20210073279A1 (en) * | 2019-09-06 | 2021-03-11 | Tableau Software, Inc. | Using Natural Language Expressions to Define Data Visualization Calculations that Span Across Multiple Rows of Data from a Database |
| CN110704479A (en) * | 2019-09-12 | 2020-01-17 | 新华三大数据技术有限公司 | Task processing method and device, electronic equipment and storage medium |
| WO2021050182A1 (en) * | 2019-09-13 | 2021-03-18 | Tableau Software, Inc. | Utilizing appropriate measure aggregation for generating data visualizations of multi-fact datasets |
| CN111428458A (en) * | 2020-03-03 | 2020-07-17 | 中国平安人寿保险股份有限公司 | Universal report generation method and device and computer readable storage medium |
| CN111367989A (en) * | 2020-06-01 | 2020-07-03 | 北京江融信科技有限公司 | Real-time data index calculation system and method |
| CN112216347A (en) * | 2020-09-14 | 2021-01-12 | 苏州创腾软件有限公司 | Scientific data genome processing method, device and storage medium |
| CN112667683A (en) * | 2020-12-25 | 2021-04-16 | 平安科技（深圳）有限公司 | Stream computing system, electronic device and storage medium therefor |
Non-Patent Citations (1)
| Title |
|---|
| 崔迪;郭小燕;陈为;: "大数据可视化的挑战与最新进展", 计算机应用, no. 07, 10 July 2017 (2017-07-10) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741089B1 (en) | Interactive location queries for raw machine data | |
| US10761687B2 (en) | User interface that facilitates node pinning for monitoring and analysis of performance in a computing environment | |
| US10205643B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with severity-state sorting | |
| US11880399B2 (en) | Data categorization using inverted indexes | |
| US20170264512A1 (en) | Systems and Methods For Monitoring And Analyzing Performance In A Computer System With State Distribution Ring | |
| CN109254901B (en) | A kind of Monitoring Indexes method and system | |
| US11106713B2 (en) | Sampling data using inverted indexes in response to grouping selection | |
| US11494395B2 (en) | Creating dashboards for viewing data in a data storage system based on natural language requests | |
| US10901811B2 (en) | Creating alerts associated with a data storage system based on natural language requests | |
| Zihayat et al. | Efficiently mining high utility sequential patterns in static and streaming data | |
| JP5490253B2 (en) | String aggregation method in numerical aggregation calculation | |
| Cao et al. | Timon: A timestamped event database for efficient telemetry data processing and analytics | |
| CN114218179A (en) | Mass log data tracing and storing device based on P2P technology | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN113239081A (en) | Streaming data calculation method | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN115168361A (en) | Label management method and device | |
| CN113434607A (en) | Behavior analysis method and device based on graph data, electronic equipment and storage medium | |
| US20190034555A1 (en) | Translating a natural language request to a domain specific language request based on multiple interpretation algorithms | |
| CN103093140A (en) | Method and system for managing authority | |
| CN110389965B (en) | Multidimensional data query and cache optimization method | |
| CN114265848A (en) | Data comparison retrieval method and device, electronic equipment and storage medium | |
| Ali et al. | A novel temporal frequent subgraph based mining algorithm using spark | |
| CN116975367A (en) | Data relationship processing method and device, electronic equipment and storage medium | |
| CN113157191A (en) | Data visualization method based on OLAP system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |