CN112448971B - Data analysis platform, data analysis method and storage medium - Google Patents
Data analysis platform, data analysis method and storage medium Download PDFInfo
- Publication number
- CN112448971B CN112448971B CN201910809468.8A CN201910809468A CN112448971B CN 112448971 B CN112448971 B CN 112448971B CN 201910809468 A CN201910809468 A CN 201910809468A CN 112448971 B CN112448971 B CN 112448971B
- Authority
- CN
- China
- Prior art keywords
- data
- analysis
- micro
- layer
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 71
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000004458 analytical method Methods 0.000 claims abstract description 99
- 238000012545 processing Methods 0.000 claims abstract description 11
- 230000005540 biological transmission Effects 0.000 claims abstract description 8
- 238000012795 verification Methods 0.000 claims description 11
- 238000000354 decomposition reaction Methods 0.000 claims description 5
- 230000001172 regenerating effect Effects 0.000 claims description 3
- 238000011161 development Methods 0.000 description 11
- 230000000007 visual effect Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/561—Adding application-functional data or data for application control, e.g. adding metadata
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a data analysis platform, and belongs to the field of data processing. The platform comprises: the WEB front end is used for acquiring data to be analyzed and configuration metadata and transmitting the data to the micro-service layer; the micro-service layer comprises a plurality of micro-service modules, is used for generating analysis rules for the data to be analyzed according to the configuration metadata, and can realize data transmission with the WEB front end and the data analysis layer; and the data analysis layer is used for executing analysis processing according to the analysis rule and the data to be analyzed and transmitting the processed data to the micro-service layer or the database. Meanwhile, a corresponding data analysis method is also provided. The embodiment of the invention aims to provide a data analysis platform and a data analysis method, which at least solve the problems of complicated coding, poor generality, non-standard analysis and the like in the existing data analysis configuration.
Description
Technical Field
The invention relates to the field of data processing, in particular to a data analysis platform, a data analysis method and a corresponding storage medium.
Background
There are various formats for data of industrial equipment, and correspondingly different parsing protocols. How to ensure the standardized operation of various data analysis is always a pain point, and becomes a problem that various Internet of things platforms need to be mainly solved. The technical means of the treatment technical proposal in the prior art are as follows:
the message delivered by the device is in a standardized data parsing format, such as json format. But this approach is detrimental to bandwidth-sensitive and security-sensitive scenarios.
The message transmitted by the device adopts customized binary stream data, but the method is difficult to understand for the analysis of the formats, and can be realized only by specific coding of each format by a developer, so that the requirements of the developer are high, and the codes are difficult to shift.
Disclosure of Invention
The embodiment of the invention aims to provide a data analysis platform and a data analysis method, which at least solve the problems of complicated coding, poor generality, non-standard analysis and the like in the existing data analysis configuration.
In order to achieve the above object, in a first aspect of the present invention, there is provided a data parsing platform, the platform comprising:
the WEB front end is used for acquiring data to be analyzed and configuration metadata and transmitting the data to the micro-service layer;
the micro-service layer comprises a plurality of micro-service modules, is used for generating analysis rules for the data to be analyzed according to the configuration metadata, and can realize data transmission with the WEB front end and the data analysis layer; and
the data analysis layer is used for executing analysis processing on the data to be analyzed according to the analysis rule and transmitting the processed data to the micro-service layer or the database.
Optionally, the micro service layer includes the following micro service modules:
the storage module is used for caching the data to be analyzed before the analysis rule is generated;
the analysis rule generation module is used for generating the analysis rule;
and the data transmission module is used for transmitting data with the WEB front end and the data analysis layer.
Optionally, the micro service layer further includes one or two of the following micro service modules:
the analysis rule verification module is used for verifying the analysis rule;
and the data deployment module is used for responding to the instruction of the user and deploying the analysis rule to the data analysis layer.
Optionally, the data analysis platform further comprises a message queue layer;
the message queue layer is used for caching messages between the micro-service layer and the data analysis layer and caching messages in the micro-service layer.
In a second aspect of the present invention, there is also provided a data parsing method, the method comprising:
performing data definition on the single data to form metadata;
acquiring the configuration of the user on the metadata, and analyzing the configuration metadata;
and generating an analysis rule according to the configuration metadata, wherein the analysis rule is used for analyzing the data to be analyzed.
Optionally, the data definition is derived from a decomposition of a data parsing protocol.
Optionally, the generating the parsing rule according to the configuration metadata includes:
and generating a JAR package from the configuration metadata, wherein the JAR package is used for analyzing the data to be analyzed.
Optionally, after the generating the parsing rule, the data parsing method further includes:
verifying the analysis rule;
if the verification is not passed, regenerating the analysis rule or generating error information.
Optionally, after the generating the parsing rule, the data parsing method further includes:
the analysis rule is deployed, the deployment comprises the step of distributing the analysis rule to an analysis execution module, and the analysis execution module is used for executing analysis on the data to be analyzed.
In a third aspect of the invention, there is also provided a machine-readable storage medium having stored thereon instructions that, when executed by a controller, enable the controller to perform the aforementioned data parsing method.
According to the technical scheme, the data analysis platform and the method are provided, and through the WEB front end and the micro-service architecture, development convenience is improved, and the problems of complicated coding, poor generality, non-standard analysis and the like in the existing data analysis configuration are solved.
Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain, without limitation, the embodiments of the invention. In the drawings:
FIG. 1 is a block diagram of a data parsing platform according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data parsing platform according to another alternative embodiment of the present invention;
FIG. 3 is a flow chart of a data parsing method according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a data parsing process according to an alternative embodiment of the present invention.
Detailed Description
The following describes specific embodiments of the present invention in detail with reference to the drawings. It should be understood that the detailed description and specific examples, while indicating and illustrating the invention, are not intended to limit the invention.
In the embodiments of the present invention, unless otherwise indicated, terms of orientation such as "upper, lower, top, bottom" are used generally with respect to the orientation shown in the drawings or with respect to the positional relationship of the various components with respect to one another in the vertical, vertical or gravitational directions.
Fig. 1 is a schematic diagram of a data parsing platform according to an embodiment of the present invention, as shown in fig. 1, where the platform includes:
the WEB front end is used for acquiring data to be analyzed and configuration metadata and transmitting the data to the micro-service layer;
the micro-service layer comprises a plurality of micro-service modules, is used for generating analysis rules for the data to be analyzed according to the configuration metadata, and can realize data transmission with the WEB front end and the data analysis layer; and
the data analysis layer is used for executing analysis processing on the data to be analyzed according to the analysis rule and transmitting the processed data to the micro-service layer or the database.
Therefore, the structure of the whole data analysis platform is clear through layering design of each function, and data interaction among modules is simplified. And the data analysis protocol is split into data definitions, and when analysis is needed, the needed logic module is configured through the micro service module to form analysis rules, so that analysis of the data to be analyzed is realized. The flexibility and portability of the analysis rule configuration are improved by splitting and flexibly configuring the definition rule of the original data analysis protocol, so that the development convenience is improved, and the problems of complicated coding, poor generality, non-standardization analysis and the like in the existing data analysis configuration are solved.
Specifically, the core architecture of the platform comprises a WEB front end, a micro-service layer and a data analysis layer. The WEB front end utilizes SVG to realize drag-type visual application, and the micro-service layer configures analysis rules of each type through metadata transmitted by WEB front end configuration and stores the analysis rules in a database; when the data needs to be analyzed, a message is sent to a data analysis layer, and the analysis rule of the data is found from a database according to the type of the data needing to be analyzed in the message, so as to analyze the data.
The processing scenario between the above functional modules is as follows:
WEB front-end processing scene: configuring analysis rules of various types into a database through metadata; and transmitting the original data to be analyzed to the micro service, wherein the micro service analyzes the metadata by utilizing the analysis rule configured in the first step and returns the analyzed data to the WEB front end.
The micro service layer processes the scene: receiving metadata transmitted by a WEB front end, and storing analysis rules of various types; the original data transmitted by the WEB front end is transmitted to a data analysis layer; and storing the data of which the analysis is completed by the data analysis layer in a database.
The data analysis layer processes the scene: the data analysis layer analyzes the original data according to analysis data configuration rules stored in the micro service layer, and the analyzed data is put into a database.
In one embodiment of the invention, the platform comprises a WEB front end, a micro service layer and a data analysis layer, wherein the functions in the micro service layer are realized by adopting a plurality of independent micro service modules, and each independent micro service module has respective functions and is matched with other micro service modules. Specifically, the setting of the micro service module in this embodiment is as follows: the micro-service layer at least comprises the following micro-service modules:
the storage module is used for caching the data to be analyzed before the analysis rule is generated;
the analysis rule generation module is used for generating the analysis rule;
and the data transmission module is used for transmitting data with the WEB front end and the data analysis layer.
In an alternative embodiment of the present invention, the micro service layer may include modules in addition to the modules described above,
the analysis rule verification module is used for verifying the analysis rule;
and the data deployment module is used for responding to the instruction of the user and deploying the analysis rule to the data analysis layer.
In an actual scene, the micro service module can be rewritten as required, and the expansion can be performed by adding a new micro service module. The micro service architecture is adopted to realize the functions of the micro service layer, so that the functional requirements are met, and the platform is also beneficial to independent and rapid deployment and expansion.
FIG. 2 is a block diagram of a data parsing platform according to another alternative embodiment of the present invention, as shown in FIG. 2, where the platform further includes a message queue layer; the message queue layer is used for caching messages between the micro-service layer and the data analysis layer; and message caching inside the micro-service layer.
Message transmission in the platform is not necessarily real-time, there may be delays or buffering, and asynchronous processing of messages may be required in some applications. At this time, the messages inside the platform need to be cached, so a message queue layer is set. The message queue layer is used for caching messages between the micro-service layer and the data analysis layer; and message caching inside the micro-service layer. The message queue layer is preferably arranged between the micro service layer and the data parsing layer, because the generation or arrival time between the data to be parsed and the parsing rules is difficult to be simultaneously, and the sequence always exists. When one of the data arrives first, the data needs to be cached, and both the data and the data wait to arrive, so that the data to be analyzed is analyzed by adopting an analysis rule. The message queue layer processing scene comprises: and the transfer of the micro service layer and the message queue message transfer is realized.
The data analysis platform provided by the embodiment can realize rapid analysis of data, can be used in various development environments, and has good processing speed and applicability.
Fig. 3 is a flowchart of a data parsing method according to an embodiment of the present invention, as shown in fig. 3: in one embodiment of the present invention, a data parsing method is further provided, where the method includes:
performing data definition on the single data to form metadata;
acquiring the configuration of the user on the metadata, and analyzing the configuration metadata;
generating an analysis rule according to the configuration metadata; the analysis rule is used for analyzing the data to be analyzed.
Specifically, the data parsing protocol is split into data definitions, which are equivalent to metadata (i.e., data describing the data), and packaged into configurable logic modules, so that the definition of each data is independent and modularized. When data analysis is needed, the needed logic modules are configured according to the actual analysis requirement, namely, the needed logic modules are selected and combined. The specific logic modules are determined according to the data to be analyzed, and may include some independent logic modules, such as an algorithm module, a data length module, and the like. And forming an analysis rule according to the configured logic module and the association relation thereof, so as to analyze the data to be analyzed.
In a specific implementation, the configurable logic module is preferably a component; the components at least comprise data attributes and corresponding analysis methods. The configurable logic module is implemented through the components, and has the advantages of good adaptability, simple development and mature technology. At present, research and development personnel design a data analysis protocol through a specification corresponding to data, and the data analysis protocol is realized through writing codes. The present application decomposes the original data parsing protocol into definitions for each data and encapsulates it into components (which may be referred to as parts or controls in different compilation environments). Each data definition is distinguished by a unique ID, and there will be different definition rules for each data definition, each definition rule being abstracted into components. In an implementation, the data is defined by the attribute in the component, and the analysis method of the data is defined by the method in the component. A specific data can be defined by a component, which is encapsulated independently of each other, and selected at the time of user configuration. The analysis rule can be obtained through the combination of the components, so that the analysis of the data to be analyzed is realized. If a resolution of a data definition is defined, it may be defined to an algorithm component, a data length component, etc., which can be independent for selection at user configuration.
Furthermore, the component is a visual component and is suitable for visual development. Visual development is the automatic generation of application software by visual development tools through the manipulation of interface elements such as menus, buttons, dialog boxes, edit boxes, radio boxes, check boxes, list boxes, scroll bars, and the like, on a graphical user interface provided by the visual development tools. The following describes this embodiment with reference to the SVG format, taking the visualization component as an example, where a plurality of formats are selectable.
SVG codes are added into the components to enable the components to be displayed as vector/grid patterns in a WEB front-end environment. The components are selected and associated by a drag operation on the graphic. The analysis rule configured by the user is obtained through the acquisition and analysis of the configuration information (selection information and association information) of the component, so that the analysis of the data to be analyzed is realized. The implementation mode of the visual component has the advantages of universality and portability, can be intuitively configured by utilizing drag operation, reduces development difficulty, is convenient for operators to understand, and reduces technical background requirements of users.
In the embodiments of the logic module, the component or the visualization component above, the rules defined by the data include algorithm definition, output format, bit definition, data name, byte length, byte order, and the like of the data. Each data definition rule is capable of parsing matched data, where a match is a match relationship of length, algorithm, or name between the data and the data definition.
In one embodiment of the invention, the data definition is derived from a decomposition of a data parsing protocol. In the prior art, data analysis is mostly completed through a data analysis protocol. The data analysis protocol includes a method for defining and analyzing data, and can complete the function of data analysis, but as a whole, the data analysis protocol cannot be rewritten and flexibly configured, and can only meet specific requirements. Through the decomposition of the data analysis protocol, the data definition in the data analysis protocol is multiplexed, so that the development workload can be reduced, the stability of data analysis can be ensured, and the risk brought by self development is avoided.
In one embodiment of the present invention, the generating the parsing rule according to the configuration metadata includes:
and generating a JAR package from the configuration metadata, wherein the JAR package is used for analyzing the data to be analyzed.
The specific mode is as follows: the logic module and its association for the acquisition configuration, where the acquisition may be configured by the user and acquired from through the front end interface. As described above, the logic module includes data definitions, which are equivalent to metadata (i.e., data describing data). Such metadata is parsed from the logic module and may be referred to as configuration metadata because such metadata is related to user configuration. The parsed configuration metadata is saved and compiled into a class file, where the main advantage of the class file resides in being platform independent, which provides a binary form of service independent of the underlying host platform. Packaging the class file into a JAR package; the jar command may be used to package one or more class files, and reference is made to the description of the jar command for details. The configured metadata can then be retrieved by invoking the JAR package.
In one embodiment of the present invention, after the generating the parsing rule, the method further includes: verifying the analysis rule; if the verification is not passed, regenerating the analysis rule or generating error information. If the verification is passed, analyzing the data to be analyzed by adopting the analysis rule: the verification herein includes normalization verification and availability verification, specifically as follows:
the method can be selected according to the actual scene, mainly verifies whether the message formats checked by the configuration metadata and the original data are correct, and further only verifies whether the message header formats are correct.
The verification here is mainly to verify the usability of the generated parsing rule. And analyzing the original data (data to be analyzed) according to the analysis rule generated by the configuration metadata, and returning the analyzed data to the user. If the user judges that the analysis result is correct, the user can prove that the analysis rule can work normally.
In one embodiment of the present invention, after the generating the parsing rule, the method further includes: deploying the parsing rule, namely: and distributing the analysis rule to an analysis execution module, wherein the analysis execution module is used for executing analysis on the data to be analyzed.
The deployment of the parsing rule is performed here, the final deployment of the deployment is performed in a data parsing layer or other data parsing devices, and the deployment process is effective in real time. The main effect of this step is to deploy the generated JAR package to a service for direct invocation by other services to avoid duplicate generation.
Fig. 4 is a flow chart of analysis in an embodiment of the present invention, which is provided for understanding the embodiment of the present invention, and is not limited thereto. As shown in fig. 4, it includes the steps of:
(1) the analysis metadata service sends a data analysis protocol to be deployed to a message queue and modifies the state into a JAR packet to be generated at the same time;
(2) the data analysis front-end service obtains the content of the data analysis protocol from the message queue, generates a JAR packet, stores the JAR packet in a machine of the data analysis front-end service, and sends the message to the message queue.
(3) And analyzing the content of the metadata service acquisition message queue, acquiring a result of generating the JAR package, and modifying the deployment state to finish the JAR generation.
(4) The user online debugging the data analysis protocol, sending the original data (namely the data to be analyzed) to the data analysis front-end service, and returning an analysis result by the data analysis front-end service.
(5) After the online debugging of the user is finished, clicking the deployment, calling the data analysis front-end service by the user service to trigger the deployment flow, deploying the JAR package to the storm cluster, and returning the deployment state to the message queue after the online debugging of the user is finished.
(6) And the analysis metadata service acquires the content of the message queue, and the modification state is deployment completion.
Embodiments of the present invention also provide a machine-readable storage medium having stored thereon instructions that, when executed by a controller, enable the controller to perform the aforementioned data parsing method.
In the technical scheme of the embodiment of the invention, the data analysis protocol is mainly split into analysis metadata, so that the analysis metadata has certain independence and configurability, and the flexibility and portability of data analysis are realized.
The alternative embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the embodiments of the present invention are not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solutions of the embodiments of the present invention within the scope of the technical concept of the embodiments of the present invention, and all the simple modifications belong to the protection scope of the embodiments of the present invention.
In addition, the specific features described in the above embodiments may be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the various possible combinations of embodiments of the invention are not described in detail.
Those skilled in the art will appreciate that all or part of the steps in a method for implementing the above embodiments may be implemented by a program stored in a storage medium, where the program includes several instructions for causing a single-chip microcomputer, chip or processor (processor) to perform all or part of the steps in a method according to the embodiments of the invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In addition, any combination of the various embodiments of the present invention may be made, so long as it does not deviate from the idea of the embodiments of the present invention, and it should also be regarded as what is disclosed in the embodiments of the present invention.
Claims (9)
1. A data parsing platform, the platform comprising:
the WEB front end is used for acquiring data to be analyzed and configuration metadata, and transmitting the data to the micro-service layer, wherein the configuration metadata are from the decomposition of a data analysis protocol, each configuration metadata is contained in a corresponding logic module, and the logic module is a component at least comprising the attribute of the data and a corresponding analysis method;
the micro service layer comprises a plurality of micro service modules, and is used for generating analysis rules for the data to be analyzed according to the configuration metadata, and can realize data transmission with the WEB front end and the data analysis layer, wherein the analysis rules are formed according to logic modules configured by a user and association relations thereof, and the user configuration comprises selecting and combining the required logic modules; and
the data analysis layer is used for executing analysis processing on the data to be analyzed according to the analysis rule and transmitting the processed data to the micro-service layer or the database.
2. The data parsing platform of claim 1, wherein the micro-service layer further comprises the following micro-service modules:
the storage module is used for caching the data to be analyzed before the analysis rule is generated;
the analysis rule generation module is used for generating the analysis rule;
and the data transmission module is used for transmitting data with the WEB front end and the data analysis layer.
3. The data parsing platform of claim 2, wherein the micro-service layer further comprises one or both of the following micro-service modules:
the analysis rule verification module is used for verifying the analysis rule;
and the data deployment module is used for responding to the instruction of the user and deploying the analysis rule to the data analysis layer.
4. A data parsing platform according to any one of claims 1 to 3, characterized in that the data parsing platform further comprises a message queue layer;
the message queue layer is used for caching messages between the micro-service layer and the data analysis layer and caching messages in the micro-service layer.
5. A method of data parsing, the method comprising:
performing data definition on single data to form metadata, wherein the data definition is from decomposition of a data analysis protocol, each data definition is contained in a corresponding logic module, and the logic module is a component at least comprising data attributes and corresponding analysis methods;
the configuration of the metadata of the user is obtained and analyzed into configuration metadata, and the configuration of the metadata of the user comprises selecting and combining a needed logic module;
generating an analysis rule according to the configuration metadata, wherein the analysis rule is used for analyzing the data to be analyzed, and the generating the analysis rule according to the configuration metadata comprises the following steps: and forming the analysis rule according to the logic module configured by the user and the association relation thereof.
6. The data parsing method of claim 5, wherein generating parsing rules from the configuration metadata comprises:
and generating a JAR package from the configuration metadata, wherein the JAR package is used for analyzing the data to be analyzed.
7. The data parsing method according to claim 5, wherein after the parsing rule is generated, the data parsing method further comprises:
verifying the analysis rule;
if the verification is not passed, regenerating the analysis rule or generating error information.
8. The data parsing method according to claim 5, wherein after the parsing rule is generated, the data parsing method further comprises:
the analysis rule is deployed, the deployment comprises the step of distributing the analysis rule to an analysis execution module, and the analysis execution module is used for executing analysis on the data to be analyzed.
9. A machine-readable storage medium having stored thereon instructions which, when executed by a controller, cause the controller to perform the data parsing method of any one of claims 5 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910809468.8A CN112448971B (en) | 2019-08-29 | 2019-08-29 | Data analysis platform, data analysis method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910809468.8A CN112448971B (en) | 2019-08-29 | 2019-08-29 | Data analysis platform, data analysis method and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112448971A CN112448971A (en) | 2021-03-05 |
| CN112448971B true CN112448971B (en) | 2024-01-23 |
Family
ID=74741302
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910809468.8A Active CN112448971B (en) | 2019-08-29 | 2019-08-29 | Data analysis platform, data analysis method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112448971B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1735091A (en) * | 2004-08-12 | 2006-02-15 | 大唐移动通信设备有限公司 | Protocol message analytic method and protocol message analytic system |
| US8868576B1 (en) * | 2012-06-28 | 2014-10-21 | Emc Corporation | Storing files in a parallel computing system based on user-specified parser function |
| CN104966239A (en) * | 2015-06-30 | 2015-10-07 | 天津爱蔻科技有限公司 | Intelligent underwriting platform based on rule engine |
| CN105893052A (en) * | 2016-04-20 | 2016-08-24 | 中国银行股份有限公司 | War packet analyzer |
| CN106506605A (en) * | 2016-10-14 | 2017-03-15 | 华南理工大学 | A kind of SaaS application construction methods based on micro services framework |
| CN106709368A (en) * | 2016-12-15 | 2017-05-24 | 天津交控科技有限公司 | Data analysis method and device |
| CN106941501A (en) * | 2017-04-26 | 2017-07-11 | 田潇河 | A kind of data communications method and device |
| CN107704265A (en) * | 2017-09-30 | 2018-02-16 | 电子科技大学 | A kind of configurable rule generating method of service-oriented stream |
| CN108108288A (en) * | 2018-01-09 | 2018-06-01 | 北京奇艺世纪科技有限公司 | A kind of daily record data analytic method, device and equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7343604B2 (en) * | 2003-07-25 | 2008-03-11 | International Business Machines Corporation | Methods and apparatus for creation of parsing rules |
| US7844957B2 (en) * | 2005-08-19 | 2010-11-30 | Sybase, Inc. | Development system with methodology providing optimized message parsing and handling |
-
2019
- 2019-08-29 CN CN201910809468.8A patent/CN112448971B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1735091A (en) * | 2004-08-12 | 2006-02-15 | 大唐移动通信设备有限公司 | Protocol message analytic method and protocol message analytic system |
| US8868576B1 (en) * | 2012-06-28 | 2014-10-21 | Emc Corporation | Storing files in a parallel computing system based on user-specified parser function |
| CN104966239A (en) * | 2015-06-30 | 2015-10-07 | 天津爱蔻科技有限公司 | Intelligent underwriting platform based on rule engine |
| CN105893052A (en) * | 2016-04-20 | 2016-08-24 | 中国银行股份有限公司 | War packet analyzer |
| CN106506605A (en) * | 2016-10-14 | 2017-03-15 | 华南理工大学 | A kind of SaaS application construction methods based on micro services framework |
| CN106709368A (en) * | 2016-12-15 | 2017-05-24 | 天津交控科技有限公司 | Data analysis method and device |
| CN106941501A (en) * | 2017-04-26 | 2017-07-11 | 田潇河 | A kind of data communications method and device |
| CN107704265A (en) * | 2017-09-30 | 2018-02-16 | 电子科技大学 | A kind of configurable rule generating method of service-oriented stream |
| CN108108288A (en) * | 2018-01-09 | 2018-06-01 | 北京奇艺世纪科技有限公司 | A kind of daily record data analytic method, device and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 元数据的数据解析技术及在卫星设计中的应用;赵飞等;《航天器工程》;20100315(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112448971A (en) | 2021-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8521359B1 (en) | Application-independent and component-isolated system and system of systems framework | |
| CN103178996B (en) | Distributed packet-switching chip model verification system and method | |
| CN102866944B (en) | Pressure testing system and method | |
| CN109240688A (en) | Interface development method, electronic device and readable storage medium storing program for executing | |
| CN107168749A (en) | A kind of Compilation Method, device, equipment and computer-readable recording medium | |
| CN110286939A (en) | Development approach, device, equipment and the storage medium of Software Development Kit | |
| CN112039824A (en) | Communication method, system, device and computer readable storage medium | |
| CN110187986B (en) | Command management method, system, device and computer readable storage medium | |
| CN105190530A (en) | Transmitting hardware-rendered graphical data | |
| CN114064152A (en) | Embedded multi-core debugging system based on dynamic loading and debugging method thereof | |
| CN112433722A (en) | Modular system code development method, device, equipment and system | |
| Chen et al. | Introduction to OPNET network simulation | |
| CN114328217A (en) | Application testing method, device, equipment, medium and computer program product | |
| CN107450993A (en) | A kind of data interactive method of distributed IEC61850 communication components | |
| Kraemer et al. | Automated encapsulation of UML activities for incremental development and verification | |
| CN112448971B (en) | Data analysis platform, data analysis method and storage medium | |
| CN113656152B (en) | Local simulation method, system, medium and electronic equipment based on container cloud environment | |
| CN109947435A (en) | The dispositions method and system of server cluster software environment | |
| WO2017084515A1 (en) | Method and device for transmitting data code stream | |
| KR101660028B1 (en) | Method and apparatus for creating bridging component between hla-dds | |
| CN113918245A (en) | Data calling method, device, equipment and computer readable storage medium | |
| CN100382516C (en) | Method and method of consistency testing for IPv6 main protocol | |
| CN112565041B (en) | Hardware information configuration method, device and medium of FF field bus system | |
| Thramboulidis et al. | Field device specification for the development of function block oriented engineering support systems | |
| CN114398082B (en) | Compatible operation method and device for frame type block chain application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |