CN110659323A - Real-time and off-line big data processing system, method, storage medium and terminal - Google Patents
Real-time and off-line big data processing system, method, storage medium and terminal Download PDFInfo
- Publication number
- CN110659323A CN110659323A CN201910914660.3A CN201910914660A CN110659323A CN 110659323 A CN110659323 A CN 110659323A CN 201910914660 A CN201910914660 A CN 201910914660A CN 110659323 A CN110659323 A CN 110659323A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- module
- configuration
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a real-time and off-line big data processing system, a method, a storage medium and a terminal, wherein the system comprises a data configuration module, a data acquisition module, a data conversion module and a data output module; the data configuration module reads the configuration file information and respectively configures the data acquisition module, the data conversion module and the data output module; the data acquisition module extracts a data source from the offline or real-time data stream and converts the data source into data information in a table format; the data conversion module converts the data information; and the data output module outputs the converted data information. The invention realizes the whole-process configuration management of big data processing through the data configuration module, and can process the acquisition of multi-source multi-format data, the conversion of mass data and the output of different formats and different data storage formats. The enterprise learning cost and the development cost are reduced, the process of deploying a big data processing flow by an enterprise is accelerated, the design is modularized and plug-in, the configuration is flexible, the expansion is simple, and the maintenance is convenient.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a real-time and offline big data processing system, method, storage medium and terminal.
Background
The complete processing flow of big data mainly comprises data acquisition, data processing and data output, the three steps are executed in sequence and are mutually dependent, and the execution of the next step can be influenced by the error of one step, especially in a scene with complex service. At present, complete configuration management is not achieved in the three stages in the market, from development and testing to item packing, compiling and deployment need to be carried out for a long time, the process is complex, errors are prone to occurring in intermediate links, and therefore not only can learning cost be brought, but also time cost can be brought. Currently, the three main phases are as follows: in the data acquisition stage, data sources are many, there are a traditional relational database, an ftp data source, a web data source, an hdfs data source and the like, and data formats are also different, and most of enterprises develop different programs for different data sources and data formats at present, so that a system is more and more bloated, maintenance difficulty is more and more large, although data extraction is realized by configuration, the supported data sources are limited, throughput is not high, and tasks are delayed. In the data processing stage, particularly in real-time data processing, most of enterprises are based on spark, storm or flink real-time computing engines at present, the three computing engines need to be subjected to development testing, compiling, online deployment and other processes, the online period is long, the product iteration period is long, each computing submodule cannot be shared, the maintenance workload is large, newly handed employees need to learn related computing engine frames, the learning cost is high, and the development cost and the online period of the enterprises are increased invisibly. In the data output stage, the same problem as that in the data acquisition stage exists, the output format and the storage mode of data are different, the data can be developed according to needs only by developing different program modules, the development workload is large, and the problem of repeated development exists. The problems are main problems faced by the current big data processing, the problems also cause that the current big data project has a long landing period and high labor cost, although some products on the market realize configuration management in an acquisition module, the products still need to be developed according to needs in the data conversion and output stages, and the configuration and monitoring of the whole process cannot be realized.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a real-time and offline big data processing system, method, storage medium and terminal, aiming at the above-mentioned deficiencies of the prior art.
The technical scheme for solving the technical problems is as follows: a real-time and off-line big data processing system comprises a data configuration module, a data acquisition module, a data conversion module and a data output module;
the data configuration module is used for reading configuration file information and respectively configuring the data acquisition module, the data conversion module and the data output module according to the configuration file information;
the data acquisition module is used for calling a corresponding plug-in according to the configuration file information to extract a data source from an offline or real-time data stream and converting the data source into data information in a table format;
the data conversion module is used for converting the data information according to the configuration file information;
and the data output module is used for outputting the converted data information to an external database or a file system according to the configuration file information.
The invention has the beneficial effects that: the real-time and off-line big data processing system realizes the full-flow configuration management of big data processing through the data configuration module, and can process the acquisition of multi-source and multi-format data, the conversion of mass data and the output of different formats and different data storage formats without development. And the enterprise learning cost and the development cost are reduced. And accelerating the process of deploying the big data processing flow by the enterprise. The system is based on the design ideas of modularization and plug-in, is flexible in configuration, simple in expansion and convenient to maintain, and can flexibly develop plug-ins to meet special functions according to own requirements.
On the basis of the technical scheme, the invention can be further improved as follows:
further: the configuration file information comprises data source configuration information, data conversion configuration information and data output information, and the data configuration module is used for correspondingly configuring the data acquisition module, the data conversion module and the data output module according to the data source configuration information, the data conversion configuration information and the data output information.
The beneficial effects of the further scheme are as follows: the data source configuration information, the data conversion configuration information and the data output information can be respectively and correspondingly configured for the data acquisition module, the data conversion module and the data output module, so that the data acquisition module can extract data information of corresponding types from the data source according to specific parameter information of different data types in the data source configuration information, complete conversion of the data information according to the data conversion configuration information and output the converted data information to target data or a system according to the data output information.
Further: the data conversion module converts the data information according to the configuration file information at least sequentially comprises: and carrying out aggregation, data filtering and data calculation processing on the data information.
The beneficial effects of the further scheme are as follows: by carrying out aggregation, data filtering and data calculation processing on the data information, the data information of a specific type can be subjected to targeted conversion, and the data information of a target type and a target parameter can be obtained, so that accurate output is realized.
Further: the real-time and off-line big data processing system also comprises a data monitoring module, wherein the data monitoring module is used for monitoring whether the data acquisition module, the data conversion module and the data output module are abnormal or not and sending alarm information when one or more of the data acquisition module, the data conversion module and the data output module are abnormal.
The beneficial effects of the further scheme are as follows: through the data monitoring module is right the data acquisition process of data acquisition module, the data conversion process of data conversion module and the data output process of data output module carry out the full flow control, and one or more in data acquisition module, data conversion module and the data output module sends alarm information when appearing unusually, and relevant personnel of being convenient for in time discover and take corresponding treatment, guarantee data processing's accuracy and timeliness.
The invention also provides a real-time and off-line big data processing method, which comprises the following steps:
the data configuration module reads configuration file information and respectively configures the data acquisition module, the data conversion module and the data output module according to the configuration file information;
the data acquisition module calls a corresponding plug-in according to the configuration file information to extract a data source from an offline or real-time data stream and converts the data source into data information in a table format;
the data conversion module converts the data information according to the configuration file information;
and the data output module outputs the converted data information to an external database or a file system according to the configuration file information.
According to the real-time and offline big data processing method, the full-flow configuration management of big data processing is realized through the data configuration module, and the acquisition of multi-source and multi-format data, the conversion of mass data and the output of different formats and different data storage formats can be processed without development. And the enterprise learning cost and the development cost are reduced. And accelerating the process of deploying the big data processing flow by the enterprise.
On the basis of the technical scheme, the invention can be further improved as follows:
further: the configuration file information comprises data source configuration information, data conversion configuration information and data output information, and the data configuration module is used for correspondingly configuring the data acquisition module, the data conversion module and the data output module according to the data source configuration information, the data conversion configuration information and the data output information.
The beneficial effects of the further scheme are as follows: the data source configuration information, the data conversion configuration information and the data output information can be respectively and correspondingly configured for the data acquisition module, the data conversion module and the data output module, so that the data acquisition module can extract data information of corresponding types from the data source according to specific parameter information of different data types in the data source configuration information, complete conversion of the data information according to the data conversion configuration information and output the converted data information to target data or a system according to the data output information.
Further: the data conversion module converts the data information according to the configuration file information at least sequentially comprises: and carrying out aggregation, data filtering and data calculation processing on the data information.
The beneficial effects of the further scheme are as follows: by carrying out aggregation, data filtering and data calculation processing on the data information, the data information of a specific type can be subjected to targeted conversion, and the data information of a target type and a target parameter can be obtained, so that accurate output is realized.
Further: the method further comprises the following steps:
the data monitoring module monitors whether the data acquisition module, the data conversion module and the data output module are abnormal or not, and sends alarm information when one or more of the data acquisition module, the data conversion module and the data output module are abnormal.
The beneficial effects of the further scheme are as follows: the data acquisition process of the data acquisition module, the data conversion process of the data conversion module and the data output process of the data output module are monitored in a full flow, and one or more of the data acquisition module, the data conversion module and the data output module sends alarm information when abnormality occurs, so that relevant personnel can find and take corresponding treatment measures in time, and the accuracy and timeliness of data treatment are guaranteed.
The invention also provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the method.
The invention also provides a real-time and off-line big data processing terminal, which comprises a memory, a processor and a computer program which is stored on the memory and can be run on the processor, and is characterized in that: the steps of the method are implemented when the computer program is executed by the processor.
Drawings
FIG. 1 is a schematic diagram of a real-time and offline big data processing system of the present invention;
FIG. 2 is a flow chart of a real-time and offline big data processing method according to the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, a real-time and offline big data processing system includes a data configuration module, a data acquisition module, a data conversion module, and a data output module;
the data configuration module is used for reading configuration file information and respectively configuring the data acquisition module, the data conversion module and the data output module according to the configuration file information;
the data acquisition module is used for calling a corresponding plug-in according to the configuration file information to extract a data source from an offline or real-time data stream and converting the data source into data information in a table format;
the data conversion module is used for converting the data information according to the configuration file information;
and the data output module is used for outputting the converted data information to an external database or a file system according to the configuration file information.
The real-time and off-line big data processing system realizes the full-flow configuration management of big data processing through the data configuration module, and can process the acquisition of multi-source and multi-format data, the conversion of mass data and the output of different formats and different data storage formats without development. And the enterprise learning cost and the development cost are reduced. And accelerating the process of deploying the big data processing flow by the enterprise. The system is based on the design ideas of modularization and plug-in, is flexible in configuration, simple in expansion and convenient to maintain, and can flexibly develop plug-ins to meet special functions according to own requirements.
The big data processing system is realized based on the spark calculation engine, an environment supported by spark needs to be deployed in advance in practice, and then a data source, a conversion rule and data output of the system are deployed, so that the whole big data processing flow can be completed. Each module is described in detail below.
In practice, a complete data processing flow mainly includes the following configuration information: the data acquisition module, the data conversion module and the data output module are respectively configured correspondingly by the data configuration module according to the data source configuration information, the data conversion configuration information and the data output information. The data source configuration information, the data conversion configuration information and the data output information can be respectively and correspondingly configured for the data acquisition module, the data conversion module and the data output module, so that the data acquisition module can extract the data information of corresponding types from the data source according to the specific parameter information of different data types in the data source configuration information, complete the conversion of the data information according to the data conversion configuration information and output the converted data information to target data or a system according to the data output information
In one or more embodiments of the present invention, the data configuration module is mainly responsible for reading configuration file information and providing the configuration file information for each subsequent module to use, and the configuration file format is a relatively flexible json format, and the specific format is as follows:
wherein the spark portion mainly configures parameters related to spark operation, such as:
the project name is as follows: app.name;
the number of threads: spare, instances
cpu core number: park.exicutor.cores;
memory: spare.
The source part mainly configures data source related information, and simultaneously supports the extraction of one or more data sources, and mainly comprises the following information: the data source type and the specific parameters of different data types, the main data sources supported at present are as follows:
the transfer part mainly completes the data conversion function: the method comprises data aggregation, data filtering, data calculation and the like, wherein the data aggregation, calculation and the like are mainly completed in an sql statement mode, and specific parameters comprise 2 parameters, namely sql and target _ table.
The sink part mainly completes data output, and the supported types are as follows:
after the data configuration module reads the configuration file information, different functional plug-ins are called to convert the offline or real-time data stream into the data information in the table format for subsequent data conversion, so that the acquisition of mainstream data sources (file, hdfs, jdbc, mysql, hive, kafka and the like) such as a traditional relational database, file type data and real-time data kafka is supported at present.
In one or more embodiments of the present invention, the converting, by the data conversion module, the data information according to the configuration file information at least sequentially includes: and carrying out aggregation, data filtering and data calculation processing on the data information. By carrying out aggregation, data filtering and data calculation processing on the data information, the data information of a specific type can be subjected to targeted conversion, and the data information of a target type and a target parameter can be obtained, so that accurate output is realized. In the invention, the data conversion module acquires the data information obtained by the data acquisition module according to the configuration file information, and converts the data in a sql statement mode familiar to most people, and the main operations include common aggregation functions (sum, count, add and the like), a plurality of data source joins, sql and the like.
In the invention, the data output module is responsible for outputting data, and the output types comprise file, main stream database (mysql, hbase, es, clickhouse and the like), kafka and the like.
Preferably, in one or more embodiments of the present invention, the real-time and offline big data processing system further includes a data monitoring module, where the data monitoring module is configured to monitor whether the data acquisition module, the data conversion module, and the data output module are abnormal, and send an alarm message when one or more of the data acquisition module, the data conversion module, and the data output module are abnormal. Through the data monitoring module is right the data acquisition process of data acquisition module, the data conversion process of data conversion module and the data output process of data output module carry out the full flow control, and one or more in data acquisition module, data conversion module and the data output module sends alarm information when appearing unusually, and relevant personnel of being convenient for in time discover and take corresponding treatment, guarantee data processing's accuracy and timeliness.
Here, the abnormality when the data acquisition module acquires the data source includes abnormality such as connection failure of the data source, data field missing, and data type error, the abnormality when the data conversion module converts the data information includes abnormality such as data type error and data calculation error, and the abnormality when the data output module outputs the data includes abnormality such as output data field missing, output data type error, and output address error.
In the invention, the alarm information can be sent according to a default target place (such as a mailbox address and the like) preset by the data configuration module, so that related personnel can find the alarm information in time.
As shown in fig. 2, the present invention further provides a real-time and offline big data processing method, which comprises the following steps:
s01, the data configuration module reads the configuration file information and respectively configures the data acquisition module, the data conversion module and the data output module according to the configuration file information;
s02, the data acquisition module calls the corresponding plug-in according to the configuration file information to extract a data source from the off-line or real-time data stream and converts the data source into data information in a table format;
s03, the data conversion module converts the data information according to the configuration file information;
and S04, the data output module outputs the converted data information to an external database or a file system according to the configuration file information.
According to the real-time and offline big data processing method, the full-flow configuration management of big data processing is realized through the data configuration module, and the acquisition of multi-source and multi-format data, the conversion of mass data and the output of different formats and different data storage formats can be processed without development. And the enterprise learning cost and the development cost are reduced. And accelerating the process of deploying the big data processing flow by the enterprise.
In one or more embodiments of the present invention, the configuration file information includes data source configuration information, data conversion configuration information, and data output information, and the data configuration module performs corresponding configuration on the data acquisition module, the data conversion module, and the data output module according to the data source configuration information, the data conversion configuration information, and the data output information, respectively. The data source configuration information, the data conversion configuration information and the data output information can be respectively and correspondingly configured for the data acquisition module, the data conversion module and the data output module, so that the data acquisition module can extract data information of corresponding types from the data source according to specific parameter information of different data types in the data source configuration information, complete conversion of the data information according to the data conversion configuration information and output the converted data information to target data or a system according to the data output information.
In one or more embodiments of the present invention, the converting, by the data conversion module, the data information according to the configuration file information at least sequentially includes: and carrying out aggregation, data filtering and data calculation processing on the data information. By carrying out aggregation, data filtering and data calculation processing on the data information, the data information of a specific type can be subjected to targeted conversion, and the data information of a target type and a target parameter can be obtained, so that accurate output is realized.
Preferably, in one or more embodiments of the present invention, the method further comprises:
the data monitoring module monitors whether the data acquisition module, the data conversion module and the data output module are abnormal or not, and sends alarm information when one or more of the data acquisition module, the data conversion module and the data output module are abnormal.
The data acquisition process of the data acquisition module, the data conversion process of the data conversion module and the data output process of the data output module are monitored in a full flow, and one or more of the data acquisition module, the data conversion module and the data output module sends alarm information when abnormality occurs, so that relevant personnel can find and take corresponding treatment measures in time, and the accuracy and timeliness of data treatment are guaranteed.
The invention also provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the method.
The invention also provides a real-time and off-line big data processing terminal, which comprises a memory, a processor and a computer program which is stored on the memory and can be run on the processor, and is characterized in that: the steps of the method are implemented when the computer program is executed by the processor.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A real-time and off-line big data processing system is characterized by comprising a data configuration module, a data acquisition module, a data conversion module and a data output module;
the data configuration module is used for reading configuration file information and respectively configuring the data acquisition module, the data conversion module and the data output module according to the configuration file information;
the data acquisition module is used for calling a corresponding plug-in according to the configuration file information to extract a data source from an offline or real-time data stream and converting the data source into data information in a table format;
the data conversion module is used for converting the data information according to the configuration file information;
and the data output module is used for outputting the converted data information to an external database or a file system according to the configuration file information.
2. The real-time and offline big data processing system according to claim 1, wherein the configuration file information includes data source configuration information, data conversion configuration information and data output information, and the data configuration module performs corresponding configuration on the data acquisition module, the data conversion module and the data output module according to the data source configuration information, the data conversion configuration information and the data output information, respectively.
3. The real-time and offline big data processing system according to claim 1, wherein the data conversion module converts the data information according to the profile information at least sequentially comprises: and carrying out aggregation, data filtering and data calculation processing on the data information.
4. The real-time and offline big data processing system according to any one of claims 1 to 3, further comprising a data monitoring module, wherein the data monitoring module is configured to monitor whether the data acquisition module, the data conversion module and the data output module are abnormal or not, and send alarm information when one or more of the data acquisition module, the data conversion module and the data output module are abnormal.
5. A real-time and off-line big data processing method is characterized by comprising the following steps:
the data configuration module reads configuration file information and respectively configures the data acquisition module, the data conversion module and the data output module according to the configuration file information;
the data acquisition module calls a corresponding plug-in according to the configuration file information to extract a data source from an offline or real-time data stream and converts the data source into data information in a table format;
the data conversion module converts the data information according to the configuration file information;
and the data output module outputs the converted data information to an external database or a file system according to the configuration file information.
6. The real-time and offline big data processing method according to claim 5, wherein the configuration file information includes data source configuration information, data conversion configuration information and data output information, and the data configuration module performs corresponding configuration on the data acquisition module, the data conversion module and the data output module according to the data source configuration information, the data conversion configuration information and the data output information.
7. The real-time and offline big data processing method according to claim 5, wherein the data conversion module converts the data information according to the configuration file information at least sequentially comprises: and carrying out aggregation, data filtering and data calculation processing on the data information.
8. The real-time and offline big data processing method according to any one of claims 5 to 7, characterized in that the method further comprises:
the data monitoring module monitors whether the data acquisition module, the data conversion module and the data output module are abnormal or not, and sends alarm information when one or more of the data acquisition module, the data conversion module and the data output module are abnormal.
9. A computer-readable storage medium, having stored thereon a computer program for performing, when executed by a processor, the method of any one of claims 5 to 8.
10. A real-time and offline big data processing terminal comprising a memory, a processor and a computer program stored on said memory and executable on said processor, characterized in that: the processor, when executing the computer program, realizes the steps of the method according to any of claims 5-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910914660.3A CN110659323A (en) | 2019-09-26 | 2019-09-26 | Real-time and off-line big data processing system, method, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910914660.3A CN110659323A (en) | 2019-09-26 | 2019-09-26 | Real-time and off-line big data processing system, method, storage medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110659323A true CN110659323A (en) | 2020-01-07 |
Family
ID=69039279
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910914660.3A Pending CN110659323A (en) | 2019-09-26 | 2019-09-26 | Real-time and off-line big data processing system, method, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659323A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625517A (en) * | 2020-04-29 | 2020-09-04 | 山东国华时代投资发展有限公司 | New energy real-time data processing method and device based on change storage |
| CN112131262A (en) * | 2020-10-29 | 2020-12-25 | 常州微亿智造科技有限公司 | Processing system and processing method of streaming data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376081A (en) * | 2014-11-18 | 2015-02-25 | 国家电网公司 | Data application processing system, handhold terminal and on-site checking data processing system |
| CN105512201A (en) * | 2015-11-26 | 2016-04-20 | 晶赞广告（上海）有限公司 | Data collection and processing method and device |
| CN106776903A (en) * | 2016-11-30 | 2017-05-31 | 国网重庆市电力公司电力科学研究院 | A kind of big data shared system and method that auxiliary tone is sought suitable for intelligent grid |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
| CN108062320A (en) * | 2016-11-08 | 2018-05-22 | 长沙博为软件技术股份有限公司 | A kind of method for the data acquisition of multipad, conversion and loading |
-
2019
- 2019-09-26 CN CN201910914660.3A patent/CN110659323A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376081A (en) * | 2014-11-18 | 2015-02-25 | 国家电网公司 | Data application processing system, handhold terminal and on-site checking data processing system |
| CN105512201A (en) * | 2015-11-26 | 2016-04-20 | 晶赞广告（上海）有限公司 | Data collection and processing method and device |
| CN108062320A (en) * | 2016-11-08 | 2018-05-22 | 长沙博为软件技术股份有限公司 | A kind of method for the data acquisition of multipad, conversion and loading |
| CN106776903A (en) * | 2016-11-30 | 2017-05-31 | 国网重庆市电力公司电力科学研究院 | A kind of big data shared system and method that auxiliary tone is sought suitable for intelligent grid |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625517A (en) * | 2020-04-29 | 2020-09-04 | 山东国华时代投资发展有限公司 | New energy real-time data processing method and device based on change storage |
| CN112131262A (en) * | 2020-10-29 | 2020-12-25 | 常州微亿智造科技有限公司 | Processing system and processing method of streaming data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3798846A1 (en) | Operation and maintenance system and method | |
| CN110659323A (en) | Real-time and off-line big data processing system, method, storage medium and terminal | |
| CN106780149A (en) | A kind of equipment real-time monitoring system based on timed task scheduling | |
| CN104767795A (en) | LTE MRO data statistical method and system based on HADOOP | |
| CN114173355B (en) | Method and system for dynamically executing network instruction with separated design running states | |
| US11016736B2 (en) | Constraint programming using block-based workflows | |
| CN114579668A (en) | Database data synchronization method | |
| CN110569174A (en) | Distributed monitoring system and method for NIFI task | |
| CN107885582B (en) | Heterogeneous container cluster migration method and controller | |
| CN112631754A (en) | Data processing method, data processing device, storage medium and electronic device | |
| CN112803587A (en) | Intelligent inspection method for state of automatic equipment based on diagnosis decision library | |
| CN106502842A (en) | Data reconstruction method and system | |
| Pavlović et al. | Synergy between Industry 4.0 and lean methodology | |
| CN107748701B (en) | Reliability analysis method for electric energy metering automation system | |
| CN116095146A (en) | Source configuration method of relay protection information model of main substation and sub station based on block chain | |
| CN112101799B (en) | Standard state dividing method and device based on wind farm data | |
| CN110781647B (en) | Method for realizing data format verification based on Flink | |
| CN114116252A (en) | System and method for storing operation measurement data of regulation and control system | |
| CN103678521A (en) | Distributed file monitoring system based on Hadoop frame | |
| AU2017269259A1 (en) | Data driven invocation of real time wind market forecasting analytics | |
| CN110989988A (en) | Micro-grid edge layer software platform based on edge calculation | |
| CN110825453A (en) | Data processing method and device based on big data platform | |
| Xu et al. | Research on condition monitoring platform for mineral processing equipment based on industrial cloud | |
| KR20200082942A (en) | Knowledge-based Refinery Plant Management and Maintenance | |
| CN116450305B (en) | SOAR platform assembly execution method and device based on distributed task scheduling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200107 |