CN112380295A - Warehouse counting system based on industrial cloud edge service - Google Patents
Warehouse counting system based on industrial cloud edge service Download PDFInfo
- Publication number
- CN112380295A CN112380295A CN202011279546.7A CN202011279546A CN112380295A CN 112380295 A CN112380295 A CN 112380295A CN 202011279546 A CN202011279546 A CN 202011279546A CN 112380295 A CN112380295 A CN 112380295A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- industrial
- layer
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
Abstract
The invention provides an industrial cloud edge service-based warehouse counting system which comprises a data acquisition module, a middleware module, a data layering module, an off-line warehouse counting module and a real-time warehouse counting module, wherein the data acquisition module is used for acquiring industrial data, the middleware module is used for receiving the industrial data and transmitting the industrial data to the data layering module, the data layering module is used for processing the received industrial data and respectively persisting the processed industrial data to the off-line warehouse counting module and the real-time warehouse module, and in addition, the off-line warehouse counting module maps the data in the real-time warehouse module. The invention can realize the combination of the offline warehouse counting and the real-time warehouse counting, and can realize the offline warehouse counting and the real-time warehouse counting through the same data link, thereby ensuring the consistency of data, reducing the resource occupation and being convenient for the later maintenance.
Description
Technical Field
The invention relates to the technical field of industrial warehouse counting, in particular to a warehouse counting system based on industrial cloud side service.
Background
With the rapid development of industrial intelligent manufacturing, industrial enterprises begin to merge digital transformation with industry 4.0, wherein offline data warehouse or real-time data warehouse formed by the production of industrial equipment data and industrial system data is often an indispensable ring in large data production systems of industrial enterprises. However, at present, the offline warehouse counting and the real-time warehouse counting of the industrial enterprise are operated respectively, the combination between the offline warehouse counting and the real-time warehouse counting is not tight, two sets of code tasks are needed to realize the offline warehouse counting and the real-time warehouse counting for the same requirement, and then starting a plurality of sets of code tasks means that the same logic needs to be calculated for many times, so that more resources are occupied, and the later maintenance difficulty is greatly increased.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the art described above. Therefore, the invention aims to provide an industrial cloud side service-based warehouse counting system, which can realize the combination of an offline warehouse counting system and a real-time warehouse counting system, and can realize the offline warehouse counting system and the real-time warehouse counting system through the same data link, thereby ensuring the consistency of data, reducing the resource occupation and facilitating the later maintenance.
In order to achieve the above object, an embodiment of the present invention provides an industrial cloud edge service-based bin counting system, which includes a data acquisition module, a middleware module, a data layering module, an offline bin counting module, and a real-time bin counting module, where the data acquisition module is configured to acquire industrial data, the middleware module is configured to receive the industrial data and transmit the industrial data to the data layering module, the data layering module is configured to process the received industrial data, and persist the processed industrial data to the offline bin counting module and the real-time bin counting module, and in addition, the offline bin counting module maps data in the real-time bin counting module.
According to the warehouse counting system based on the industrial cloud side service, the industrial data are collected through the data collection module, the industrial data are received through the middleware module and transmitted to the data layering module, the received industrial data are processed through the data layering module, the processed industrial data are respectively persisted into the offline warehouse counting module and the real-time warehouse counting module, in addition, the offline warehouse counting module can also map data in the real-time warehouse module, therefore, the offline warehouse counting and the real-time warehouse counting can be combined, the offline warehouse counting and the real-time warehouse counting can be achieved through the same data link, the consistency of the data can be guaranteed, the resource occupation can be reduced, and the later maintenance is facilitated.
In addition, the warehouse system based on the industrial cloud edge service according to the embodiment of the present invention may further have the following additional technical features:
according to one embodiment of the invention, the industrial data includes industrial business data and industrial device data, and the industrial device data is stored in a RabbitMQ cluster.
According to one embodiment of the invention, the data acquisition module comprises: a flash-Decode-Interceptor for decrypting the industrial business data to collect the industrial business data; a flux-RabbitMQ-plug-in for collecting the industrial device data RabbitMQ cluster.
According to one embodiment of the invention, the flow-Decode-Interreceptor Interceptor definition implements an Interreceptor interface, and the flow-RabbitMQ-Plugin plug-in definition implements a Configurable interface and an EventDriveSource interface.
According to one embodiment of the invention, the middleware module is message middleware Kafka.
According to one embodiment of the invention, the data layering module comprises a data original layer, a data detail layer, a data aggregation layer and a data mart layer, wherein the data original layer, the data detail layer and the data aggregation layer are respectively connected to the offline data bin module, and the data mart layer is connected to the real-time data bin module.
According to an embodiment of the present invention, the processing of the received industrial data by the data layering module specifically includes: adopting FlinkSql to create a flow table to consume the received industrial data; and writing the industrial data after consumption processing into the data original layer, the data detail layer, the data aggregation layer and the data mart layer in sequence according to different layering logics, wherein when data in the data original layer, the data detail layer and the data aggregation layer are written into the next layering, the data in the data original layer, the data detail layer and the data aggregation layer are persisted to the offline data warehouse module, and the data in the data mart layer is persisted to the real-time data warehouse module.
According to one embodiment of the invention, the offline binning module is provided with an external partition table for mapping data in the real-time binning module.
According to one embodiment of the invention, the offline binning module employs Hive as a storage engine, and the real-time binning module employs HBase as a storage engine.
According to an embodiment of the present invention, the warehouse counting system based on the industrial cloud edge service further includes a storage module, and the storage module is configured to mirror and store task codes of the data acquisition module, the middleware module, the data layering module, the off-line warehouse counting module, and the real-time warehouse counting module.
Drawings
Fig. 1 is a block diagram of a warehouse system based on an industrial cloud edge service according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a warehouse system based on an industrial cloud edge service according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a block diagram of a warehouse system based on an industrial cloud edge service according to an embodiment of the present invention.
As shown in fig. 1, the warehouse counting system based on the industrial cloud edge service according to the embodiment of the present invention includes a data collection module 10, a middleware module 20, a data layering module 30, an offline warehouse counting module 40, and a real-time warehouse counting module 50. The data acquisition module 10 is configured to acquire industrial data, the middleware module 20 is configured to receive the industrial data and transmit the industrial data to the data layering module 30, the data layering module 30 is configured to process the received industrial data and persist the processed industrial data into the offline bin counting module 40 and the real-time bin counting module 50, respectively, and in addition, the offline bin counting module 40 maps data in the real-time bin counting module 50.
In one embodiment of the invention, the industrial data includes industrial business data and industrial device data, and the industrial device data is stored in a RabbitMQ cluster.
In one embodiment of the present invention, as shown in FIG. 2, the data acquisition module 10 may include a flux-Decode-Interreceptor Interceptor 101 and a flux-RabbitMQ-plug-in 102. The flux-Decode-Interceptor 101 may be configured to decrypt the industrial service data to acquire the industrial service data, and the flux-RabbitMQ-plug-in unit 102 may be configured to acquire the RabbitMQ cluster of the industrial device data. It should be noted that the flow-Decode-Interceptor 101 and the flow-RabbitMQ-plug-in 102 can be obtained by performing secondary development on flow.
Wherein, the FLUME-Decode-Interreceptor Interceptor 101 can realize collecting HTTP message data of the industrial service system, namely, the function of encrypting and decrypting the HTTP message data of the industrial service system can be realized, so that when the Flume-Decode-Interceptor 101 is developed, the Flume-Decode-interpolator Interceptor 101 can be customized to realize the interpolator interface, the asymmetric encryption private key can be read in the initialization method, HTTP message data of the industrial service system can be decrypted and serialized into JSON through the read asymmetric encryption private key in the interrupt single event processing method, then, the intercept single event processing method can be circularly called in the intercept batch event processing method, and finally, the flash-Decode-Interreceptor Interceptor 101 can be packaged and put under the lib folder of the flash root directory, so that when a flash task is started, the HTTP message data of the industrial service system may be collected through a flux-decoder-Interceptor 101.
The flow-RabbitMQ-plug-in card 102 can realize RabbitMQSource function, and the RabbitMQSource inherits an abstract Source abstract class and can realize a Configurable interface and an EventDriveSource interface, so that when the flow-RabbitMQ-plug-in card 102 is developed, the flow-RabbitMQ-plug-in card 102 can be customized to realize the Configurable interface and the EventDriveSource interface, RaitMQ cluster connection can be initialized in the Configurable method, RabbitMQ cluster data can be consumed in the start method in a multithreading mode, and finally the flow-RabbitMQ-plug-in card 102 can be packaged and placed under a lib file folder of a flow root directory, so that when a flow task is started, industrial device data can be collected through the flow-RabbitMQ-plug-in card 102.
In one embodiment of the present invention, the industrial service data and the industrial device data collected by the flux-decoder-Interceptor 101 and the flux-RabbitMQ-plug-in 102 may be written into the middleware module 20, for example, the message middleware Kafka, and the middleware module 20, i.e., the message middleware Kafka, may serve as a data real-time link to transmit the written industrial service data and the industrial device data to the data layering module 30.
In one embodiment of the present invention, the offline binning module 40 may employ Hive as a storage engine, and the real-time binning module 50 employs HBase as a storage engine, wherein machine learning model training and analysis may be provided externally by employing Hive storage engine, and sub-second level query may be provided externally by employing HBase storage engine.
In one embodiment of the present invention, as shown in fig. 2, the data layering module 30 includes a data origin layer 301, a data detail layer 302, a data aggregation layer 303 and a data mart layer 304, wherein the data origin layer 301, the data detail layer 302 and the data aggregation layer 303 are respectively connected to the offline data bin module 40, the data mart layer 304 is connected to the real-time data bin module 50, and the data origin layer 301, the data detail layer 302, the data aggregation layer 303 and the data mart layer 304 are further provided with a HiveCatalog and a hierarchical database.
In an embodiment of the present invention, the data layering module 30 may process the received industrial data, specifically, the Flink Sql may be used to create a flow table to consume the received industrial data, and the consumed and processed industrial data may be written into the data raw layer 301, the data fine layer 302, the data aggregation layer 303, and the data mart layer 304 correspondingly in sequence according to different layering logics, that is, the layering logics of the data raw layer 301, the data fine layer 302, the data aggregation layer 303, and the data mart layer 304, where when the data in the data raw layer 301, the data fine layer 302, and the data aggregation layer 303 is written into the next layering, the data in the data raw layer 301, the data fine layer 302, and the data aggregation layer 303 is persisted into the offline data warehouse module 40, for example, Hive, and the data in the data mart layer 304 is persisted into the real-time data warehouse module 50, for example, HBase. Furthermore, when the data in the data origin layer 301, the data specification layer 302, and the data aggregation layer 303 write to the next hierarchical layer, the table structures of the data origin layer 301, the data specification layer 302, and the data aggregation layer 303 will also persist into the offline binning module 40, e.g., Hive, and the table structure of the data mart layer 304 will also persist into the real-time binning module 50, e.g., HBase.
It is further noted that the offline binning module 40 may be provided with an external area table, for example, a Hive external area table may be created, and the external area table, i.e., the Hive external area table, may be used to map data in the real-time binning module 50, for example, HBase, i.e., data mart layer 304 data, thereby enabling the combination of the offline binning module 40 and the real-time binning module 50.
In summary, in the warehouse system based on the industrial cloud edge service according to the embodiment of the present invention, Flink and Hive, and Hive and HBase may be integrated, where a relevant dependency package for Flink and Hive may be placed under a Flink root directory lib folder and a Hive Conf directory may be specified, and a relevant dependency package for Hive and HBase may be copied from the lib folder under the HBase root directory to the Hive root directory lib folder. It should be noted that, Flink, e.g., Flink Sql, may be used as the real-time processing engine, and Hive, e.g., Hive Metastore, may be used as the metadata storage of the Flink.
In an embodiment of the present invention, the industrial cloud edge service-based warehouse system may further include a storage module, and the storage module may be configured to mirror and store task codes of the data collection module 10, the middleware module 20, the data layering module 30, the offline warehouse module 40, and the real-time warehouse module 50. Specifically, the flux, Kafka, Flink, Hive, HBase and related task codes Jar in the warehouse system based on the industrial cloud edge service according to the embodiment of the present invention may be packaged into a mirror image by using a Docker container technology, so that the distribution service of the industrial cloud platform may be quickly deployed in a factory, and automatic management may be implemented when the task codes are upgraded, modified and newly added at a later stage, and a human does not need to be dispatched to an industrial site for deployment, thereby reducing maintenance cost.
According to the warehouse counting system based on the industrial cloud side service, the industrial data are collected through the data collection module, the industrial data are received through the middleware module and transmitted to the data layering module, the received industrial data are processed through the data layering module, the processed industrial data are respectively persisted into the offline warehouse counting module and the real-time warehouse counting module, and in addition, the offline warehouse counting module is mapped to the data in the real-time warehouse counting module, so that the offline warehouse counting and the real-time warehouse counting can be combined, the offline warehouse counting and the real-time warehouse counting can be realized through the same data link, the consistency of the data can be guaranteed, the resource occupation can be reduced, and the later maintenance is facilitated.
In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The meaning of "plurality" is two or more unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; either directly or indirectly through intervening media, either internally or in any other relationship. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the present invention, unless otherwise expressly stated or limited, the first feature "on" or "under" the second feature may be directly contacting the first and second features or indirectly contacting the first and second features through an intermediate. Also, a first feature "on," "over," and "above" a second feature may be directly or diagonally above the second feature, or may simply indicate that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature may be directly under or obliquely under the first feature, or may simply mean that the first feature is at a lesser elevation than the second feature.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. The utility model provides a storehouse system that counts based on industry cloud limit service which characterized in that, includes data acquisition module, middleware module, data layering module, off-line storehouse module and real-time storehouse module, wherein, data acquisition module is used for gathering industrial data, middleware module is used for receiving industrial data and transmits it to data layering module, data layering module is used for handling received industrial data to with after handling industrial data persist respectively to off-line storehouse module with in the real-time storehouse module, in addition, off-line storehouse module maps the data in the real-time storehouse module.
2. The industrial cloud edge service-based data bin system of claim 1 wherein the industrial data comprises industrial traffic data and industrial device data, and the industrial device data is stored in a RabbitMQ cluster.
3. The industrial cloud edge service based binning system of claim 2, wherein said data collection module comprises:
a flash-Decode-Interceptor for decrypting the industrial business data to collect the industrial business data;
a flux-RabbitMQ-plug-in for collecting the industrial device data RabbitMQ cluster.
4. The industrial cloud edge service-based data warehouse system of claim 3, wherein the flow-decoder-Interceptor definition implements an Interceptor interface, and the flow-RabbitMQ-plug-in definition implements a Configurable interface and an eventDriveSource interface.
5. The industrial cloud edge service based binning system of claim 2, wherein said middleware module is message middleware Kafka.
6. The industrial cloud edge service based binning system of claim 2, wherein the data layering module comprises a data raw layer, a data detail layer, a data aggregation layer, and a data mart layer, wherein the data raw layer, the data detail layer, and the data aggregation layer are connected to the offline binning module, respectively, and the data mart layer is connected to the real-time binning module.
7. The industrial cloud edge service-based warehouse counting system according to claim 6, wherein the data layering module processes the received industrial data, and specifically comprises:
adopting Flink Sql to create a flow table to consume the received industrial data;
writing the industrial data after consumption processing into the data original layer, the data detail layer, the data aggregation layer and the data mart layer in sequence according to different hierarchical logics, wherein,
when data in the data origin layer, the data detail layer and the data aggregation layer is written into a next hierarchy, the data in the data origin layer, the data detail layer and the data aggregation layer will persist to the offline data bin module, and the data in the data mart layer will persist to the real-time data bin module.
8. The industrial cloud edge service based binning system of claim 2, wherein the offline binning module is provided with an external partition table for mapping data in the real-time binning module.
9. The industrial cloud edge service based binning system of claim 8, wherein said offline binning module employs Hive as a storage engine and said real-time binning module employs HBase as a storage engine.
10. The industrial cloud edge service based binning system of claim 9, further comprising a storage module for mirroring and storing task codes of said data collection module, said middleware module, said data layering module, said off-line binning module, and said real-time binning module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279546.7A CN112380295B (en) | 2020-11-16 | 2020-11-16 | Warehouse counting system based on industrial cloud edge service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279546.7A CN112380295B (en) | 2020-11-16 | 2020-11-16 | Warehouse counting system based on industrial cloud edge service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112380295A true CN112380295A (en) | 2021-02-19 |
| CN112380295B CN112380295B (en) | 2022-04-08 |
Family
ID=74585477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011279546.7A Active CN112380295B (en) | 2020-11-16 | 2020-11-16 | Warehouse counting system based on industrial cloud edge service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380295B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112965994A (en) * | 2021-03-31 | 2021-06-15 | 杭州海康威视数字技术股份有限公司 | Sequential writing method and device of Flink framework and electronic equipment |
| CN113407617A (en) * | 2021-06-25 | 2021-09-17 | 交控科技股份有限公司 | Real-time and off-line service unified processing method and device based on big data technology |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108984610A (en) * | 2018-06-11 | 2018-12-11 | 华南理工大学 | A kind of method and system based on the offline real-time processing data of big data frame |
| CN109684352A (en) * | 2018-12-29 | 2019-04-26 | 江苏满运软件科技有限公司 | Data analysis system, method, storage medium and electronic equipment |
| CN110457320A (en) * | 2019-08-06 | 2019-11-15 | 深圳萨摩耶互联网金融服务有限公司 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
| CN111753008A (en) * | 2020-06-30 | 2020-10-09 | 珠海迈越信息技术有限公司 | Set top box viewing method and system based on big data analysis |
-
2020
- 2020-11-16 CN CN202011279546.7A patent/CN112380295B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108984610A (en) * | 2018-06-11 | 2018-12-11 | 华南理工大学 | A kind of method and system based on the offline real-time processing data of big data frame |
| CN109684352A (en) * | 2018-12-29 | 2019-04-26 | 江苏满运软件科技有限公司 | Data analysis system, method, storage medium and electronic equipment |
| CN110457320A (en) * | 2019-08-06 | 2019-11-15 | 深圳萨摩耶互联网金融服务有限公司 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
| CN111753008A (en) * | 2020-06-30 | 2020-10-09 | 珠海迈越信息技术有限公司 | Set top box viewing method and system based on big data analysis |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112965994A (en) * | 2021-03-31 | 2021-06-15 | 杭州海康威视数字技术股份有限公司 | Sequential writing method and device of Flink framework and electronic equipment |
| CN112965994B (en) * | 2021-03-31 | 2022-05-03 | 杭州海康威视数字技术股份有限公司 | Sequential writing method and device of Flink framework and electronic equipment |
| CN113407617A (en) * | 2021-06-25 | 2021-09-17 | 交控科技股份有限公司 | Real-time and off-line service unified processing method and device based on big data technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112380295B (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2017202952B2 (en) | System for linking diverse data systems | |
| CN112380295B (en) | Warehouse counting system based on industrial cloud edge service | |
| CN108776699B (en) | Method and device for processing meteorological data and satellite remote sensing data | |
| CN109669776B (en) | Detection task processing method, device and system | |
| CN109299060A (en) | Distributed file method and system based on image rectangular blocks | |
| CN110688348A (en) | File management system | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN107800554B (en) | Data acquisition method, device and system | |
| CN111079515A (en) | Area monitoring method, device, terminal and storage medium based on remote sensing big data | |
| CN108733546A (en) | A kind of log collection method, device and equipment | |
| CN103870557A (en) | Database-based electronic file storage system | |
| CN102882960B (en) | A kind of sending method of resource file and device | |
| CN110377489A (en) | The method and apparatus of log information processing | |
| US11200205B2 (en) | Displaying an alert and options when deleting a file that is associated with a sequence of files | |
| CN110765082B (en) | Hadoop file processing method and device, storage medium and server | |
| CN112667683A (en) | Stream computing system, electronic device and storage medium therefor | |
| KR102013149B1 (en) | Method for collecting and managing of data and system for executint the method | |
| CN116450620A (en) | Database design method and system for multi-source multi-domain space-time reference data | |
| CN106355539A (en) | Police resource real-time state information display method and system | |
| CN111104399A (en) | Mass power grid data management and storage system | |
| CN115809226A (en) | Data docking method and system for intelligent management system of coal washery | |
| CN114153695A (en) | Loose coupling and high-expansibility non-buried point data acquisition method based on Android | |
| CN112148513B (en) | Method, device and storage medium for dumping single-point service data | |
| CN103023740A (en) | Information interaction bus system and electric power data transmission method | |
| CN112468578A (en) | Real-time industrial data acquisition system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |