CN112417244A - Big data acquisition and processing system of thing allies oneself with - Google Patents
Big data acquisition and processing system of thing allies oneself with Download PDFInfo
- Publication number
- CN112417244A CN112417244A CN202011293690.6A CN202011293690A CN112417244A CN 112417244 A CN112417244 A CN 112417244A CN 202011293690 A CN202011293690 A CN 202011293690A CN 112417244 A CN112417244 A CN 112417244A
- Authority
- CN
- China
- Prior art keywords
- data
- calling
- unit
- different
- repository
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012216 screening Methods 0.000 claims abstract description 26
- 238000013500 data storage Methods 0.000 claims abstract description 9
- 238000004458 analytical method Methods 0.000 abstract description 8
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses an internet of things big data acquisition and processing system which comprises a data acquisition unit, a screening module, a data storage module, a calling module, a standby data calling unit and a big data processing unit. According to the invention, the data are firstly screened and classified through the data screening module, different data are respectively stored in different storage modules, the system structure is convenient to have more pertinence during data processing, the processing speed is improved, the occurrence of system disorder is prevented, and the system structure is matched with different calling units, when specific data are required to be processed, data calling analysis is carried out through the specific calling unit, the calculated amount of large data is reduced, the stability of the whole system is improved, and through the standby data calling module, when a certain specific calling unit is damaged, data calling is timely replaced, and the stability of the whole system is further ensured.
Description
Technical Field
The invention relates to the technical field of big data acquisition and processing of an internet of things, in particular to a big data acquisition and processing system of the internet of things.
Background
With the development of modern technology, the internet is used more and more, in the era of big data internet, more and more data are shared, especially in factories which are based on the internet for production, if more data are needed, the big data acquisition system of the internet-based internet of things can be processed slowly or crashed, and therefore the big data acquisition and processing system of the internet of things is provided.
Disclosure of Invention
Therefore, the invention aims to provide an internet of things big data acquisition and processing system, which firstly screens and classifies data through a data screening module, and different data are respectively stored in different storage modules, so that the system structure is convenient to have more pertinence during data processing, improves the processing speed, prevents the occurrence of system disorder, and is matched with different calling units, when specific data needs to be processed, data calling analysis is carried out through the specific calling unit, the calculated amount of big data is reduced, the stability of the whole system is improved, and through a standby data calling module, the data calling is timely replaced when a certain specific calling unit is damaged, and the stability of the whole system is further ensured.
To solve the above technical problem, according to an aspect of the present invention, the present invention provides the following technical solutions:
an internet of things big data acquisition and processing system, comprising:
a data acquisition unit for acquiring specific physical link big data;
the screening module comprises a data screening unit connected to the data acquisition unit, and first type data, second type data, third type data and fourth type data which are respectively and independently connected with the data screening unit;
the data storage module comprises a first storage bank, a second storage bank, a third storage bank and a fourth storage bank which are respectively connected to the first kind of data, the second kind of data, the third kind of data and the fourth kind of data;
the calling module comprises a first calling unit, a second calling unit, a third calling unit, a fourth calling unit and control units, wherein the first calling unit, the second calling unit, the third calling unit and the fourth calling unit are respectively connected to the first repository, the second repository, the third repository and the fourth repository, and the control units are respectively connected to different calling units;
the standby data calling unit is respectively connected to different repositories through different connecting wires;
and the big data processing unit is used for processing data and is connected with different calling units.
As a preferable scheme of the system for acquiring and processing big data of the internet of things, the data screening unit screens a plurality of data types, and different screened data are distributed independently.
As a preferred scheme of the big data acquisition and processing system of the internet of things, the system is characterized in that the storage modules are provided with a plurality of data types, and the data types are respectively in one-to-one correspondence with the different storage modules.
As a preferred scheme of the big data collecting and processing system of the internet of things, the calling unit is also provided with a plurality of calling units, and each single calling unit calls the data file in the corresponding storage library.
The system is characterized in that the system comprises a plurality of calling units, a big data processing unit and a plurality of data processing units, wherein the calling units are connected with the big data processing unit through a network.
As a preferable scheme of the system for acquiring and processing big data of an internet of things according to the present invention, the spare data invoking unit is configured to execute invoking a file in any one of the repositories.
Compared with the prior art, the invention has the beneficial effects that: screening classification is firstly carried out to data through the data screening module, store different data respectively to the storage module who does not, such system architecture is convenient more pertinent when carrying out data processing, improve the processing speed, prevent the chaotic condition of system, and mutually support with different calling unit, when specific data need be handled, carry out data call analysis through specific calling unit, reduce the calculated amount of big data, improve entire system's stability, call the module through reserve data, guarantee like this when a certain specific calling unit damages, in time replace and carry out data call, further guarantee entire system's stability.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise. Wherein:
fig. 1 is a system block diagram of a big data acquisition and processing system of the internet of things.
In the figure: 100. a data acquisition unit; 200. a screening module; 210. a data screening unit; 220. a first type of data; 230. a second type of data; 240. a third type of data; 250. a fourth type of data; 300. a data storage module; 310. a first repository; 320. a second repository; 330. a third repository; 340. a fourth repository; 400. calling a module; 410. a first calling unit; 420. a second calling unit; 430. a third calling unit; 440. a fourth calling unit; 450. a control unit; 500. a standby data call unit; 600. big data processing unit.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described herein, and it will be apparent to those of ordinary skill in the art that the present invention may be practiced without departing from the spirit and scope of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
Next, the present invention will be described in detail with reference to the drawings, and in the detailed description of the embodiments of the present invention, the cross-sectional views illustrating the structure of the device are not enlarged partially according to the general scale for convenience of illustration, and the drawings are only examples, which should not limit the scope of the present invention. In addition, the three-dimensional dimensions of length, width and depth should be included in the actual fabrication.
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The invention provides an internet of things big data acquisition and processing system, which is characterized in that data are firstly screened and classified through a data screening module, different data are respectively stored in different storage modules, the system structure is more targeted when data are processed, the processing speed is improved, the disordered condition of the system is prevented, the system structure is matched with different calling units, when specific data need to be processed, data calling analysis is carried out through the specific calling unit, the calculated amount of the big data is reduced, the stability of the whole system is improved, and the standby data calling module ensures that when one specific calling unit is damaged, data calling is timely replaced, and the stability of the whole system is further ensured.
Fig. 1 is a schematic diagram illustrating an overall structure of an embodiment of a big data collecting and processing system of an internet of things according to the present invention, and referring to fig. 1, a main body of the big data collecting and processing system of the internet of things according to the present embodiment includes: the data processing system comprises a data acquisition unit 100, a screening module 200, a data storage module 300, a calling module 400, a standby data calling unit 500 and a big data processing unit 600.
And the data acquisition unit 100 is used for acquiring external specific physical association big data and completing production work based on the internet in a matching manner.
The screening module 200 is used for data classification, prevents data confusion, and maintains stable operation of the system, specifically, the screening module 200 includes a data screening unit 210 connected to the data obtaining unit 100, and a first type of data 220, a second type of data 230, a third type of data 240, and a fourth type of data 250 which are respectively and independently connected to the data screening unit 210, and when the screening module is specifically used, the data screening unit 210 divides and classifies different data to form different types of data, so as to facilitate storage of files of the same type.
The data storage module 300 is configured to store the classified data individually, specifically, the data storage module 300 includes a first repository 310, a second repository 320, a third repository 330, and a fourth repository 340, which are connected to the first class data 220, the second class data 230, the third class data 240, and the fourth class data 250, respectively, and when the data storage module is used specifically, the data classified by the data screening unit 210 is stored in different repositories, so that the data storage module facilitates performing individual call analysis in a later period, prevents data confusion, increases a load of data processing, and reduces a processing speed.
The calling module 400 is used for calling data independently, so as to further improve the data processing rate, specifically, the calling module 400 includes a first calling unit 410, a second calling unit 420, a third calling unit 430, a fourth calling unit 440, which are respectively connected to the first repository 310, the second repository 320, the third repository 330, and the fourth repository 340, and a control unit 450, which is respectively connected to different calling units, when the calling module is used specifically, the control unit 450 controls the first calling unit 410 to call the first type of data 220 in the first repository 310, and after the calling is completed, the data is directly transmitted to the big data processing unit 600 through the first calling unit 410 to perform data analysis processing.
Spare data calls unit 500 and is used for emergent use, further maintains entire system's stability, and is concrete, spare data calls unit 500 and is connected to different repositories through different connecting wires respectively, and when concrete use, when a certain specific call unit damaged, in time the supplementarily carries out data call.
Big data processing unit 600 is used for data processing and is connected with different call units
With reference to fig. 1, in the big data collecting and processing system for internet of things according to this embodiment, when the system is used specifically, the data obtaining unit 100 obtains external specific big data, the production work based on the internet is completed cooperatively, the data screening unit 210 divides and classifies different data to form different types of data, so as to store files of the same type, the data classified by the data screening unit 210 are stored in different storage banks, so as to facilitate later-stage independent call analysis, prevent data confusion, increase the load of data processing, reduce the processing speed, the control unit 450 controls the first call unit 410 to call the first type of data 220 in the first storage bank 310, the data after call is directly transmitted to the big data processing unit 600 through the first call unit 410 to perform data analysis processing, such a system structure facilitates direct analysis of a certain type of files, the system structure is convenient to have pertinence in data processing, improves the processing speed, prevents the occurrence of system disorder, is matched with different calling units, performs data calling analysis through the specific calling unit when the specific data needs to be processed, reduces the calculated amount of big data and improves the stability of the whole system, and ensures that when the specific calling unit is damaged through the standby data calling module, and data calling is carried out in time in a replacement mode, so that the stability of the whole system is further ensured.
While the invention has been described above with reference to an embodiment, various modifications may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In particular, the various features of the disclosed embodiments of the invention may be used in any combination, provided that no structural conflict exists, and the combinations are not exhaustively described in this specification merely for the sake of brevity and resource conservation. Therefore, it is intended that the invention not be limited to the particular embodiments disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (6)
1. An internet of things big data acquisition and processing system is characterized by comprising:
a data acquisition unit (100) that acquires specific physical association big data;
the screening module (200) comprises a data screening unit (210) connected to the data acquisition unit (100), and a first type of data (220), a second type of data (230), a third type of data (240) and a fourth type of data (250) which are respectively and independently connected with the data screening unit (210);
a data storage module (300) comprising a first repository (310), a second repository (320), a third repository (330) and a fourth repository (340) connected to a first type of data (220), a second type of data (230), a third type of data (240) and a fourth type of data (250), respectively;
a calling module (400) comprising a first calling unit (410), a second calling unit (420), a third calling unit (430), a fourth calling unit (440) connected to the first repository (310), the second repository (320), the third repository (330) and the fourth repository (340), respectively, and a control unit (450) connected to different calling units, respectively;
the standby data calling unit (500) is respectively connected to different repositories through different connecting wires;
and the big data processing unit (600) is used for data processing and is connected with different calling units.
2. The big data collecting and processing system of the internet of things of claim 1, wherein the data screening unit (210) screens out several kinds of data, and different screened data are distributed independently.
3. The big data acquisition and processing system of the internet of things as claimed in claim 2, wherein the storage modules are provided in a plurality, and different data types are respectively in one-to-one correspondence with different storage modules.
4. The big data acquisition and processing system of the internet of things as claimed in claim 3, wherein the number of the calling units is also several, and each individual calling unit calls the data file in the corresponding storage library.
5. The big data collecting and processing system of the internet of things of claim 4, wherein when different data are called by a plurality of different calling units, the different data are transmitted to the big data processing unit (600) for data processing.
6. The big data collecting and processing system of claim 5, wherein the spare data calling unit (500) is configured to perform calling a file in any one of the repositories.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293690.6A CN112417244A (en) | 2020-11-18 | 2020-11-18 | Big data acquisition and processing system of thing allies oneself with |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293690.6A CN112417244A (en) | 2020-11-18 | 2020-11-18 | Big data acquisition and processing system of thing allies oneself with |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112417244A true CN112417244A (en) | 2021-02-26 |
Family
ID=74774730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011293690.6A Pending CN112417244A (en) | 2020-11-18 | 2020-11-18 | Big data acquisition and processing system of thing allies oneself with |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417244A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113343691A (en) * | 2021-07-01 | 2021-09-03 | 浙江万里学院 | Big data monitoring terminal and application method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140114972A1 (en) * | 2012-10-22 | 2014-04-24 | Palantir Technologies, Inc. | Sharing information between nexuses that use different classification schemes for information access control |
| CN103955784A (en) * | 2014-04-01 | 2014-07-30 | 南京航空航天大学 | Manufacturing workshop individual electronic display board system based on data collection coprocessor |
| CN107391390A (en) * | 2017-07-04 | 2017-11-24 | 深圳齐心集团股份有限公司 | A kind of computer big data storage system |
| CN110147398A (en) * | 2019-04-25 | 2019-08-20 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, medium and electronic equipment |
| CN111652485A (en) * | 2020-05-25 | 2020-09-11 | 青海绿能数据有限公司 | New energy data acquisition and analysis system based on big data platform |
-
2020
- 2020-11-18 CN CN202011293690.6A patent/CN112417244A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140114972A1 (en) * | 2012-10-22 | 2014-04-24 | Palantir Technologies, Inc. | Sharing information between nexuses that use different classification schemes for information access control |
| CN103955784A (en) * | 2014-04-01 | 2014-07-30 | 南京航空航天大学 | Manufacturing workshop individual electronic display board system based on data collection coprocessor |
| CN107391390A (en) * | 2017-07-04 | 2017-11-24 | 深圳齐心集团股份有限公司 | A kind of computer big data storage system |
| CN110147398A (en) * | 2019-04-25 | 2019-08-20 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, medium and electronic equipment |
| CN111652485A (en) * | 2020-05-25 | 2020-09-11 | 青海绿能数据有限公司 | New energy data acquisition and analysis system based on big data platform |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113343691A (en) * | 2021-07-01 | 2021-09-03 | 浙江万里学院 | Big data monitoring terminal and application method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106802826B (en) | Service processing method and device based on thread pool | |
| CN103246592A (en) | Monitoring acquisition system and method | |
| CN102629219B (en) | Reduce in parallel computation frame holds adaptive load balancing method | |
| CN103327072A (en) | Cluster load balancing method and system | |
| CN105677462A (en) | Distributed task system based on internet of things and business processing method | |
| US7111188B2 (en) | Dynamically configurable fault tolerance in autonomic computing with multiple service points | |
| CN101146127A (en) | A client buffer update method and device in distributed system | |
| CN111880939A (en) | Container dynamic migration method and device and electronic equipment | |
| CN110188135A (en) | Document generating method and equipment | |
| CN104994156A (en) | Load balancing method and system for cluster | |
| CN110058940B (en) | Data processing method and device in multi-thread environment | |
| US10613992B2 (en) | Systems and methods for remote procedure call | |
| CN105933408A (en) | Implementation method and device of Redis universal middleware | |
| CN112114973A (en) | Data processing method and device | |
| CN111400294B (en) | Data anomaly monitoring method, device and system | |
| CN107707644A (en) | Processing method, device, storage medium, processor and the terminal of request message | |
| CN112417244A (en) | Big data acquisition and processing system of thing allies oneself with | |
| CN110543462A (en) | Microservice reliability prediction method, prediction device, electronic device, and storage medium | |
| CN109933430A (en) | The method and apparatus for distributing graphics processor | |
| Kiran et al. | Asymmetric Key-Value Split Pattern Assumption over MapReduce Behavioral Model | |
| CN107172193A (en) | A kind of load-balancing method and its device based on cluster | |
| CN105740077A (en) | Task assigning method applicable to cloud computing | |
| CN108848134A (en) | A kind of server monitoring information collecting method and device | |
| CN106502842A (en) | Data reconstruction method and system | |
| CN109343896A (en) | A kind of big data processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210226 |
|
| RJ01 | Rejection of invention patent application after publication |