CN115514784A - Multisource data acquisition middle platform based on Internet of things - Google Patents
Multisource data acquisition middle platform based on Internet of things Download PDFInfo
- Publication number
- CN115514784A CN115514784A CN202211018685.3A CN202211018685A CN115514784A CN 115514784 A CN115514784 A CN 115514784A CN 202211018685 A CN202211018685 A CN 202211018685A CN 115514784 A CN115514784 A CN 115514784A
- Authority
- CN
- China
- Prior art keywords
- data
- cleaning
- acquisition
- acquiring
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a multisource data acquisition middle platform based on the Internet of things, and relates to the technical field of multisource data acquisition. In order to solve prior art, wash the data of gathering through data washing unit, nevertheless can't classify the data, because the multisource of data, it is great to wash the degree of difficulty, can't guarantee the problem of data quality. The multi-source data acquisition central station based on the Internet of things comprises a front-end acquisition unit, a data cleaning unit, a data processing unit and a storage unit; the data acquisition is carried out through the front-end acquisition unit, the data cleaning unit carries out data cleaning, the cleaned data processing unit carries out unified processing on the data, the storage unit stores the data of the front-end acquisition unit and the data processing unit, a data center platform is constructed, and on the basis of a data warehouse and a data platform, the data are produced to be data API service one by one and are provided for services in a more efficient mode.
Description
Technical Field
The invention relates to the technical field of multi-source data acquisition, in particular to a multi-source data acquisition middle platform based on the Internet of things.
Background
The data center station aggregates and governs cross-domain data, abstractly packages the data into service, and provides the service for the foreground to use the logical concept of service value, the multi-source data acquisition is the basis of the data center station, and at present, related patents exist for the multi-source data acquisition, such as application numbers: CN201711266242.5 discloses a multi-source heterogeneous data rapid acquisition system, which performs data acquisition through a data acquisition unit, performs data cleaning on the acquired data through a data cleaning unit, performs data normalization through a data processing unit, and transmits the sorted data to a cloud server for a user to access.
However, the above patents still have the following problems in practical use:
1. in the prior art, the collected data are cleaned through a data cleaning unit, but the data cannot be classified, and the data cannot be guaranteed in quality due to the fact that the data are multi-sourced and the cleaning difficulty is high;
2. in the prior art, the formulation of a data cleaning strategy belongs to the determination of a working direction on a macroscopic layer, and when the data cleaning strategy is implemented in specific implementation work, the cleaning work cannot be formulated in a detailed manner, so that the ordered implementation of the cleaning work is difficult to ensure.
Disclosure of Invention
The invention aims to provide a multisource data acquisition center station based on the Internet of things, which is used for producing data into individual data API (application programming interface) services on the basis of a data warehouse and a data platform by constructing the data center station and providing the data API services for businesses in a more efficient mode so as to solve the problems in the background art.
In order to achieve the purpose, the invention provides the following technical scheme:
the multi-source data acquisition central station based on the Internet of things comprises a front-end acquisition unit, a data cleaning unit, a data processing unit and a storage unit;
the front-end acquisition unit is used for acquiring multi-source data actively uploaded by more than one acquisition terminal, classifying the multi-source data, packaging the classified multi-source data into a subdata set and conveying the subdata set to the data cleaning unit;
the data cleaning unit is used for supplementing the missing part of the subdata set, correcting the incorrect part, screening and eliminating the redundant part, and finally integrating the subdata set and transmitting the subdata set to the data processing unit;
the data processing unit is used for matching the subdata sets with the service scenes one by one, determining the corresponding relation, determining the priority of the service scenes and sequentially generating data APIs (application programming interfaces) based on the priority;
and the storage unit is used for establishing a communication channel with the front-end acquisition unit and the data processing unit and establishing a multi-source database and an algorithm tool model database.
Further, the front end acquisition unit includes:
a data acquisition module to:
acquiring sensor data and video monitoring data acquired by each acquisition terminal;
wherein the sensor data comprises: positioning a dynamic object, environment perception data and monitoring data of a static object;
a data classification module to:
reading the sensor data and the video monitoring data, wherein the sensor data and the video monitoring data consist of a plurality of variable values;
determining data characteristics in data, determining the category of the data according to the data characteristics, and calling a classification rule of the category of the data from a rule base;
and classifying according to the extracted classification rule and the variable values, and packaging into a plurality of sub-data sets according to classification.
Further, a data cleansing unit, comprising:
a data arrangement module for:
obtaining the plurality of sub data sets, carrying out big data behavior analysis by using a related algorithm, and removing the duplicate value and the null value of the numerical value in the sub data sets;
converting the subdata sets to form a unified data structure, unifying formats of digital data in the subdata sets, and unifying timestamp formats;
the strategy making module is used for acquiring the cleaning modes in the cleaning database and making corresponding cleaning strategies based on the cleaning modes;
the system is also used for matching and cleaning the cleaning method in the database according to different types of data;
a cleaning implementation module, which is used for corresponding the subdata sets to the data quality models in the model database one by one, carrying out quality check on the imported subdata sets and monitoring the cleaning process;
the method is also used for supplementing and perfecting problem data, automatically discovering redundant data, establishing a mapping relation for the redundant data and generating a new standard data.
Further, the policy making module includes:
acquiring a data model corresponding to the subdata set; wherein the data model comprises: single models, single-level models, and multi-level models;
matching the data model with the cleaning strategy one by one, and simultaneously acquiring cleaning rules of the cleaning strategy;
and searching, merging and mapping problem data based on a single model according to similarity matching, establishing a model tree according to the subdata set, hooking mapping among organizations according to the association between the trees, and supplementing, adjusting and mapping the data by combining manual intervention.
Further, the cleaning implementation module is further configured to:
establishing a mapping relation table after acquiring the generated mapping relation, determining a problem data source of the redundant data, determining the pre-stopping data and marking a label;
and acquiring information push in a data acquisition platform, adjusting by combining the actual condition of data and the pre-stop data, establishing a pre-stop data input and output storage list, and highlighting the marked stop data.
Further, a data processing unit comprising:
the service scene matching module is used for acquiring data characteristics in service types, clustering and carding the data characteristics, constructing service scenes according to clustered data and corresponding to the standard data one by one;
the data algorithm module is used for establishing an algorithm tool model and producing a service label corresponding to the service scene for the algorithm tool model;
the data center platform construction module is used for acquiring the processing tools matched with the standard data from the tool database, and acquiring matched tool construction and maintenance data center platforms;
and the data governance module is used for combing governance standards of the standard data and acquiring a data governance tool from the data algorithm module to solve the data quality and safety problems of the standard data around the service scene.
Further, the memory cell includes:
the multi-source database is used for establishing a communication road with the front-end acquisition unit and storing the front-end data acquired by the front-end acquisition unit;
the algorithm tool model database is used for establishing a communication road with the data processing unit and providing an algorithm tool model for the data processing unit;
and the database management module is used for carrying out classification management on the front-end data and the algorithm tool model data.
Further, a database management module comprising:
the input submodule is used for receiving the front-end data and the algorithm tool model data and then sending the front-end data and the algorithm tool model data to the identification submodule;
an identification submodule for:
distinguishing whether the front-end data and the algorithm tool model data are valid data or invalid data, and meanwhile, corresponding to the historical data one by one, distinguishing whether the data are repeated data;
when the front-end data and the algorithm tool model data are judged to be invalid data or repeated data, the identification submodule sends the invalid data or the repeated data to the deletion submodule;
the deleting submodule is used for deleting the fast food invalid data or the repeated data;
and the classification submodule is used for acquiring the identified effective data from the identification submodule and classifying the parameter data.
Compared with the prior art, the invention has the beneficial effects that:
1. the data acquisition is carried out through the front-end acquisition unit, the data cleaning unit carries out data cleaning, the cleaned data processing unit carries out unified processing on the data, the storage unit stores the data of the front-end acquisition unit and the data processing unit, a data center platform is constructed, and on the basis of a data warehouse and a data platform, the data are produced to be data API service one by one and are provided for services in a more efficient mode.
2. The data are packaged into a plurality of sub-data sets, the collection integrity is guaranteed, the cleaning difficulty is further reduced, the data collection quality is improved, the accuracy of a data calculation result is guaranteed, problem data are supplemented and perfected by realizing work distribution and monitoring of stock data transformation, then redundant data are automatically found and a mapping relation is established for the redundant data, and meanwhile a new standard datum is generated, so that the formulation of data cleaning strategies, rules and the like is effectively guided, the quality analysis is carried out on all data, and the consistency, the integrity, the compliance and the redundancy of the data are guaranteed.
3. The problem data is retrieved, combined and mapped through similarity matching, mapping and hanging among organizations are performed through association between trees, data supplement, adjustment and mapping are performed through word segmentation, semantic recognition and other technologies in combination with manual intervention, targeted cleaning methods are selected according to different types of data, the affiliation of the cleaned problem data is finally determined, the problem data is prevented from being in the using process and directly stopping to influence services, and the application range is expanded.
4. The method comprises the steps of determining the priority of a business scene by determining the one-to-one correspondence relationship between data and the business scene, providing a basis for the construction of a data center, combing data standards, component data safety and privacy specifications, analyzing data value, exploring scenes, producing more data services, providing strong data acquisition and storage capacity through a multi-source database and an algorithm tool model database, ensuring the performance and stability of the data services and the data quality and accuracy by a database management module, and simultaneously having strong service management capacity and helping business personnel to explore and find the business value of the data.
Drawings
FIG. 1 is a block diagram of a multi-source data acquisition center of the present invention;
FIG. 2 is a block diagram of a front end acquisition unit of the present invention;
FIG. 3 is a block diagram of a data cleansing unit according to the present invention;
FIG. 4 is a block diagram of a data processing unit of the present invention;
FIG. 5 is a block diagram of a memory cell of the present invention;
FIG. 6 is a diagram of database management module sub-modules of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to solve the technical problems that the existing data acquisition system can only process single data, or the processing is not complete and clear enough, and the data acquisition and processing are not rapid enough, please refer to fig. 1, the embodiment provides the following technical solutions:
the multi-source data acquisition center based on the Internet of things comprises a front-end acquisition unit, a data cleaning unit, a data processing unit and a storage unit; the front-end acquisition unit is used for acquiring the actively uploaded multi-source data from less than one acquisition terminal, classifying the multi-source data, packaging the classified multi-source data into a sub-data set, and conveying the sub-data set to the data cleaning unit; the data cleaning unit is used for supplementing the missing part in the subdata set, correcting the incorrect part, screening and eliminating the redundant repeated part, and finally integrating the subdata set and transmitting the subdata set to the data processing unit; the data processing unit is used for matching the subdata sets with the service scenes one by one, determining the corresponding relation, determining the priority of the service scenes and sequentially generating data APIs (application programming interfaces) based on the priority; and the storage unit is used for establishing a communication channel with the front-end acquisition unit and the data processing unit and establishing a multi-source database and an algorithm tool model database.
Specifically, data acquisition is carried out through the front end acquisition unit, data cleaning is carried out through the data cleaning unit, the data processing unit after cleaning carries out unified processing of data, the storage unit stores the data of front end acquisition unit and data processing unit, constructs the platform in the data, on the basis of data warehouse and data platform, produces data for individual data API service to provide the business for the more efficient mode.
In order to solve the technical problems that the existing front-end data collection cannot guarantee the collection integrity, and the cleaning difficulty is increased due to the multisource of the data and the inability to classify the data, please refer to fig. 2, the embodiment provides the following technical scheme:
a front end acquisition unit comprising: the data acquisition module is used for acquiring sensor data and video monitoring data acquired by each acquisition terminal; wherein the sensor data comprises: positioning dynamic objects, environment perception data and monitoring data of static objects; the data classification module is used for reading the sensor data and the video monitoring data, and the sensor data and the video monitoring data consist of a plurality of variable values; determining data characteristics in data, determining the category of the data according to the data characteristics, and calling a classification rule of the category of the data from a rule base; and classifying according to the extracted classification rule and the variable values, and packing into a plurality of sub-data sets according to classification.
Specifically, the data are packaged into a plurality of sub-data sets according to the classification rules through different classification rules of multi-source data acquired by different acquisition terminals, so that the acquisition integrity is ensured, the cleaning difficulty is further reduced, the data acquisition quality is improved, and the accuracy of a data calculation result is ensured.
In order to solve the technical problem that the data quality cannot be guaranteed due to the multisource of the data and the high cleaning difficulty, please refer to fig. 3, this embodiment provides the following technical solutions:
a data cleansing unit comprising: the data sorting module is used for acquiring the plurality of subdata sets, performing big data behavior analysis by using a related algorithm, and removing the duplication and null values of the numerical values in the subdata sets; converting the subdata sets to form a unified data structure, unifying formats of digital data in the subdata sets, and unifying timestamp formats; the strategy making module is used for acquiring the cleaning modes in the cleaning database and making corresponding cleaning strategies based on the cleaning modes; the system is also used for matching and cleaning the cleaning method in the database according to different types of data; a cleaning implementation module, which is used for corresponding the subdata sets to the data quality models in the model database one by one, carrying out quality check on the imported subdata sets and monitoring the cleaning process; the method is also used for supplementing and perfecting problem data, automatically discovering redundant data, establishing a mapping relation for the redundant data and generating a new standard data.
Specifically, the imported stock data is subjected to quality inspection, work distribution and monitoring of stock data transformation are realized, relevant responsible persons can supplement and perfect problem data according to authority, then redundant data is automatically found, a mapping relation is established for the redundant data, and a new standard data is generated at the same time, so that the formulation of data cleaning strategies, rules and the like is effectively guided, the quality of all data is analyzed by means of a professional data analysis tool, and the consistency, integrity, compliance and redundancy of the data are ensured.
Referring to fig. 3, in order to solve the technical problem that direct disabling of data in the using process may affect a service in the prior art, the following technical solutions are provided in this embodiment:
a policy making module comprising: acquiring a data model corresponding to the subdata set; wherein the data model comprises: single models, single-level models, and multi-level models; matching the data model with the cleaning strategy one by one, and simultaneously acquiring cleaning rules of the cleaning strategy; performing retrieval, combination and mapping processing on problem data according to similarity matching based on a single model, establishing a model tree according to the subdata set, hooking mapping among organizations according to the association between the trees, and performing data supplement, adjustment and mapping by combining manual intervention;
a cleaning implementation module further configured to: establishing a mapping relation table after acquiring the generated mapping relation, determining a problem data source of the redundant data, determining pre-stop data and performing label marking; and acquiring information push in a data acquisition platform, adjusting by combining the actual condition of data and the pre-stop data, establishing a pre-stop data input and output storage list, and highlighting the marked stop data.
Specifically, single model data is cleaned, problem data retrieval, merging and mapping processing are carried out through similarity matching, data cleaning of a single-level model is carried out, mapping and hanging among organizations are carried out through association between trees, data cleaning of a multi-level model is carried out, data supplement, adjustment and mapping are carried out through technologies such as word segmentation and semantic recognition in combination with manual intervention, targeted cleaning methods are selected according to different types of data, the attribution of the cleaned problem data is finally determined, the problem data is prevented from being in the using process, direct outage influences are caused on services, and the application range is expanded.
In order to solve the technical problem that detailed rules of cleaning work cannot be formulated and it is difficult to ensure the orderly cleaning work in the specific implementation work, please refer to fig. 4, this embodiment provides the following technical solutions:
a data processing unit comprising: the service scene matching module is used for acquiring data characteristics in service types, clustering and carding the data characteristics, constructing service scenes according to clustered data and corresponding to the standard data one by one; the data algorithm module is used for establishing an algorithm tool model and producing a service label corresponding to the service scene for the algorithm tool model; the data center platform construction module is used for acquiring the processing tools matched with the standard data from the tool database and acquiring matched tool construction and maintenance data center platforms; and the data governance module is used for combing governance standards of the standard data and acquiring a data governance tool from the data algorithm module to solve the data quality and safety problems of the standard data around the service scene.
Specifically, the priority of the service scene is determined by determining the one-to-one correspondence relationship between the data and the service scene, a basis is provided for the construction of a data center, data are collected, stored, processed and processed by using tools of the center, data standards, member data safety and privacy standards are combed, data quality and safety problems are solved around the service scene by using open source decentralized data management tools, data value is analyzed, the scene is explored, and more data services are produced.
In order to solve the technical problem in the prior art that the data storage management performance and stability are poor due to the fact that the data cannot be planned in a coordinated manner in the data storage, please refer to fig. 5 to 6, the following technical solutions are provided in this embodiment:
a memory cell, comprising: the multi-source database is used for establishing a communication road with the front-end acquisition unit and storing the front-end data acquired by the front-end acquisition unit; the algorithm tool model database is used for establishing a communication road with the data processing unit and providing an algorithm tool model for the data processing unit; the database management module is used for carrying out classification management on the front-end data and the algorithm tool model data;
a database management module comprising: the input submodule is used for receiving the front-end data and the algorithm tool model data and then sending the front-end data and the algorithm tool model data to the identification submodule; the identification submodule is used for identifying whether the front-end data and the algorithm tool model data are valid data or invalid data, and meanwhile, the identification submodule corresponds to the historical data one by one and identifies whether the data are repeated data; when the front-end data and the algorithm tool model data are judged to be invalid data or repeated data, the identification submodule sends the invalid data or the repeated data to the deletion submodule; the deletion submodule is used for deleting the instant invalid data or the repeated data; and the classification submodule is used for acquiring the identified effective data from the identification submodule and classifying the parameter data.
Specifically, powerful data acquisition and storage capacity is provided through a multi-source database and an algorithm tool model database, the database management module ensures the performance and stability of data service and the quality and accuracy of data, and meanwhile, the database management module has strong service management capacity and helps business personnel to explore and find the business value of the data.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered as the technical solutions and the inventive concepts of the present invention within the technical scope of the present invention.
Claims (8)
1. Multisource data acquisition middle station based on thing networking, its characterized in that: the system comprises a front-end acquisition unit, a data cleaning unit, a data processing unit and a storage unit;
the front-end acquisition unit is used for acquiring multi-source data actively uploaded by more than one acquisition terminal, classifying the multi-source data, packaging the classified multi-source data into a subdata set and conveying the subdata set to the data cleaning unit;
the data cleaning unit is used for supplementing the missing part of the subdata set, correcting the incorrect part, screening and eliminating the redundant part, and finally integrating the subdata set and transmitting the subdata set to the data processing unit;
the data processing unit is used for matching the subdata sets with the service scenes one by one, determining the corresponding relation, determining the priority of the service scenes and sequentially generating data APIs (application programming interfaces) based on the priority;
and the storage unit is used for establishing a communication channel with the front-end acquisition unit and the data processing unit and establishing a multi-source database and an algorithm tool model database.
2. The internet-of-things based multi-source data acquisition center of claim 1, wherein: a front end acquisition unit comprising:
a data acquisition module to:
acquiring sensor data and video monitoring data acquired by each acquisition terminal;
wherein the sensor data comprises: positioning dynamic objects, environment perception data and monitoring data of static objects;
a data classification module to:
reading the sensor data and the video monitoring data, wherein the sensor data and the video monitoring data consist of a plurality of variable values;
determining data characteristics in data, determining the category of the data according to the data characteristics, and calling a classification rule of the category of the data from a rule base;
and classifying according to the extracted classification rule and the variable values, and packaging into a plurality of sub-data sets according to classification.
3. The internet-of-things-based multi-source data acquisition center station of claim 2, wherein: a data cleansing unit comprising:
a data collation module for:
acquiring the plurality of subdata sets, performing big data behavior analysis by using a related algorithm, and removing the duplication and null values of the numerical values in the subdata sets;
converting the subdata sets to form a unified data structure, unifying formats of digital data in the subdata sets, and unifying timestamp formats;
the strategy making module is used for acquiring the cleaning modes in the cleaning database and making corresponding cleaning strategies based on the cleaning modes;
the system is also used for matching and cleaning the cleaning methods in the database according to different types of data;
the cleaning implementation module is used for corresponding the subdata sets to the data quality models in the model database one by one, checking the quality of the imported subdata sets and monitoring the cleaning process;
the method is also used for supplementing and perfecting problem data, automatically discovering redundant data, establishing a mapping relation for the redundant data and generating a new standard data.
4. The internet-of-things-based multi-source data acquisition center station of claim 3, wherein: a policy making module comprising:
acquiring a data model corresponding to the subdata set; wherein the data model comprises: single models, single-level models, and multi-level models;
matching the data model with the cleaning strategy one by one, and simultaneously acquiring cleaning rules of the cleaning strategy;
and searching, merging and mapping problem data based on a single model according to similarity matching, establishing a model tree according to the subdata set, hooking mapping among organizations according to the association between the trees, and supplementing, adjusting and mapping the data by combining manual intervention.
5. The internet-of-things-based multi-source data acquisition center station of claim 3, wherein: a cleaning implementation module further configured to:
establishing a mapping relation table after acquiring the generated mapping relation, determining a problem data source of the redundant data, determining the pre-stopping data and marking a label;
and acquiring information push in a data acquisition platform, adjusting by combining the actual condition of data and the pre-stop data, establishing a pre-stop data input and output storage list, and highlighting the marked stop data.
6. The internet-of-things-based multi-source data acquisition center station of claim 3, wherein: a data processing unit comprising:
the service scene matching module is used for acquiring data characteristics in service types, clustering and carding the data characteristics, constructing service scenes according to clustered data and corresponding to the standard data one by one;
the data algorithm module is used for establishing an algorithm tool model and producing a service label corresponding to the service scene for the algorithm tool model;
the data center platform construction module is used for acquiring the processing tools matched with the standard data from the tool database and acquiring matched tool construction and maintenance data center platforms;
and the data governance module is used for combing governance standards of the standard data and acquiring a data governance tool from the data algorithm module to solve the data quality and safety problems of the standard data around the service scene.
7. The internet-of-things-based multi-source data acquisition center station of claim 6, wherein: a memory cell, comprising:
the multi-source database is used for establishing a communication road with the front-end acquisition unit and storing the front-end data acquired by the front-end acquisition unit;
the algorithm tool model database is used for establishing a communication road with the data processing unit and providing an algorithm tool model for the data processing unit;
and the database management module is used for carrying out classification management on the front-end data and the algorithm tool model data.
8. The internet-of-things-based multi-source data acquisition center of claim 7, wherein: a database management module comprising:
the input submodule is used for receiving the front-end data and the algorithm tool model data and then sending the front-end data and the algorithm tool model data to the identification submodule;
an identification submodule for:
distinguishing whether the front-end data and the algorithm tool model data are valid data or invalid data, and meanwhile, corresponding to the historical data one by one, distinguishing whether the data are repeated data;
when the front-end data and the algorithm tool model data are judged to be invalid data or repeated data, the identification submodule sends the invalid data or the repeated data to the deletion submodule;
the deletion submodule is used for deleting the instant invalid data or the repeated data;
and the classification submodule is used for acquiring the identified effective data from the identification submodule and classifying the parameter data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211018685.3A CN115514784A (en) | 2022-08-24 | 2022-08-24 | Multisource data acquisition middle platform based on Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211018685.3A CN115514784A (en) | 2022-08-24 | 2022-08-24 | Multisource data acquisition middle platform based on Internet of things |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115514784A true CN115514784A (en) | 2022-12-23 |
Family
ID=84501295
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211018685.3A Pending CN115514784A (en) | 2022-08-24 | 2022-08-24 | Multisource data acquisition middle platform based on Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115514784A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117493777A (en) * | 2023-12-29 | 2024-02-02 | 成都秦川物联网科技股份有限公司 | Ultrasonic flowmeter data cleaning method, system and device based on Internet of things |
-
2022
- 2022-08-24 CN CN202211018685.3A patent/CN115514784A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117493777A (en) * | 2023-12-29 | 2024-02-02 | 成都秦川物联网科技股份有限公司 | Ultrasonic flowmeter data cleaning method, system and device based on Internet of things |
| CN117493777B (en) * | 2023-12-29 | 2024-03-15 | 成都秦川物联网科技股份有限公司 | Ultrasonic flowmeter data cleaning method, system and device based on Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046035B (en) | Data automation processing method, system, computer equipment and readable storage medium | |
| CN111144938B (en) | Method and system for rating sales lead applicable to automobile industry | |
| CN110034966B (en) | Data flow classification method and system based on machine learning | |
| CN108537422A (en) | Security risk early warning system and method | |
| CN111104521A (en) | Anti-fraud detection method and detection system based on graph analysis | |
| CN109308290B (en) | Efficient data cleaning and converting method based on CIM | |
| CN116415206B (en) | Operator multiple data fusion method, system, electronic equipment and computer storage medium | |
| CN115514784A (en) | Multisource data acquisition middle platform based on Internet of things | |
| CN113613287A (en) | Automatic data acquisition system based on edge calculation | |
| CN114218318B (en) | Data processing system and method for electric power big data | |
| CN113111140A (en) | Method for rapidly analyzing multi-source marine business observation data | |
| CN116541782A (en) | Power marketing data anomaly identification method | |
| CN114003600A (en) | Data processing method, system, electronic device and storage medium | |
| CN110826845B (en) | Multidimensional combination cost allocation device and method | |
| CN112487265A (en) | Data processing method and device, computer storage medium and electronic equipment | |
| CN115269438A (en) | Automatic testing method and device for image processing algorithm | |
| CN112291088B (en) | Method for automatically combing and classifying Web interfaces | |
| CN113037551B (en) | Quick identification and positioning method for sensitive-related services based on traffic slice | |
| CN111914147A (en) | Suspected actual control person credit investigation method and system for enterprise | |
| CN114491168A (en) | Method and system for regulating and controlling cloud sample data sharing, computer equipment and storage medium | |
| CN112965948A (en) | Management service center system based on data | |
| CN111866075A (en) | Acquisition and storage system of electric power information big data | |
| CN116166472B (en) | Data recovery method and system for stored data | |
| CN116049700B (en) | Multi-mode-based operation and inspection team portrait generation method and device | |
| CN109189743A (en) | A kind of the super node identification filter method and system of the low consumption of resources towards the real-time diagram data of big flow |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |