CN117009325A - Data processing optimization method and system based on Internet of Things platform - Google Patents
Data processing optimization method and system based on Internet of Things platform Download PDFInfo
- Publication number
- CN117009325A CN117009325A CN202311106088.0A CN202311106088A CN117009325A CN 117009325 A CN117009325 A CN 117009325A CN 202311106088 A CN202311106088 A CN 202311106088A CN 117009325 A CN117009325 A CN 117009325A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- target
- processed
- distributed server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000005457 optimization Methods 0.000 title claims abstract description 29
- 238000007405 data analysis Methods 0.000 claims abstract description 36
- 238000003860 storage Methods 0.000 claims abstract description 16
- 238000012795 verification Methods 0.000 claims description 14
- 230000000295 complement effect Effects 0.000 claims description 11
- 238000012550 audit Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000013523 data management Methods 0.000 claims description 5
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 230000004044 response Effects 0.000 abstract description 8
- 230000008569 process Effects 0.000 abstract description 7
- 238000013500 data storage Methods 0.000 abstract description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000006855 networking Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 239000012535 impurity Substances 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a data processing optimization method, a system, electronic equipment and a storage medium based on an Internet of things platform, wherein target data are acquired through data acquisition equipment in response to a data acquisition request sent by network data, and the target data are sent to a distributed server cluster; after receiving the target data, the distributed server cluster performs data analysis and data processing on the target data to obtain processed data, and sends the processed data to a database; summarizing the processed data in the database and then sending the summarized data to a business database of the client; and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database. The data processing optimization method based on the Internet of things platform solves the problems that in the prior art, the processing process is complicated after the Internet of things sensor collects a large amount of data, and the storage cost is high when the data are more or the data storage time is longer.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data processing optimization method, a system, electronic equipment and a storage medium based on an internet of things platform.
Background
The method aims to mine a large amount of data collected by the sensors of the Internet of things to a deeper value, and processes and analyzes the large amount of data collected by the sensors of the Internet of things, but the existing data collecting and processing process is complicated, filtering and impurity removing are inconvenient, the defects of non-intercommunication and non-sharing exist among the data, and the more the data are, the longer the storage time is, the higher the cost for storing the data is.
In the aspect of data acquisition, the invention reduces the repeatability of data acquisition work while ensuring the accuracy of the data, and further improves the accuracy and the integrity of the data.
What is needed is a method for optimizing data processing based on an internet of things platform.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing optimization method, a system, electronic equipment and a storage medium based on an Internet of things platform, which are used for solving the problems that in the prior art, the processing process is complicated after a large amount of data is collected by an Internet of things sensor, and the storage cost is high when the data is more or the data storage time is longer.
To achieve the above objective, an embodiment of the present invention provides a data processing optimization method based on an internet of things platform, where the method specifically includes:
responding to a data acquisition request sent by network data, acquiring target data through data acquisition equipment, and sending the target data to a distributed server cluster;
after the distributed server cluster receives the target data, carrying out data analysis and data processing on the target data to obtain processed data, and sending the processed data to a database;
summarizing the processed data in the database and then sending the summarized data to a business database of a client;
and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database.
Based on the technical scheme, the invention can also be improved as follows:
further, the responding to the data acquisition request sent by the network data acquires the target data through the data acquisition equipment and sends the target data to the distributed server cluster, and the method comprises the following steps:
after the data acquisition equipment receives the data acquisition request sent by the network data, identifying whether the data acquisition request exists in an acquisition task, and acquiring target data through the data acquisition equipment when the target data to be acquired exists in the acquisition task;
when the target data to be acquired does not exist in the acquisition task, initiating a data acquisition request to an acquisition object, and carrying out data acquisition again
Further, the data acquisition device acquires target data in response to a data acquisition request sent by network data, and sends the target data to a distributed server cluster, and the method further includes:
when the target data needs to be subjected to the supplementary recording, auditing the supplementary recording data to obtain auditing results, wherein the auditing results comprise that auditing is passed and auditing is not passed;
and when the audit is passed, sending the complement data to a distributed server cluster, and when the audit is not passed, not performing complement on the complement data.
Further, the data acquisition device acquires target data in response to a data acquisition request sent by network data, and sends the target data to a distributed server cluster, and the method further includes:
when target data needs to be checked, checking the check data to obtain checking results, wherein the checking results comprise that checking is passed and checking is not passed;
when the verification passes, the correct field is automatically covered on the original data, and when the verification fails, the original data is not changed.
Further, after the distributed server cluster receives the target data, performing data analysis and data processing on the target data to obtain processed data, and sending the processed data to a database, including:
after the distributed server cluster receives the target data, the target data is sent to a data analysis module for data analysis, the data analyzed by the data analysis module is sent to a data processing module for data processing, and the processed data is sent to a database.
Further, the database comprises a basic database, a switching tool, an intermediate database and a service database;
the exchange tool performs data cleaning conversion on the processed data stored in the basic database, and sends the processed data subjected to the data cleaning conversion to the intermediate database;
the intermediate library converts the format of the processed data and sends the processed data to a service database;
and the service database performs operations of adding, deleting, modifying, inquiring, importing and exporting on the processed data through the data management module.
Further, the data processing optimization method based on the internet of things platform further comprises the following steps:
and constructing a temporary cluster which is connected with the distributed server, and inquiring the data information of the distributed server cluster through the temporary cluster.
A data processing optimization system based on an internet of things platform, comprising:
the data acquisition equipment is used for responding to a data acquisition request sent by the network data, acquiring target data and sending the target data to the distributed server cluster;
the distributed server cluster comprises a data analysis module, a data processing module and a database;
the data analysis module is used for carrying out data analysis on the target data;
the data processing module is used for carrying out data processing on the data analyzed by the data analysis module to obtain processed data, and sending the processed data to the database;
the database is used for summarizing the processed data and then sending the summarized data to a business database of the client;
and the display terminal is used for responding to the operation request of the data and executing the operation corresponding to the operation request on the processed data in the service database.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method when the computer program is executed.
A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method.
The embodiment of the invention has the following advantages:
according to the data processing optimization method based on the Internet of things platform, in response to a data acquisition request sent by network data, target data are acquired through data acquisition equipment, and the target data are sent to a distributed server cluster; after the distributed server cluster receives the target data, carrying out data analysis and data processing on the target data to obtain processed data, and sending the processed data to a database; summarizing the processed data in the database and then sending the summarized data to a business database of a client; responding to an operation request of data, and executing an operation corresponding to the operation request on the processed data in the service database; the problem of among the prior art thing networking sensor collect a large amount of data aftertreatment process loaded down with trivial details complicacy, and when data are more or data save time is longer, the storage cost is higher is solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It will be apparent to those skilled in the art from this disclosure that the drawings described below are merely exemplary and that other embodiments may be derived from the drawings provided without undue effort.
The structures, proportions, sizes, etc. shown in the present specification are shown only for the purposes of illustration and description, and are not intended to limit the scope of the invention, which is defined by the claims, so that any structural modifications, changes in proportions, or adjustments of sizes, which do not affect the efficacy or the achievement of the present invention, should fall within the scope of the invention.
FIG. 1 is a flow chart of a data processing optimization method based on an Internet of things platform;
FIG. 2 is a first architecture diagram of the data processing optimization system based on the Internet of things platform of the present invention;
FIG. 3 is a second architecture diagram of the data processing optimization system based on the Internet of things platform of the present invention;
fig. 4 is a schematic diagram of an entity structure of an electronic device according to the present invention.
Wherein the reference numerals are as follows:
the system comprises a data acquisition device 10, a data auditing module 101, a complement data submodule 1011, a verification data submodule 1012, a distributed server cluster 20, a data analysis module 201, a data processing module 202, a database 203, a display terminal 30, a temporary cluster 40, an electronic device 50, a processor 501, a memory 502 and a bus 503.
Detailed Description
Other advantages and advantages of the present invention will become apparent to those skilled in the art from the following detailed description, which, by way of illustration, is to be read in connection with certain specific embodiments, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The distributed type refers to that different services are distributed in different places or a plurality of sub-services are split by the same service module and are deployed on different servers, so that the problem of high concurrency is solved. Each node in the distribution may be clustered. The cluster means that the same service is deployed on a plurality of machines, so that the availability of the system is improved. In short, the distributed type improves the efficiency by shortening the execution time of a single task, and the cluster improves the efficiency by improving the number of tasks executed in a unit time.
The invention is based on an internet of things platform and independently builds a distributed data acquisition integrated system. The system comprises a distributed server cluster 20, a data acquisition device 10 and a display terminal 30. In turn, a database 203, a data analysis module 201, and a data processing module 202 are built in the distributed server cluster 20.
Examples
Fig. 1 is a flowchart of an embodiment of a data processing optimization method based on an internet of things platform, and as shown in fig. 1, the embodiment of the invention provides a data processing optimization method based on an internet of things platform, which comprises the following steps:
s101, responding to a data acquisition request sent by network data, acquiring target data by the data acquisition equipment 10 and sending the target data to the distributed server cluster 20;
specifically, after the data acquisition device 10 receives the data acquisition request sent by the network data, it identifies whether the data acquisition request exists in an acquisition task, and when target data to be acquired exists in the acquisition task, the target data is acquired by the data acquisition device 10;
and when the target data to be acquired does not exist in the acquisition task, initiating a data acquisition request to an acquisition object, and carrying out data acquisition again.
When the target data needs to be subjected to the supplementary recording, auditing the supplementary recording data to obtain auditing results, wherein the auditing results comprise that auditing is passed and auditing is not passed;
and when the audit is passed, the complement data is sent to the distributed server cluster 20, and when the audit is not passed, the complement data is not complement.
When target data needs to be checked, checking the check data to obtain checking results, wherein the checking results comprise that checking is passed and checking is not passed;
when the verification passes, the correct field is automatically covered on the original data, and when the verification fails, the original data is not changed.
S102, after the distributed server cluster receives target data, data analysis and data processing are carried out on the target data to obtain processed data, and the processed data are sent to a database 203;
specifically, after receiving the target data, the distributed server cluster 20 sends the target data to the data analysis module 201 for data analysis, sends the data analyzed by the data analysis module 201 to the data processing module 202 for data processing, and sends the processed data to the database 203.
S103, summarizing the processed data in the database 2 and then sending the summarized data to a business database of the client;
specifically, the database 203 includes a base database, a switching tool, an intermediate database, and a service database;
the exchange tool performs data cleaning conversion on the processed data stored in the basic database, and sends the processed data subjected to the data cleaning conversion to the intermediate database;
the intermediate library converts the format of the processed data and sends the processed data to a service database;
and the service database performs operations of adding, deleting, modifying, inquiring, importing and exporting on the processed data through the data management module.
S104, responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database.
Specifically, a user can query the required data information through a service database of the client, and can also perform operations of adding, deleting, modifying, querying, importing and exporting the processed data through a data management module.
The data processing optimization method based on the Internet of things platform further comprises the following steps:
a temporary cluster 40 is constructed which remains connected to the distributed server, and the data information of the distributed server cluster 20 is queried through the temporary cluster 40.
A temporary cluster 40 is constructed and the connection information of the temporary cluster 40 and the distributed server cluster 20 is configured. Querying the data information of the distributed server cluster 20; creating an intermediate table at temporary cluster 40; writing data in a round-robin or random fashion to an intermediate table created by temporary cluster 40; performing merging of the data fragments; packaging the data file; transmitting the file to the distributed server cluster 20; the data information of the distributed server cluster 20 is refreshed. When the data volume is huge, the method improves the stability of data storage, improves the timeliness of data query, and ensures the overall use stability of the data of the Internet of things platform.
According to the data processing optimization method based on the Internet of things platform, in response to a data acquisition request sent by network data, target data are acquired through data acquisition equipment 10, and the target data are sent to a distributed server cluster 20; after the distributed server cluster 20 receives the target data, performing data analysis and data processing on the target data to obtain processed data, and sending the processed data to the database 203; summarizing the processed data in the database 203 and then sending the summarized data to a service database of a client; and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database. The invention solves the problem of relatively complex data acquisition, reduces repeated workload, further improves data accuracy and comprehensiveness, reduces network communication complexity, improves network communication and data storage efficiency, optimizes data warehouse entry performance, ensures data stability, optimizes data query function and reduces query waiting time. The problem of among the prior art thing networking sensor collect a large amount of data aftertreatment process loaded down with trivial details complicacy, and when data are more or data save time is longer, the storage cost is higher is solved.
The data processing optimization method based on the Internet of things platform can ensure the accuracy of data, reduce the repeatability of data acquisition work and further improve the accuracy and the integrity of the data; the system can be reused after one-time acquisition, so that the network communication complexity is reduced, and the network communication and data storage efficiency is improved; the data warehouse-in performance is optimized, and the stability of the data is ensured. Through the guarantee of various indexes of the data, the user experience satisfaction is greatly improved.
FIGS. 2-3 are flowcharts of an embodiment of a data processing optimization system based on an Internet of things platform according to the present invention; as shown in fig. 2-3, the data processing optimization system based on the internet of things platform provided by the embodiment of the invention includes the following steps:
the data acquisition device 10 is used for responding to a data acquisition request sent by network data, acquiring target data and sending the target data to the distributed server cluster 20;
the distributed server cluster 20 comprises a data analysis module 201, a data processing module 202 and a database 203;
the data analysis module 201 is configured to perform data analysis on the target data;
the data processing module 202 is configured to perform data processing on the data analyzed by the data analysis module 201, obtain processed data, and send the processed data to the database 203;
the database 203 is configured to send the processed data to a service database of a client after summarizing the processed data;
and a display terminal 30 for responding to the operation request of the data and executing the operation corresponding to the operation request on the processed data in the service database.
The data acquisition device 10 is further configured to:
after receiving a data acquisition request sent by the network data, identifying whether the data acquisition request exists in an acquisition task, and acquiring target data through data acquisition equipment 10 when the target data to be acquired exists in the acquisition task;
and when the target data to be acquired does not exist in the acquisition task, initiating a data acquisition request to an acquisition object, and carrying out data acquisition again.
The data acquisition device 10 further comprises a data auditing module 101, and the data auditing module 101 further comprises a supplementary data sub-module 1011 and a verification data sub-module 1012;
the data auditing module 101 is used for auditing whether the data is correct;
in the supplementary recording data sub-module 1011, the supplementary recording data source network transmits data, and the supplementary recording data sub-module 1011 functions as supplementary recording data and inputs an auditing result, wherein the auditing result comprises pass or fail: if the verification is passed, the data is sent to the database 203; if the verification is not passed, the data is not subjected to the complement.
In the check data sub-module 1012, the check data source network transmits data, and the check data sub-module 1012 functions as check data and inputs an audit result, wherein the audit result comprises pass and fail: if the verification passes, the verification field automatically and directly covers the correct data to the original data; if the verification is not passed, the verification field data is not changed.
The database 203 comprises a basic database, a switching tool, an intermediate database and a business database;
the exchange tool performs data cleaning conversion on the processed data stored in the basic database, and sends the processed data subjected to the data cleaning conversion to the intermediate database;
the intermediate library converts the format of the processed data and sends the processed data to a service database;
and the service database performs operations of adding, deleting, modifying, inquiring, importing and exporting on the processed data through the data management module.
The data processing optimization method based on the Internet of things platform further comprises the following steps:
a temporary cluster 40 is constructed which remains connected to the distributed server, and the data information of the distributed server cluster 20 is queried through the temporary cluster 40.
According to the data processing optimization system based on the Internet of things platform, the data acquisition equipment 10 responds to a data acquisition request sent by network data, acquires target data and sends the target data to the distributed server cluster 20; performing data analysis on the target data through a data analysis module 201; the data processing module 202 performs data processing on the data analyzed by the data analysis module 201 to obtain processed data, and sends the processed data to the database 203; the processed data is summarized through a database 203 and then sent to a business database of a client; an operation corresponding to the operation request is performed on the processed data in the service database by the display terminal 30 in response to the operation request of the data. The data processing optimization method based on the Internet of things platform solves the problems that in the prior art, the processing process is complicated after the Internet of things sensor collects a large amount of data, and the storage cost is high when the data are more or the data storage time is longer.
Fig. 4 is a schematic diagram of an entity structure of an electronic device according to an embodiment of the present invention, as shown in fig. 4, an electronic device 50 includes: a processor 501 (processor), a memory 502 (memory), and a bus 503;
wherein, the processor 501 and the memory 502 complete the communication with each other through the bus 503;
the processor 501 is configured to invoke program instructions in the memory 502 to perform the methods provided by the above-described method embodiments, for example, including: in response to a data acquisition request sent by network data, acquiring target data by the data acquisition device 10 and sending the target data to the distributed server cluster 20; after the distributed server cluster 20 receives the target data, performing data analysis and data processing on the target data to obtain processed data, and sending the processed data to the database 203; summarizing the processed data in the database 203 and then sending the summarized data to a service database of a client; and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database.
The present embodiment provides a non-transitory computer readable storage medium storing computer instructions that cause a computer to perform the methods provided by the above-described method embodiments, for example, including: in response to a data acquisition request sent by network data, acquiring target data by the data acquisition device 10 and sending the target data to the distributed server cluster 20; after the distributed server cluster 20 receives the target data, performing data analysis and data processing on the target data to obtain processed data, and sending the processed data to the database 203; summarizing the processed data in the database 203 and then sending the summarized data to a service database of a client; and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: various storage media such as ROM, RAM, magnetic or optical disks may store program code.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the embodiments or the methods of some parts of the embodiments.
While the invention has been described in detail in the foregoing general description and specific examples, it will be apparent to those skilled in the art that modifications and improvements can be made thereto. Accordingly, such modifications or improvements may be made without departing from the spirit of the invention and are intended to be within the scope of the invention as claimed.
Claims (10)
1. The data processing optimization method based on the Internet of things platform is characterized by comprising the following steps of:
responding to a data acquisition request sent by network data, acquiring target data through data acquisition equipment, and sending the target data to a distributed server cluster;
after the distributed server cluster receives the target data, carrying out data analysis and data processing on the target data to obtain processed data, and sending the processed data to a database;
summarizing the processed data in the database and then sending the summarized data to a business database of a client;
and responding to the operation request of the data, and executing the operation corresponding to the operation request on the processed data in the service database.
2. The data processing optimization method based on the internet of things platform according to claim 1, wherein the responding to the data acquisition request sent by the network data, acquiring the target data by the data acquisition device, and sending the target data to the distributed server cluster, includes:
after the data acquisition equipment receives the data acquisition request sent by the network data, identifying whether the data acquisition request exists in an acquisition task, and acquiring target data through the data acquisition equipment when the target data to be acquired exists in the acquisition task;
and when the target data to be acquired does not exist in the acquisition task, initiating a data acquisition request to an acquisition object, and carrying out data acquisition again.
3. The data processing optimization method based on the internet of things platform according to claim 1, wherein the responding to the data acquisition request sent by the network data acquires target data through the data acquisition device and sends the target data to the distributed server cluster, and further comprising:
when the target data needs to be subjected to the supplementary recording, auditing the supplementary recording data to obtain auditing results, wherein the auditing results comprise that auditing is passed and auditing is not passed;
and when the audit is passed, sending the complement data to a distributed server cluster, and when the audit is not passed, not performing complement on the complement data.
4. The data processing optimization method based on the internet of things platform according to claim 1, wherein the responding to the data acquisition request sent by the network data acquires target data through the data acquisition device and sends the target data to the distributed server cluster, and further comprising:
when target data needs to be checked, checking the check data to obtain checking results, wherein the checking results comprise that checking is passed and checking is not passed;
when the verification passes, the correct field is automatically covered on the original data, and when the verification fails, the original data is not changed.
5. The method for optimizing data processing based on the platform of the internet of things according to claim 1, wherein after the target data is received by the distributed server cluster, performing data analysis and data processing on the target data to obtain processed data, and sending the processed data to a database, the method comprising:
after the distributed server cluster receives the target data, the target data is sent to a data analysis module for data analysis, the data analyzed by the data analysis module is sent to a data processing module for data processing, and the processed data is sent to a database.
6. The data processing optimization method based on the internet of things platform according to claim 1, wherein the database comprises a basic database, a switching tool, an intermediate database and a service database;
the exchange tool performs data cleaning conversion on the processed data stored in the basic database, and sends the processed data subjected to the data cleaning conversion to the intermediate database;
the intermediate library converts the format of the processed data and sends the processed data to a service database;
and the service database performs operations of adding, deleting, modifying, inquiring, importing and exporting on the processed data through the data management module.
7. The data processing optimization method based on the internet of things platform according to claim 1, further comprising:
and constructing a temporary cluster which is connected with the distributed server, and inquiring the data information of the distributed server cluster through the temporary cluster.
8. The data processing optimizing system based on the platform of the Internet of things is characterized by comprising:
the data acquisition equipment is used for responding to a data acquisition request sent by the network data, acquiring target data and sending the target data to the distributed server cluster;
the distributed server cluster comprises a data analysis module, a data processing module and a database;
the data analysis module is used for carrying out data analysis on the target data;
the data processing module is used for carrying out data processing on the data analyzed by the data analysis module to obtain processed data, and sending the processed data to the database;
the database is used for summarizing the processed data and then sending the summarized data to a business database of the client;
and the display terminal is used for responding to the operation request of the data and executing the operation corresponding to the operation request on the processed data in the service database.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed.
10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311106088.0A CN117009325A (en) | 2023-08-30 | 2023-08-30 | Data processing optimization method and system based on Internet of Things platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311106088.0A CN117009325A (en) | 2023-08-30 | 2023-08-30 | Data processing optimization method and system based on Internet of Things platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117009325A true CN117009325A (en) | 2023-11-07 |
Family
ID=88561896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311106088.0A Pending CN117009325A (en) | 2023-08-30 | 2023-08-30 | Data processing optimization method and system based on Internet of Things platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117009325A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170026450A1 (en) * | 2015-02-04 | 2017-01-26 | Xinyu Xingbang Information Industry Co., Ltd. | Method and system for data processing in multiple data sources based on http protocol |
| CN112788143A (en) * | 2021-01-19 | 2021-05-11 | 澜途集思生态科技集团有限公司 | Distributed data acquisition and summarization method based on Internet of things |
-
2023
- 2023-08-30 CN CN202311106088.0A patent/CN117009325A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170026450A1 (en) * | 2015-02-04 | 2017-01-26 | Xinyu Xingbang Information Industry Co., Ltd. | Method and system for data processing in multiple data sources based on http protocol |
| CN112788143A (en) * | 2021-01-19 | 2021-05-11 | 澜途集思生态科技集团有限公司 | Distributed data acquisition and summarization method based on Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105493028B (en) | Data consistency and rollback for cloud analysis | |
| CN103838867A (en) | Log processing method and device | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN111367760B (en) | Log collection method and device, computer equipment and storage medium | |
| CN103810272A (en) | Data processing method and system | |
| CN112559567A (en) | Query method and device suitable for OLAP query engine | |
| CN112328631A (en) | Production fault analysis method and device, electronic equipment and storage medium | |
| CN112084249A (en) | Access record extraction method and device | |
| CN114549068A (en) | Short link generation method, equipment, device and computer readable storage medium | |
| CN108650123B (en) | Fault information recording method, device, equipment and storage medium | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN113778947A (en) | Data import method, device and equipment of kafka stream processing platform | |
| CN112579406B (en) | Log call chain generation method and device | |
| CN117009325A (en) | Data processing optimization method and system based on Internet of Things platform | |
| CN110765089A (en) | Distributed equipment and log recording method, device and storage medium thereof | |
| CN116109322A (en) | Data acquisition method, data acquisition device, and computer-readable storage medium | |
| CN112860456A (en) | Log processing method and device | |
| CN112162951A (en) | Information retrieval method, server and storage medium | |
| CN107995301B (en) | Rapid data receiving and transmitting method based on Internet | |
| CN110737707A (en) | Multi-system log searching method and device and computer readable storage medium | |
| CN116170321B (en) | Data collection method, device, equipment and storage medium for link tracking | |
| CN110750563A (en) | Multi-model data processing method, system, device, electronic equipment and storage medium | |
| CN113515494B (en) | Database processing method based on distributed file system and electronic equipment | |
| CN116340363B (en) | Data storage and loading method based on relational database and related device | |
| CN112035425B (en) | Log storage method and device and computer system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |