CN117009334A - Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium - Google Patents
Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117009334A CN117009334A CN202310977860.XA CN202310977860A CN117009334A CN 117009334 A CN117009334 A CN 117009334A CN 202310977860 A CN202310977860 A CN 202310977860A CN 117009334 A CN117009334 A CN 117009334A
- Authority
- CN
- China
- Prior art keywords
- data
- agricultural
- source heterogeneous
- massive
- protocol
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 238000003672 processing method Methods 0.000 title claims description 27
- 238000000034 method Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 22
- 238000004140 cleaning Methods 0.000 claims abstract description 21
- 238000007405 data analysis Methods 0.000 claims abstract description 12
- 239000002689 soil Substances 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012544 monitoring process Methods 0.000 claims description 14
- 241000607479 Yersinia pestis Species 0.000 claims description 13
- 238000007726 management method Methods 0.000 claims description 11
- 238000013145 classification model Methods 0.000 claims description 7
- 239000003337 fertilizer Substances 0.000 claims description 7
- 230000007613 environmental effect Effects 0.000 claims description 6
- 230000009467 reduction Effects 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 241001465754 Metazoa Species 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 4
- 244000144972 livestock Species 0.000 claims description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000007667 floating Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 235000015097 nutrients Nutrition 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 239000002881 soil fertilizer Substances 0.000 claims description 3
- 238000013480 data collection Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000012271 agricultural production Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/02—Agriculture; Fishing; Forestry; Mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Economics (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- Mining & Mineral Resources (AREA)
- Marine Sciences & Fisheries (AREA)
- Animal Husbandry (AREA)
- Agronomy & Crop Science (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A method for intelligently accessing and processing massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage media belong to the field of agricultural big data processing. The method aims to solve the problems of low data collection efficiency and poor data quality in mass agricultural multi-source heterogeneous sensing data processing. The application constructs an agricultural data grading model and sets a classification label; formulating a division rule, a sensor data cleaning rule and a sensor data association rule of a service scene, and constructing an agricultural service application scene rule base; performing data access and intelligent matching of protocol analysis program packages on the collected massive agricultural multi-source heterogeneous data to complete data analysis, and performing classification tag calibration on the analyzed massive agricultural multi-source heterogeneous data, pushing the classified tags to a cache queue according to the classification tags to obtain massive agricultural multi-source heterogeneous data marked with the classification tags; and inputting the data into an agricultural business application scene rule base for processing to obtain an agricultural business application scene theme data set. The application is convenient for quickly acquiring the effective data according to the service requirement.
Description
Technical Field
The application belongs to the field of agricultural big data processing, and particularly relates to an intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and a storage medium.
Background
With the development of the Internet of things and the Internet technology, mass pre-production, during-production and post-production data are accumulated in the intelligent agriculture field, the agricultural production environment is complex, the agricultural multi-source heterogeneous sensing data has the characteristics of large data volume, rich data sources, high data dimension and the like under the requirement of multi-target monitoring, and the mass multi-source heterogeneous sensing data access and data cleaning become the technical key points in the field of agricultural large data in order to better research and develop the sensing data in the intelligent agriculture field.
Aiming at the access of massive agricultural multi-source heterogeneous sensing data, how to realize intelligent dynamic matching data analysis protocol package and how to finish cleaning and dimension reduction of the massive agricultural multi-source heterogeneous sensing data is an urgent problem to be solved in the field of multi-source heterogeneous big data processing.
Disclosure of Invention
The application aims to solve the problems of low data collection efficiency and poor data quality in mass agricultural multi-source heterogeneous sensing data processing and provides an intelligent access and processing method, electronic equipment and a storage medium for mass agricultural multi-source heterogeneous sensing data.
In order to achieve the above purpose, the present application is realized by the following technical scheme:
an intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data comprises the following steps:
s1, constructing an agricultural data grading model and setting a classification label;
s2, formulating a division rule, a sensor data cleaning rule and a sensor data association rule of a service scene, and constructing an agricultural service application scene rule base;
s3, carrying out data access and intelligent matching of protocol analysis program packages on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1 to complete data analysis, and obtaining analyzed massive agricultural multi-source heterogeneous data;
s4, calibrating the classification labels of the massive agricultural multi-source heterogeneous data analyzed in the step S3, pushing the massive agricultural multi-source heterogeneous data to a cache queue according to the classification labels, and obtaining massive agricultural multi-source heterogeneous data calibrated with the classification labels;
s5, extracting massive agricultural multi-source heterogeneous data of the calibrated classification labels obtained in the step S4 from the cache, inputting the massive agricultural multi-source heterogeneous data into an agricultural business application scene rule base for sensor data cleaning and dimension reduction processing, generating interest indexes, associating the interest indexes with the classification labels, storing association relations, and pushing the massive agricultural multi-source heterogeneous data of the associated interest indexes to a cache queue according to the classification labels;
s6, setting the synchronization time of the interest indexes according to the interest indexes, reading all associated classification labels according to the interest indexes when the set synchronization time of the interest indexes is reached, extracting massive agricultural multi-source heterogeneous data of the associated interest indexes according to the classification labels, synchronizing the extracted massive agricultural multi-source heterogeneous data of the associated interest indexes to a historical database corresponding to the interest indexes, obtaining an agricultural business application scene theme data set, and completing processing of the massive agricultural multi-source heterogeneous sensing data.
Further, the specific implementation method of the step S1 includes the following steps:
s1.1, acquiring an agricultural Internet of things protocol, historical data and a protocol analysis program;
s1.2, constructing an agricultural data grading model, adopting redis as a characteristic database of the agricultural data grading model, and adopting a hash data structure for storing information of an agricultural Internet of things protocol;
s1.3, classifying different sensing data specified in an agricultural data classification model according to types, and setting classification labels;
s1.4, mapping the protocol name of the agricultural Internet of things protocol acquired in the step S1.1 with a protocol analysis program, adopting an identification bit of first frame data of the agricultural Internet of things protocol as a characteristic value, setting the characteristic value as the first 2 bytes of the first frame data of the agricultural Internet of things protocol, storing the characteristic value into a characteristic database of an agricultural data grading model, setting a key storage protocol name, and setting a key corresponding value field to comprise the characteristic value, the first frame length, the first frame character string, a classification label and the highest matching degree; and storing the agricultural Internet of things protocol name and the score by adopting a zipList data structure of Zset, and sequencing the access quantity of the characteristic values by using the score of the agricultural Internet of things protocol name.
Further, step S1.3 sets classification labels including a primary classification label and a secondary classification label, wherein the secondary classification label is refinement of the primary classification label, and the primary classification label is used alone or in combination with the secondary classification label;
the first-level classification tag comprises life information L, environment information E and quality safety Q;
the secondary classification label of the life information L comprises a plant P and an animal A;
the secondary classification label of the environment information E comprises a water body W, soil S, livestock X and weather M;
the secondary classification labels of the quality safety Q comprise smell N, taste T, hearing H and vision V.
Further, the specific implementation method of the step S2 includes the following steps:
s2.1, making a division rule of a service scene: dividing an agricultural business application scene into a soil and fertilizer management scene, a crop growth scene, a pest control scene, an agricultural product quality and a safety monitoring scene;
the sensor data types related to the soil and fertilizer management scene comprise soil nutrient data, environment data and moisture data; the sensor data related to the crop growth scene includes environmental data; the sensing data related to the pest control scene comprises environmental data and pest history data; the sensor data related to the agricultural product quality and safety monitoring scene comprise environment real-time data and crop growth data;
s2.2, formulating a sensor data cleaning rule:
s2.2.1, setting a sensor data type floating point number to reserve two decimal places;
s2.2.2, setting a sensor data unit, wherein the sensor data unit keeps consistency;
s2.2.3, setting a range of sensor data and filtering abnormal data;
s2.2.4, setting the time interval of sensor data to be less than 10 minutes, adopting an extraction mode, selecting the time interval of 30 minutes or 1 hour for extracting data storage, and deleting the rest redundant data;
s2.3, formulating a sensor data association rule: the agricultural business application scene rule base comprises definition of interest indexes and corresponding relation between the interest indexes and classification labels;
s2.3.1, interest index definition: soil fertilizer management is soil, crop growth is crop pest control is insolation quality and safety monitoring of agricultural products is quality;
s2.3.2, the correspondence between interest indexes and classification labels is: the classification label corresponding to the oil comprises: the class labels corresponding to E_ S, E _ M, E _W and crop comprise: the class labels corresponding to L_ P, E _M and insert include: the class labels corresponding to the quality of E_ M, L _A comprise: l_ P, Q _v.
Further, the specific implementation method of the step S3 includes the following steps:
s3.1, carrying out data access on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1, and establishing connection between the agricultural data grading model and the Internet of things equipment adopting a TCP or UDP protocol;
s3.2, intercepting 1 byte and 2 bytes of first frame data sent by the accessed massive agricultural multi-source heterogeneous data by the agricultural data grading model, searching in a characteristic database protocol hash table of the agricultural data grading model according to the sequence of protocol names in a characteristic database of the agricultural data grading model, comparing the characteristic values in the characteristic database of the agricultural data grading model, if the characteristic values are successfully matched and the length of the first frame data is successfully matched, reading a first frame character string of a protocol in the characteristic database of the agricultural data grading model, checking the matching degree, finding a protocol analysis program packet corresponding to the protocol with the highest matching degree, completing data analysis of the protocol, and updating the score corresponding to the protocol in the characteristic database; the calculation formula of the matching degree P (C, D) check is as follows:
P(C，D)＝sqrt(C×D)/(|C|×|D|)
wherein C is the first frame data character string of the access equipment, and D is the first frame character string of the protocol read in the characteristic database.
Further, in step S4, the classification label calibration is performed on the massive agricultural multi-source heterogeneous data analyzed in step S3, and the classification label calibration of the massive agricultural multi-source heterogeneous data is determined by reading the classification labels stored in the protocol table in the feature database of the matched agricultural data classification model.
Further, in step S5, the interest index and the category label are associated in a one-to-one relationship or a one-to-many relationship.
Further, in step S6, different application scenes in the theme dataset of the agricultural service application scene are set to correspond to different interest indexes.
The electronic equipment comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data when executing the computer program.
The computer readable storage medium is stored with a computer program, and the computer program realizes the intelligent access and processing method of the massive agricultural multi-source heterogeneous sensing data when being executed by a processor.
The application has the beneficial effects that:
the application provides an intelligent access and processing method of massive agricultural multi-source heterogeneous sensing data, which provides an agricultural data grading model to realize intelligent matching of data sources and data protocol analysis packages, and provides a method for setting hit scores and ranking for each protocol by constructing a feature database, wherein each comparison is compared from the beginning of the current ranking, so that the matching time is shortened, and classification labels are set for each protocol, thereby facilitating data cleaning and classification storage.
The application provides an intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, which provides an agricultural service application scene rule base, performs data cleaning and filtering aiming at service requirements in a data cleaning link, unifies data formats according to classification labels, improves data quality and reduces data storage cost.
According to the intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, the interest index and the agricultural business application scene theme data set are formed through the classification label-cache mechanism, so that the data access time is reduced in the subsequent data analysis and mining links, and the effective data can be conveniently and rapidly acquired according to business requirements.
Drawings
FIG. 1 is a flow chart of an intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, which is disclosed by the application;
fig. 2 is a detailed flow diagram of a method for intelligent access and processing of massive agricultural multi-source heterogeneous sensing data according to the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and detailed description. It should be understood that the embodiments described herein are for purposes of illustration only and are not intended to limit the application, i.e., the embodiments described are merely some, but not all, of the embodiments of the application. The components of the embodiments of the present application generally described and illustrated in the figures herein can be arranged and designed in a wide variety of different configurations, and the present application can have other embodiments as well.
Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to fall within the scope of the present application.
For further understanding of the application, the following detailed description is to be taken in conjunction with fig. 1-2, in which the following detailed description is given, of the application:
the first embodiment is as follows:
an intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data comprises the following steps:
s1, constructing an agricultural data grading model and setting a classification label;
further, the specific implementation method of the step S1 includes the following steps:
s1.1, acquiring an agricultural Internet of things protocol, historical data and a protocol analysis program;
s1.2, constructing an agricultural data grading model, adopting redis as a characteristic database of the agricultural data grading model, and adopting a hash data structure for storing information of an agricultural Internet of things protocol;
s1.3, classifying different sensing data specified in an agricultural data classification model according to types, and setting classification labels;
s1.4, mapping the protocol name of the agricultural Internet of things protocol acquired in the step S1.1 with a protocol analysis program, adopting an identification bit of first frame data of the agricultural Internet of things protocol as a characteristic value, setting the characteristic value as the first 2 bytes of the first frame data of the agricultural Internet of things protocol, storing the characteristic value into a characteristic database of an agricultural data grading model, setting a key storage protocol name, and setting a key corresponding value field to comprise the characteristic value, the first frame length, the first frame character string, a classification label and the highest matching degree; storing the agricultural Internet of things protocol name and the score by adopting a zipList data structure of Zset, and sequencing the access quantity of the characteristic values by using the score of the agricultural Internet of things protocol name;
further, step S1.3 sets classification labels including a primary classification label and a secondary classification label, wherein the secondary classification label is refinement of the primary classification label, and the primary classification label is used alone or in combination with the secondary classification label;
the first-level classification tag comprises life information L, environment information E and quality safety Q;
the secondary classification label of the life information L comprises a plant P and an animal A;
the secondary classification label of the environment information E comprises a water body W, soil S, livestock X and weather M;
the secondary classification label of the quality safety Q comprises smell N, taste T, hearing H and vision V;
s2, formulating a division rule, a sensor data cleaning rule and a sensor data association rule of a service scene, and constructing an agricultural service application scene rule base;
further, the specific implementation method of the step S2 includes the following steps:
s2.1, making a division rule of a service scene: dividing an agricultural business application scene into a soil and fertilizer management scene, a crop growth scene, a pest control scene, an agricultural product quality and a safety monitoring scene;
the sensor data types related to the soil and fertilizer management scene comprise soil nutrient data, environment data and moisture data; the sensor data related to the crop growth scene includes environmental data; the sensing data related to the pest control scene comprises environmental data and pest history data; the sensor data related to the agricultural product quality and safety monitoring scene comprise environment real-time data and crop growth data;
s2.2, formulating a sensor data cleaning rule:
s2.2.1, setting a sensor data type floating point number to reserve two decimal places;
s2.2.2, setting a sensor data unit, wherein the sensor data unit keeps consistency;
s2.2.3, setting a range of sensor data and filtering abnormal data;
s2.2.4, setting the time interval of sensor data to be less than 10 minutes, adopting an extraction mode, selecting the time interval of 30 minutes or 1 hour for extracting data storage, and deleting the rest redundant data;
s2.3, formulating a sensor data association rule: the agricultural business application scene rule base comprises definition of interest indexes and corresponding relation between the interest indexes and classification labels;
s2.3.1, interest index definition: soil fertilizer management is soil, crop growth is crop pest control is insolation quality and safety monitoring of agricultural products is quality;
s2.3.2, the correspondence between interest indexes and classification labels is: the class label corresponding to the tail comprises E_ S, E _ M, E _W, and the class label corresponding to the crop comprises: the class labels corresponding to L_ P, E _M and insert include: the class labels corresponding to the quality of E_ M, L _A comprise: l_ P, Q _v.
S3, carrying out data access and intelligent matching of protocol analysis program packages on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1 to complete data analysis, and obtaining analyzed massive agricultural multi-source heterogeneous data;
further, the specific implementation method of the step S3 includes the following steps:
s3.1, carrying out data access on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1, and establishing connection between the agricultural data grading model and the Internet of things equipment adopting a TCP or UDP protocol;
s3.2, intercepting 1 byte and 2 bytes of first frame data sent by the accessed massive agricultural multi-source heterogeneous data by the agricultural data grading model, searching in a characteristic database protocol hash table of the agricultural data grading model according to the sequence of protocol name ordering in a characteristic database of the agricultural data grading model, comparing in the characteristic database of the agricultural data grading model through characteristic values, if the characteristic values are successfully matched and the length of the first frame data is successfully matched, reading a first frame character string of a protocol in the characteristic database of the agricultural data grading model, checking the matching degree, finding a protocol analysis program packet corresponding to the protocol with the highest matching degree, completing data analysis of the protocol, and updating the corresponding score of the protocol in the characteristic database; the calculation formula of the matching degree P (C, D) check is as follows:
P(C，D)＝sqrt(C×D)/(|C|×|D|)
wherein C is the first frame data character string of the access equipment, D is the first frame character string of the protocol read from the characteristic database;
s4, calibrating the classification labels of the massive agricultural multi-source heterogeneous data analyzed in the step S3, pushing the massive agricultural multi-source heterogeneous data to a cache queue according to the classification labels, and obtaining massive agricultural multi-source heterogeneous data calibrated with the classification labels;
further, in the step S4, the mass agricultural multi-source heterogeneous data analyzed in the step S3 is subjected to classification label calibration, and the classification label calibration of the mass agricultural multi-source heterogeneous data is determined by reading the classification labels stored in the protocol table in the feature database of the matched agricultural data classification model;
s5, extracting massive agricultural multi-source heterogeneous data of the calibrated classification labels obtained in the step S4 from the cache, inputting the massive agricultural multi-source heterogeneous data into an agricultural business application scene rule base for sensor data cleaning and dimension reduction processing, generating interest indexes, associating the interest indexes with the classification labels, storing association relations, and pushing the massive agricultural multi-source heterogeneous data of the associated interest indexes to a cache queue according to the classification labels;
further, in step S5, the association between the interest index and the classification label is a one-to-one relationship or a one-to-many relationship;
s6, setting the synchronization time of the interest indexes according to the interest indexes, reading all associated classification labels according to the interest indexes when the set synchronization time of the interest indexes is reached, extracting massive agricultural multi-source heterogeneous data of the associated interest indexes according to the classification labels, synchronizing the extracted massive agricultural multi-source heterogeneous data of the associated interest indexes to a historical database corresponding to the interest indexes to obtain an agricultural business application scene theme data set, and completing processing of the massive agricultural multi-source heterogeneous sensing data;
further, in step S6, different application scenes in the theme dataset of the agricultural service application scene are set to correspond to different interest indexes.
The intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data is characterized by constructing an agricultural data grading model and formulating an agricultural business application scene rule base; accessing massive multi-source heterogeneous data, completing data analysis by utilizing an agricultural data grading model, attaching classification labels to data sources, and pushing data into a cache queue; and pulling the cache data, finishing data cleaning according to the data cleaning rules in the agricultural business application scene rule base, generating interest indexes, associating the interest indexes with the classification labels, storing association relations, and pushing the cache queues of different topics by the data according to the classification labels. Setting different synchronization time for different interest indexes, avoiding data pressure caused by massive data, reading all associated classification labels according to the interest indexes after the set time is reached, completing cache data pulling according to the classification labels, synchronizing the data of the same interest index into a historical database of the interest index subject, and completing construction of an agricultural business application scene subject data set.
The intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data is convenient, quick, clear in flow, capable of saving protocol matching time, reducing trial-and-error cost and data access cost, and provides a new problem solving angle and thought reference for the field.
The second embodiment is as follows:
according to the method for intelligently accessing and processing massive agricultural multi-source heterogeneous sensing data, an agricultural data classification model link is constructed, the agricultural multi-source heterogeneous sensing data are classified and classified according to monitoring objects. The primary is respectively life information (L), environment information (E) and quality safety (Q), and the secondary refines the large category of the primary: the life information is further divided into plants (P) and animals (A), the environment information is further divided into water (W), soil (S), livestock (X) and weather (M), and the quality safety is further divided into smell (N), taste (T), hearing (H) and vision (V);
an agricultural business application scene rule base link is formulated, association rules are formulated according to agricultural business application scenes, and the business scenes can be divided into soil and fertilizer management, crop growth, pest control, agricultural product quality, safety monitoring and the like, and data cleaning rules are added or modified according to requirements;
and in the data access link, the massive agricultural multi-source heterogeneous data are intelligently matched with the protocol analysis program package by utilizing the agricultural data hierarchical model, so that data analysis is completed, and if the matched protocol analysis program package is not found in the process, the equipment connection is discarded and closed.
Data cleaning and dimension reduction: and (3) pulling the cache data according to classification labels in a classified manner, finishing data cleaning and dimension reduction processing according to rules of an agricultural business application scene rule base, generating interest indexes, associating the interest indexes with the classification labels, storing association relations, and pushing the cache queues of different topics according to the classification labels by the data.
Establishing agricultural business application scene theme data: setting different synchronization time for different interest indexes to avoid data pressure caused by massive data, reading all associated classification labels according to the interest indexes after the set time is reached, completing cache data pulling according to the classification labels, synchronizing the data of the same interest index into a historical database of the interest index subject, and completing construction of an agricultural business application scene subject data set.
And a third specific embodiment:
according to the intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data, an application scene is simulated, and access and processing processes of soil moisture content monitoring equipment and meteorological station equipment are simulated. Soil moisture content monitoring facilities data source: data1, meteorological station equipment data source: data2, 2 types of data sources are accessed, cleaned and stored. The implementation of the procedure is performed in accordance with the method:
the data of the equipment is input into an agricultural data grading model and a characteristic database, the equipment is connected with a platform to send first frame data, the data access of data1 is taken as an example, the characteristic value of the first frame data of the data1 is intercepted and successfully matched with the characteristic database in a protocol, the length is successfully matched, the matching degree verification is carried out, and if a plurality of matched protocols are obtained, the data analysis is carried out according to the protocol with high matching degree according to the matching degree verification.
The matching degree verification link specifically comprises the following steps:
data1 is accessed into the initial frame data C, ABCDEFG;
the first frame character string D inquired in the characteristic database: ABCHIJK;
wherein 11 characters are total, and: A. b, C, D, E, F, G, H, I, J, K, the two strings are converted into vectors in two 11-dimensional spaces, regardless of the association and order between them:
C：{1、1、1、1、1、1、1、0、0、0、0}
D：{1、1、1、0、0、0、0、1、1、1、1}
the matching degree is calculated as follows:
P＝sqrt(3)/(sqrt(7)×sqrt(7))＝0.2474358297
reserving 4-bit decimal for P, comparing the P with the highest matching degree field of the protocol characteristic database, and taking the highest value to store the highest matching degree field;
the protocol analysis is completed to obtain Data1 and Data2, the Data labeling link is carried out, classification labels E_S corresponding to the protocols in the feature database are read, classification labels E_S are pasted on the Data1, classification labels E_M are pasted on the Data2 according to the flow, cache Data1 is pushed to the topic E_S, and cache Data2 is pushed to the topic E_M;
pulling the cache Data1 and Data2, filtering abnormal values according to the rule of cleaning the agricultural business application scene rule base Data, wherein the time interval of the two Data is five minutes, extracting half-hour Data as analysis basis, and discarding other Data;
when the soil management subject data set is formulated, the data set synchronization time of the soil subject is set to be 24 hours, data of a classification label E_ S, E _ M, E _W corresponding to the soil is pulled regularly, and a history database of the soil subject is stored.
The specific embodiment IV is as follows:
the electronic equipment comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data when executing the computer program.
The computer device of the present application may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. And the processor is used for realizing the steps of the intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data when executing the computer program stored in the memory.
The processor may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
Fifth embodiment:
the computer readable storage medium is stored with a computer program, and the computer program realizes the intelligent access and processing method of the massive agricultural multi-source heterogeneous sensing data when being executed by a processor.
The computer readable storage medium of the present application may be any form of storage medium that is read by a processor of a computer device, including but not limited to a nonvolatile memory, a volatile memory, a ferroelectric memory, etc., and has a computer program stored thereon, and when the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned method for intelligently accessing and processing massive agricultural multi-source heterogeneous sensing data may be implemented.
The computer program comprises computer program code which may be in source code form, object code form, executable file or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Although the application has been described above with reference to specific embodiments, various modifications may be made and equivalents may be substituted for elements thereof without departing from the scope of the application. In particular, the features of the disclosed embodiments may be combined with each other in any manner so long as there is no structural conflict, and the exhaustive description of these combinations is not given in this specification solely for the sake of brevity and resource saving. Therefore, it is intended that the application not be limited to the particular embodiments disclosed herein, but that the application will include all embodiments falling within the scope of the appended claims.
Claims (10)
1. The intelligent access and processing method for the massive agricultural multi-source heterogeneous sensing data is characterized by comprising the following steps of:
s1, constructing an agricultural data grading model and setting a classification label;
s2, formulating a division rule, a sensor data cleaning rule and a sensor data association rule of a service scene, and constructing an agricultural service application scene rule base;
s3, carrying out data access and intelligent matching of protocol analysis program packages on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1 to complete data analysis, and obtaining analyzed massive agricultural multi-source heterogeneous data;
s4, calibrating the classification labels of the massive agricultural multi-source heterogeneous data analyzed in the step S3, pushing the massive agricultural multi-source heterogeneous data to a cache queue according to the classification labels, and obtaining massive agricultural multi-source heterogeneous data calibrated with the classification labels;
s5, extracting massive agricultural multi-source heterogeneous data of the calibrated classification labels obtained in the step S4 from the cache, inputting the massive agricultural multi-source heterogeneous data into an agricultural business application scene rule base for sensor data cleaning and dimension reduction processing, generating interest indexes, associating the interest indexes with the classification labels, storing association relations, and pushing the massive agricultural multi-source heterogeneous data of the associated interest indexes to a cache queue according to the classification labels;
s6, setting the synchronization time of the interest indexes according to the interest indexes, reading all associated classification labels according to the interest indexes when the set synchronization time of the interest indexes is reached, extracting massive agricultural multi-source heterogeneous data of the associated interest indexes according to the classification labels, synchronizing the extracted massive agricultural multi-source heterogeneous data of the associated interest indexes to a historical database corresponding to the interest indexes, obtaining an agricultural business application scene theme data set, and completing processing of the massive agricultural multi-source heterogeneous sensing data.
2. The intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 1, wherein the specific implementation method of the step S1 comprises the following steps:
s1.1, acquiring an agricultural Internet of things protocol, historical data and a protocol analysis program;
s1.2, constructing an agricultural data grading model, adopting redis as a characteristic database of the agricultural data grading model, and adopting a hash data structure for storing information of an agricultural Internet of things protocol;
s1.3, classifying different sensing data specified in an agricultural data classification model according to types, and setting classification labels;
s1.4, mapping the protocol name of the agricultural Internet of things protocol acquired in the step S1.1 with a protocol analysis program, adopting an identification bit of first frame data of the agricultural Internet of things protocol as a characteristic value, setting the characteristic value as the first 2 bytes of the first frame data of the agricultural Internet of things protocol, storing the characteristic value into a characteristic database of an agricultural data grading model, setting a key storage protocol name, and setting a key corresponding value field to comprise the characteristic value, the first frame length, the first frame character string, a classification label and the highest matching degree; and storing the agricultural Internet of things protocol name and the score by adopting a zipList data structure of Zset, and sequencing the access quantity of the characteristic values by using the score of the agricultural Internet of things protocol name.
3. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 2, wherein the step S1.3 is characterized in that the set classification labels comprise a primary classification label and a secondary classification label, the secondary classification label is refinement of the primary classification label, and the primary classification label is used alone or in combination with the secondary classification label;
the first-level classification tag comprises life information L, environment information E and quality safety Q;
the secondary classification label of the life information L comprises a plant P and an animal A;
the secondary classification label of the environment information E comprises a water body W, soil S, livestock X and weather M;
the secondary classification labels of the quality safety Q comprise smell N, taste T, hearing H and vision V.
4. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 3, wherein the specific implementation method of the step S2 comprises the following steps:
s2.1, making a division rule of a service scene: dividing an agricultural business application scene into a soil and fertilizer management scene, a crop growth scene, a pest control scene, an agricultural product quality and a safety monitoring scene;
the sensor data types related to the soil and fertilizer management scene comprise soil nutrient data, environment data and moisture data; the sensor data related to the crop growth scene includes environmental data; the sensing data related to the pest control scene comprises environmental data and pest history data; the sensor data related to the agricultural product quality and safety monitoring scene comprise environment real-time data and crop growth data;
s2.2, formulating a sensor data cleaning rule:
s2.2.1, setting a sensor data type floating point number to reserve two decimal places;
s2.2.2, setting a sensor data unit, wherein the sensor data unit keeps consistency;
s2.2.3, setting a range of sensor data and filtering abnormal data;
s2.2.4, setting the time interval of sensor data to be less than 10 minutes, adopting an extraction mode, selecting the time interval of 30 minutes or 1 hour for extracting data storage, and deleting the rest redundant data;
s2.3, formulating a sensor data association rule: the agricultural business application scene rule base comprises definition of interest indexes and corresponding relation between the interest indexes and classification labels;
s2.3.1, interest index definition: soil fertilizer management is soil, crop growth is crop pest control is insolation quality and safety monitoring of agricultural products is quality;
s2.3.2, the correspondence between interest indexes and classification labels is: the classification label corresponding to the oil comprises: the class labels corresponding to E_ S, E _ M, E _W and crop comprise: the class labels corresponding to L_ P, E _M and insert include: the class labels corresponding to the quality of E_ M, L _A comprise: l_ P, Q _v.
5. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 4, wherein the specific implementation method of the step S3 comprises the following steps:
s3.1, carrying out data access on the collected massive agricultural multi-source heterogeneous data by utilizing the agricultural data grading model constructed in the step S1, and establishing connection between the agricultural data grading model and the Internet of things equipment adopting a TCP or UDP protocol;
s3.2, intercepting 1 byte and 2 bytes of first frame data sent by the accessed massive agricultural multi-source heterogeneous data by the agricultural data grading model, searching in a characteristic database protocol hash table of the agricultural data grading model according to the sequence of protocol names in a characteristic database of the agricultural data grading model, comparing the characteristic values in the characteristic database of the agricultural data grading model, if the characteristic values are successfully matched and the length of the first frame data is successfully matched, reading a first frame character string of a protocol in the characteristic database of the agricultural data grading model, checking the matching degree, finding a protocol analysis program packet corresponding to the protocol with the highest matching degree, completing data analysis of the protocol, and updating the score corresponding to the protocol in the characteristic database; the calculation formula of the matching degree P (C, D) check is as follows:
P(C，D)＝sqrt(C×D)/(|C|×|D|)
wherein C is the first frame data character string of the access equipment, and D is the first frame character string of the protocol read in the characteristic database.
6. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 5, wherein in step S4, the massive agricultural multi-source heterogeneous data analyzed in step S3 is classified by reading classification labels stored in a protocol table in a feature database of a matched agricultural data classification model, and determining the classification labels as the analyzed massive agricultural multi-source heterogeneous data.
7. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 6, wherein in step S5, the interest index and the classification label are associated in a one-to-one relationship or a one-to-many relationship.
8. The intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to claim 7, wherein in step S6, different application scenes in the agricultural business application scene theme dataset are set to correspond to different interest indexes.
9. The electronic device is characterized by comprising a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the intelligent accessing and processing method for massive agricultural multi-source heterogeneous sensing data according to any one of claims 1-8 when executing the computer program.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements a method for intelligent access and processing of massive agricultural multi-source heterogeneous sensor data according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310977860.XA CN117009334B (en) | 2023-08-04 | 2023-08-04 | Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310977860.XA CN117009334B (en) | 2023-08-04 | 2023-08-04 | Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117009334A true CN117009334A (en) | 2023-11-07 |
| CN117009334B CN117009334B (en) | 2024-03-01 |
Family
ID=88565102
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310977860.XA Active CN117009334B (en) | 2023-08-04 | 2023-08-04 | Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117009334B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591025A (en) * | 2023-11-27 | 2024-02-23 | 海南榕树家信息科技有限公司 | Multi-source heterogeneous data processing system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200193324A1 (en) * | 2018-12-14 | 2020-06-18 | The Regents Of The University Of Michigan | System and method for unifying heterogenous datasets using primitives |
| CN112347093A (en) * | 2020-11-05 | 2021-02-09 | 哈尔滨航天恒星数据系统科技有限公司 | Method for facilitating cleaning, integrating and storing of mass multi-source heterogeneous data |
| CN115794929A (en) * | 2023-01-13 | 2023-03-14 | 中企云链（北京）金融信息服务有限公司 | Data management system and data management method for data mart |
-
2023
- 2023-08-04 CN CN202310977860.XA patent/CN117009334B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200193324A1 (en) * | 2018-12-14 | 2020-06-18 | The Regents Of The University Of Michigan | System and method for unifying heterogenous datasets using primitives |
| CN112347093A (en) * | 2020-11-05 | 2021-02-09 | 哈尔滨航天恒星数据系统科技有限公司 | Method for facilitating cleaning, integrating and storing of mass multi-source heterogeneous data |
| CN115794929A (en) * | 2023-01-13 | 2023-03-14 | 中企云链（北京）金融信息服务有限公司 | Data management system and data management method for data mart |
Non-Patent Citations (1)
| Title |
|---|
| 陈世超;崔春雨;张华;马戈;朱凤华;商秀芹;熊刚;: "制造业生产过程中多源异构数据处理方法综述", 大数据, no. 05, 23 June 2020 (2020-06-23) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591025A (en) * | 2023-11-27 | 2024-02-23 | 海南榕树家信息科技有限公司 | Multi-source heterogeneous data processing system |
| CN117591025B (en) * | 2023-11-27 | 2024-05-10 | 海南榕树家信息科技有限公司 | Multi-source heterogeneous data processing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117009334B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117009334B (en) | Intelligent access and processing method for massive agricultural multi-source heterogeneous sensing data, electronic equipment and storage medium | |
| CN111797835A (en) | Disease identification method, disease identification device and terminal equipment | |
| CN112650923A (en) | Public opinion processing method and device for news events, storage medium and computer equipment | |
| CN109063660B (en) | Crop identification method based on multispectral satellite image | |
| CN111563759B (en) | Identification and analysis system for agricultural product traceability process based on AI technology | |
| CN116401417B (en) | Hierarchical storage method based on massive agricultural product data | |
| CN112445690B (en) | Information acquisition method and device and electronic equipment | |
| KR20210109575A (en) | Information processing devices and information processing systems | |
| CN115618021A (en) | Method and device for recommending suitable planting area of crop variety | |
| Machado-de-Souza et al. | Local drivers of the structure of a tropical bird-seed dispersal network | |
| CN113220827B (en) | Construction method and device of agricultural corpus | |
| CN113434644B (en) | Agricultural technology knowledge service method and system | |
| CN108551473B (en) | Agricultural product communication method and device based on visual agriculture | |
| TWI692737B (en) | Agricultural data service server, method and system thereof | |
| Chatterjee et al. | Design and Development of Smart Farming using ML and IoT in India | |
| CN117541084B (en) | Method and system for predicting yield of quinoa in grouting period | |
| CN117745148B (en) | Multi-source data-based rice stubble flue-cured tobacco planting quality evaluation method and system | |
| CN117077004B (en) | Species identification method, system, device and storage medium | |
| US11580116B2 (en) | Viewport location based method and apparatus for generation and promotion of type ahead results in a multi-source agricultural parcel search | |
| CN117114917B (en) | AI prediction processing method and system applied to digital agriculture | |
| CN107886297A (en) | Plant information management method and management system | |
| US20220012234A1 (en) | Method and apparatus for generation and promotion of type ahead results in a multi-source agricultural parcel search | |
| US20220012253A1 (en) | Method and apparatus for rapid search for agricultural parcels and generation of relevant search results | |
| CN114219972A (en) | Image classification method and device, storage medium and electronic device | |
| CN116757727A (en) | Customer grouping processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |