WO2023004881A1

WO2023004881A1 - Smart agriculture aiot distributed big data storage platform

Info

Publication number: WO2023004881A1
Application number: PCT/CN2021/111626
Authority: WO
Inventors: 刘天琼
Original assignee: 深圳市爱云信息科技有限公司
Priority date: 2021-07-28
Filing date: 2021-08-09
Publication date: 2023-02-02
Also published as: CN113297196A

Abstract

A smart agricultural AIOT distributed big data storage platform, comprising a smart agricultural big data basic support platform and a smart agricultural data center; the smart agricultural big data basic support platform is used for managing and supporting the full life cycle of a huge volume of big data; and the smart agricultural data center is used for planning basic service data stored in the smart agricultural big data basic support platform, so as to ensure that the smart agricultural AIOT distributed big data storage platform has a distributed architecture effect. The present invention employs a distributed architecture, and provides, by providing management and support for the full life cycle of a huge volume of big data, a data model basis for a smart agricultural big data service platform established on the basis of artificial intelligence and Internet information sharing.

Description

Smart Agriculture AIOT Distributed Big Data Storage Platform

technical field

The invention relates to the technical field of smart agriculture, in particular to an AIOT distributed big data storage platform for smart agriculture.

Background technique

Agriculture is the foundation of the national economy. With the improvement of agricultural industrialization and scale, and Internet of Things technology, cloud computing technology, big data technology, geographic information system, remote sensing and global positioning system technology are more and more widely used in the agricultural field , the traditional agricultural farming mode has gradually exposed some shortcomings, mainly reflected in: First, the severe island of agricultural information. The agricultural sector is managed in sections, and the departmental application systems are mostly vertical and isolated systems, with a low degree of information sharing. Second, the comprehensive utilization rate of data is not high. Agricultural data involves many types of data, inconsistent data structure, and uneven data quality. The workload of data analysis and sorting is heavy, and the comprehensive utilization rate of data is not high. Third, market supply and marketing information is asymmetric. Agricultural products fluctuate greatly due to the influence of the market, but farmers have limited access to market information, and it is difficult to grasp the latest market information in a timely manner, resulting in a disconnection of information between agricultural producers and consumers. Fourth, extensive management. my country's agricultural production is relatively scattered, and it is difficult to collect and analyze agricultural-related data. It is difficult to achieve accuracy, efficiency and timely processing, resulting in inaccurate agricultural decision-making.

The above-mentioned disadvantages of the traditional agricultural farming mode lead to the inability of the traditional agricultural information system to solve the following problems, which can only be solved by the integration of modern information technologies such as the Internet of Things, cloud computing, big data and "3S" technology with agriculture : First, how to realize data sharing and exchange among various agricultural departments and other government departments, so as to achieve the purpose of inter-departmental information interconnection; second, how to make scientific decisions through agricultural product market information and climate and meteorological information, and guide farmers to rationalize crops Planting, to avoid the phenomenon of "there is no price but no market" or "there is no price when there is a market", and realize the balance of supply and marketing of agricultural products; the third is how to effectively monitor the crop environment through IoT sensors, so as to achieve the best growth environment for crop growth and Fertilization scientifically enables crops to obtain relatively high yields, thereby increasing farmers' income; fourth, how to identify agricultural products and inputs through QR codes, bar codes, etc., to achieve effective management and quality and safety traceability of agricultural products and inputs; fifth is How to use the Internet to develop agricultural e-commerce and agricultural leisure tourism, help farmers broaden the sales channels of agricultural products, solve the problems of unsalable agricultural products and increase income.

To sum up, the use of artificial intelligence technology and Internet information sharing to establish a smart agricultural big data service platform has become the development trend of agricultural modernization. The smart agricultural big data service platform provides the data model foundation.

Contents of the invention

The purpose of the present invention is to provide a smart agricultural AIOT distributed big data storage platform, adopt a distributed architecture, and provide intelligent agricultural big data based on artificial intelligence and Internet information sharing by providing management and support for the entire life cycle of massive big data The service platform provides the data model foundation.

In order to achieve the above purpose, the technical solution adopted in the present invention is: the present invention provides a smart agricultural AIOT distributed big data storage platform, including a smart agricultural big data basic support platform and a smart agricultural data middle platform. The data basic support platform is used to manage and support the full life cycle of massive big data, and the smart agricultural data center is used to plan the basic business data stored on the smart agricultural big data basic support platform to ensure that the The smart agriculture AIOT distributed big data storage platform plays a role of distributed architecture; the smart agricultural big data basic support platform and the smart agricultural data middle platform realize data exchange through computer application program interfaces and networks;

The smart agricultural big data basic support platform includes a data acquisition system, a data management system and a data storage system, the data acquisition system is used for data collection, and the data management system is used for fusing the data collected by the data acquisition system and data governance, the data storage system is used to store the data analyzed and processed by the data governance system; the data acquisition system includes a structured data acquisition module, an unstructured data acquisition module and a real-time data acquisition module, The structured data collection module is used to collect structured data, the unstructured data module is used to collect unstructured data, and the real-time data collection module is used to collect real-time data; the data The governance system includes a data extraction module, a data cleaning module, a data conversion module and a data loading module, the data extraction module is used to obtain business data from the data collected by the data acquisition system, and the data cleaning module is used to convert the The defective data acquired by the data extraction module is corrected and normalized to meet the required data quality standards, and the data conversion module is used to convert the data collected by the data acquisition system and the data processed by the data extraction module to obtain In line with the requirements of the data warehouse model, the data loading module is used to store the data converted by the data conversion module into the target database; the data storage system includes a business database and a distributed massive spatial database, and the business database is used for Store business data related to agricultural resources and agricultural decision-making, and the distributed massive spatial database is used to store remote sensing image data, video data, and IoT sensor device data;

The smart agricultural data center includes a shared resource library, a smart agricultural theme resource library, and a basic resource library. The shared resource database meets the public and institutional sharing needs for agricultural data through a shared exchange platform. The smart agricultural theme resource library It is a resource library of different agricultural themes customized according to applications and needs. The basic resource library is used to store data shared and used by multiple systems; the shared resource library includes the full-cycle shared resource library of pollution-free products, and the shared resources library, seed demand shared resource library, characteristic agricultural product supply shared resource library, and digital agricultural government affairs shared resource library. The shared resource database is used to provide seed demand release and query functions for individuals and manufacturers engaged in agricultural production, and the shared resource database for characteristic agricultural product supply is used to provide characteristic agricultural product supply query functions for individuals and manufacturers engaged in characteristic agricultural product transactions. The digital agricultural government sharing resource library is used to provide government agencies with the release and query functions of agriculture-related government information; the smart agriculture theme resource library includes agricultural output theme resource library, industrial layout theme resource library, environmental monitoring theme resource library, Agricultural product safety theme resource bank, agricultural product logistics theme resource bank, fishery and aquatic product theme resource bank, animal husbandry theme resource bank, disease and pest control theme resource bank, soil fertility theme resource bank, the agricultural output theme resource bank is used to store information related to agricultural output data and provide query functions, the industrial layout subject resource library is used to store data related to agricultural industrial layout and provide query functions, and the environmental monitoring subject resource library is used to store environmental monitoring data related to agricultural production and provide query functions function, the agricultural product safety theme resource library is used to store data related to agricultural product safety and provide query functions, the agricultural product logistics theme resource library is used to store data related to agricultural product logistics and provide query functions, and the fishery and aquatic product theme resources The library is used to store data related to fishery and aquatic products and provide query functions. The animal husbandry theme resource library is used to store data related to animal husbandry and provide query functions. The pest control theme resource library is used to store information related to disease and pest control. data and provide query function, the soil fertility subject resource library is used to store data related to soil fertility and provide query function; the basic resource library includes administrative unit basic resource library, basic terrain basic resource library, agricultural enterprise basic resource Library, basic resource library of agricultural resources, and basic resource library of image resources. The basic resource library of administrative units is used to store the data of administrative units related to agriculture and perform primary common processing on them for other system calls. The basic resource of basic terrain The library is used to store basic terrain data related to agriculture and perform primary common processing on it for other system calls. The agricultural enterprise basic resource library is used to store agricultural enterprise data and perform primary common processing on it for other system calls. The basic resource library of agricultural resources is used to store agricultural resources and carry out primary commonality Processing for mobilization by other systems, the image resources basic resource library is used to store image resources related to agriculture and perform primary common processing on them for mobilization by other systems.

Further, the smart agricultural data center also includes a near-source acquisition database, and the near-source acquisition database is modeled according to the source system, so as to keep the original appearance of the business data obtained from the data storage system as much as possible.

Further, the shared resource library is an external shared resource library established based on relevant data obtained from the near-source acquisition database through a shared platform, using a data service model, on the basis of ensuring information security; the intelligent agriculture theme The resource library uses ETL tools to process the relevant data obtained from the near-source collection database for generality, application-oriented, and customized on demand; the basic resource library performs primary commonality processing on the relevant data obtained from the near-source collection database And extract common attributes.

Further, the structured data includes data obtained from the smart agricultural resource management system, smart agricultural production management system, smart agricultural supply chain management system, smart agricultural party building management system, and government affairs information from smart cities; the structured data Stored in structured databases and distributed databases, which can be transmitted in real time or offline through data interface protocols; the unstructured data includes satellite remote sensing image data, spatial geographic data, intelligent IoT sensor data and frequency data, and is distributed using Hadoop clusters The storage of massive data; the real-time data includes sensors, remote sensing image data and massive concurrent data.

Further, the business data is land, water, climate, population, agricultural economy and agricultural resource data as administrative units of counties and townships, as well as basic farmland demarcation and demarcation, standard farmland, land second-round contracting, and expert knowledge of agricultural decision-making Soil nutrients, heavy metals, and pesticide residues in the database, cultivated land fertility survey and quality evaluation research; the business database formulates supporting attribute data collection standards and specifications while formulating the metadata database, data dictionary, and data table structure series; the distributed The architecture of the massive spatial database adopts the distributed file management system based on the Hadoop system and the MPP database designed based on the MPP+Share-nothing technology.

Further, the data extraction of the data extraction module includes the following situations: if the database management systems between the business operation database and the data warehouse are exactly the same, only need to establish a corresponding connection relationship to use the ETL tool to directly access, or call the corresponding SQL statements or stored procedures; if the data warehouse system and the database management system of the business operation database are different, use ETL tools to export them into text files or Excel files, and then perform unified data extraction; if the amount of data to be extracted is very large , using the incremental extraction method, in the form of marker bits or timestamps, before each extraction, first judge whether it is to extract marker bits or the latest time, and then extract the data from the data source.

Further, the defect data selected by the data cleaning module includes repeated values, missing data, data errors, data range confusion, dirty data, and data inconsistencies; repeated values are not unique, and many values represent the same Meaning; data range confusion means that the same value will be applied to different occasions, representing different meanings.

Further, the data cleaning process of the data cleaning module includes the following steps:

S01: Define business data sources, identify data sources that meet the requirements, and decide when to perform data cleaning;

S02: Analyze the business data source, analyze whether the data of the data source conforms to the rules and definitions of the business, and whether there is an abnormal data structure;

S03: Standardize the data, define the data in a standardized format, and convert it;

S04: Correct the wrong data through business rules, define the standard of whether it is correct data, and determine how to deal with the wrong data;

S05: Merge data, merge multiple data belonging to the same entity, and there should be a deduplication function when merging;

S06: Summarize the types of data errors, improve the integrity and correctness of the cleaning program by summarizing the types of data errors, thereby reducing the possibility of major data problems.

Further, the conversion process of the data conversion module includes the following steps:

S11: Handling of null values: If some fields have null values captured during the conversion process, the null values need to be replaced with certain data or loaded directly without any conversion;

S12: Standardize the data format: standardize and unify the data format according to the data type of each field in the business data source, for example, uniformly convert the numeric type into a string type;

S13: Split or merge fields according to business requirements;

S14: Replace missing data: replace missing data according to business requirements;

S15: Filter data according to business rules;

S16: Conversion of data uniqueness according to the coding table: data conversion is performed according to the business specification formulated by the coding table to realize the consistency of the internal data of the data warehouse system.

Further, the data loading strategy of the data loading module includes the loading method of the timestamp, the loading method of the full table comparison, the loading method by reading the log table, and the loading method after deleting the entire table; the loading of the timestamp The method is to add a timestamp field to the table of the source system, compare the current time of the system with the value of the timestamp, and decide which business data needs to be extracted, which can realize incremental loading of data, which is a relatively common loading method; full The loading method of table comparison is to compare each piece of data with all records of the target table before data loading, and judge whether the data is updated or inserted according to whether the primary key values are the same. When the amount of data is relatively large, there is The shortcomings of long time consumption and low efficiency usually also improve the comparison of the whole table, and incremental extraction is performed in the form of slowly changing dimensions such as version number and tag field; the method of loading by reading the log table is when the source data table occurs When changing, the information of the log table is constantly updated, and the information of the log table is used as a basis for data loading. The maintenance of the log table is relatively troublesome, and there will be certain risks; the way to load after deleting the entire table is to delete the data before loading the data. All the data in the target table, and then load all the data, but the incremental loading of data cannot be achieved, the efficiency is low, and the implementation method is relatively simple.

Compared with the prior art, the beneficial effect of the present invention is that the intelligent agricultural AIOT distributed big data storage platform provided by the present invention includes the intelligent agricultural big data basic support platform and the intelligent agricultural data middle platform, and the intelligent agricultural big data basic support platform It is used to manage and support the full life cycle of massive big data. The smart agricultural data center is used to plan the basic business data stored on the smart agricultural big data basic support platform to ensure that the smart agricultural AIOT distributed big The data storage platform plays the role of a distributed architecture; the present invention adopts a distributed architecture to provide data models for the smart agricultural big data service platform based on artificial intelligence and Internet information sharing by providing management and support for the entire life cycle of massive big data Base.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 is a system structure diagram of the intelligent agriculture AIOT distributed big data storage platform provided by the embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

In the drawings of this embodiment, the same or similar symbols correspond to the same or similar components; The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, or in a specific orientation. Construction and operation, so the words describing the positional relationship in the drawings are only for illustrative purposes, and should not be construed as limitations on this patent. Those of ordinary skill in the art can understand the specific meanings of the above terms according to specific situations.

The technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Referring to Fig. 1, the intelligent agriculture AIOT distributed big data storage platform provided by the present invention includes a smart agricultural big data basic support platform and a smart agricultural data middle platform. Life cycle management and support, the smart agricultural data center is used to plan the basic business data stored in the smart agricultural big data basic support platform, so as to ensure that the smart agricultural AIOT distributed big data storage platform can play a distributed role Architectural role; the smart agricultural big data basic support platform and the smart agricultural data middle platform realize data exchange through a computer application program interface and a network;

The smart agricultural data center includes a shared resource library, a smart agricultural theme resource library, and a basic resource library. The shared resource database meets the public and institutional sharing needs for agricultural data through a shared exchange platform. The smart agricultural theme resource library It is a resource library of different agricultural themes customized according to applications and needs. The basic resource library is used to store data shared and used by multiple systems; the shared resource library includes the full-cycle shared resource library of pollution-free products, and the shared resources library, seed demand shared resource library, characteristic agricultural product supply shared resource library, and digital agricultural government affairs shared resource library. The shared resource database is used to provide seed demand release and query functions for individuals and manufacturers engaged in agricultural production, and the shared resource database for characteristic agricultural product supply is used to provide characteristic agricultural product supply query functions for individuals and manufacturers engaged in characteristic agricultural product transactions. The digital agricultural government sharing resource library is used to provide government agencies with the release and query functions of agriculture-related government information; the smart agriculture theme resource library includes agricultural output theme resource library, industrial layout theme resource library, environmental monitoring theme resource library, Agricultural product safety theme resource bank, agricultural product logistics theme resource bank, fishery and aquatic product theme resource bank, animal husbandry theme resource bank, disease and pest control theme resource bank, soil fertility theme resource bank, the agricultural output theme resource bank is used to store information related to agricultural output data and provide query functions, the industrial layout subject resource library is used to store data related to agricultural industrial layout and provide query functions, and the environmental monitoring subject resource library is used to store environmental monitoring data related to agricultural production and provide query functions function, the agricultural product safety theme resource library is used to store data related to agricultural product safety and provide query functions, the agricultural product logistics theme resource library is used to store data related to agricultural product logistics and provide query functions, and the fishery and aquatic product theme resources The library is used to store data related to fishery and aquatic products and provide query functions. The animal husbandry theme resource library is used to store data related to animal husbandry and provide query functions. The pest control theme resource library is used to store information related to disease and pest control. data and provide query function, the soil fertility subject resource library is used to store data related to soil fertility and provide query function; the basic resource library includes administrative unit basic resource library, basic terrain basic resource library, agricultural enterprise basic resource Library, basic resource library of agricultural resources, and basic resource library of image resources. The basic resource library of administrative units is used to store the data of administrative units related to agriculture and perform primary common processing on them for other system calls. The basic resource of basic terrain The library is used to store basic terrain data related to agriculture and perform primary common processing on it for other system calls. The agricultural enterprise basic resource library is used to store agricultural enterprise data and perform primary common processing on it for other system calls. The basic resource library of agricultural resources is used to store agricultural resources and carry out primary commonality Processing for mobilization by other systems, the image resource base resource library is used to store image resources related to agriculture and perform primary common processing on them for mobilization by other systems.

The smart agriculture AIOT distributed big data storage platform provided by the above technical solutions includes the smart agricultural big data basic support platform and the smart agricultural data middle platform. Support, the smart agricultural data center is used to plan the basic business data stored on the smart agricultural big data basic support platform, so as to ensure that the smart agricultural AIOT distributed big data storage platform plays a role in distributed architecture; the present invention adopts distributed By providing full life cycle management and support for massive big data, it provides a data model foundation for the smart agricultural big data service platform based on artificial intelligence and Internet information sharing.

As an embodiment of the present invention, the smart agricultural data center further includes a near-source acquisition database, and the near-source acquisition database is modeled according to the source system, so as to keep the original appearance of the business data obtained from the data storage system as much as possible.

As an embodiment of the present invention, the shared resource library is an external shared resource library established on the basis of ensuring information security, using a data service mode through a shared platform, and relying on relevant data obtained from the near-source collection database ; The intelligent agriculture subject resource library adopts ETL tools to process the relevant data obtained from the near-source collection database in a common way, and is application-oriented and customized on demand; Relevant data undergoes primary common processing and refines common attributes.

As an embodiment of the present invention, the structured data includes data obtained from the smart agricultural resource management system, smart agricultural production management system, smart agricultural supply chain management system, smart agricultural party building management system, and government affairs information from smart cities The structured data is stored in a structured database and a distributed database, and can be transmitted in real time or offline through a data interface protocol; the unstructured data includes satellite remote sensing image data, spatial geographic data, intelligent IoT sensor data and frequency data , using Hadoop clusters to store distributed massive data; the real-time data includes sensors, remote sensing image data, and massive concurrent data.

As an embodiment of the present invention, the business data is land, water, climate, population, agricultural economy and agricultural resource data as well as basic farmland demarcation and demarcation, standard farmland, land two rounds Soil nutrients, heavy metals and pesticide residues in contracting, agricultural decision-making expert knowledge base, cultivated land fertility survey and quality evaluation research; the business database formulates supporting attribute data collection standards and Specification; the architecture of the distributed massive spatial database adopts the distributed file management system based on the Hadoop system, and the MPP database based on the MPP+Share-nothing technology design.

As an embodiment of the present invention, the data extraction of the data extraction module includes the following situations: if the database management systems between the business operation database and the data warehouse are completely the same, only need to establish a corresponding connection relationship to use the ETL tool Direct access, or call the corresponding SQL statement or stored procedure; if the database management system of the data warehouse system and the business operation database are different, use the ETL tool to export it into a text file or Excel file, and then perform unified data extraction; if necessary The amount of extracted data is very large. The incremental extraction method is used in the form of marker bits or timestamps. Before each extraction, it is first judged whether it is the extraction marker bit or the latest time, and then the data from the data source is extracted.

As an embodiment of the present invention, the defect data selected by the data cleaning module includes numerical repetition, data missing, data error, data range confusion, dirty data, and data inconsistency; numerical repetition is a standard that is not unique, Many values represent the same meaning; data range confusion means that the same value will be applied to different occasions and represent different meanings.

Specifically, the data cleaning process of the data cleaning module includes the following steps:

Specifically, the conversion process of the data conversion module includes the following steps:

S13: Split or merge fields according to business requirements;

S15: Filter data according to business rules;

Specifically, the data loading strategy of the data loading module includes the loading mode of the timestamp, the loading mode of the full table comparison, the loading mode by reading the log table, and the loading mode after the full table is deleted; the loading of the timestamp The method is to add a timestamp field to the table of the source system, compare the current time of the system with the value of the timestamp, and decide which business data needs to be extracted, which can realize incremental loading of data, which is a relatively common loading method; full The loading method of table comparison is to compare each piece of data with all records of the target table before data loading, and judge whether the data is updated or inserted according to whether the primary key values are the same. When the amount of data is relatively large, there is The shortcomings of long time consumption and low efficiency usually also improve the comparison of the whole table, and incremental extraction is performed in the form of slowly changing dimensions such as version number and tag field; the method of loading by reading the log table is when the source data table occurs When changing, the information of the log table is constantly updated, and the information of the log table is used as a basis for data loading. The maintenance of the log table is relatively troublesome, and there will be certain risks; the way to load after deleting the entire table is to delete the data before loading the data. All the data in the target table, and then load all the data, but the incremental loading of data cannot be achieved, the efficiency is low, and the implementation method is relatively simple.

Preferably, the implementation of all modules involved in the technical solution of the present invention adopts open, mature, open source program architecture and program codes, and the business processes, business terms and implemented functions involved in the technical solution of the present invention are all Based on the common knowledge in the field, those skilled in the art can easily adopt the existing and public program architecture and program codes to implement according to the description of the technical solution.

The embodiment of the present invention has been described in detail above, but the creation of the present invention is not limited to this embodiment. Those skilled in the art can also make many equivalent modifications or replacements without violating the spirit of the present invention. These equivalent modifications or replacements are all included in the scope of protection defined by the claims of the present application.

Claims

The smart agriculture AIOT distributed big data storage platform is characterized in that it includes a smart agricultural big data basic support platform and a smart agricultural data middle platform, and the smart agricultural big data basic support platform is used to manage the entire life cycle of massive big data and support, the smart agricultural data center is used to plan the basic business data stored on the smart agricultural big data basic support platform, so as to ensure that the smart agricultural AIOT distributed big data storage platform plays a role in distributed architecture; The smart agricultural big data basic support platform and the smart agricultural data middle platform realize data exchange through a computer application program interface and a network;

The smart agricultural big data basic support platform includes a data acquisition system, a data management system and a data storage system, the data acquisition system is used for data collection, and the data management system is used for fusing the data collected by the data acquisition system and data governance, the data storage system is used to store the data analyzed and processed by the data governance system; the data acquisition system includes a structured data acquisition module, an unstructured data acquisition module and a real-time data acquisition module, The structured data collection module is used to collect structured data, the unstructured data module is used to collect unstructured data, and the real-time data collection module is used to collect real-time data; the data The governance system includes a data extraction module, a data cleaning module, a data conversion module and a data loading module, the data extraction module is used to obtain business data from the data collected by the data acquisition system, and the data cleaning module is used to convert the The defective data acquired by the data extraction module is corrected and normalized to meet the required data quality standards, and the data conversion module is used to convert the data collected by the data acquisition system and the data processed by the data extraction module to obtain In line with the requirements of the data warehouse model, the data loading module is used to store the data converted by the data conversion module into the target database; the data storage system includes a business database and a distributed massive spatial database, and the business database is used for Store business data related to agricultural resources and agricultural decision-making, and the distributed massive spatial database is used to store remote sensing image data, video data, and IoT sensor device data;

The smart agricultural data center includes a shared resource library, a smart agricultural theme resource library, and a basic resource library. The shared resource database meets the public and institutional sharing needs for agricultural data through a shared exchange platform. The smart agricultural theme resource library It is a resource library of different agricultural themes customized according to applications and needs. The basic resource library is used to store data shared and used by multiple systems; the shared resource library includes the full-cycle shared resource library of pollution-free products, and the shared resources library, seed demand shared resource library, characteristic agricultural product supply shared resource library, and digital agricultural government affairs shared resource library. The shared resource database is used to provide seed demand release and query functions for individuals and manufacturers engaged in agricultural production, and the shared resource database for characteristic agricultural product supply is used to provide characteristic agricultural product supply query functions for individuals and manufacturers engaged in characteristic agricultural product transactions. The digital agricultural government sharing resource library is used to provide government agencies with the release and query functions of agriculture-related government information; the smart agriculture theme resource library includes agricultural output theme resource library, industrial layout theme resource library, environmental monitoring theme resource library, Agricultural product safety theme resource bank, agricultural product logistics theme resource bank, fishery and aquatic product theme resource bank, animal husbandry theme resource bank, disease and pest control theme resource bank, soil fertility theme resource bank, the agricultural output theme resource bank is used to store information related to agricultural output data and provide query functions, the industrial layout subject resource library is used to store data related to agricultural industrial layout and provide query functions, and the environmental monitoring subject resource library is used to store environmental monitoring data related to agricultural production and provide query functions function, the agricultural product safety theme resource library is used to store data related to agricultural product safety and provide query functions, the agricultural product logistics theme resource library is used to store data related to agricultural product logistics and provide query functions, and the fishery and aquatic product theme resources The library is used to store data related to fishery and aquatic products and provide query functions. The animal husbandry theme resource library is used to store data related to animal husbandry and provide query functions. The pest control theme resource library is used to store information related to disease and pest control. data and provide query function, the soil fertility subject resource library is used to store data related to soil fertility and provide query function; the basic resource library includes administrative unit basic resource library, basic terrain basic resource library, agricultural enterprise basic resource Library, basic resource library of agricultural resources, and basic resource library of image resources. The basic resource library of administrative units is used to store the data of administrative units related to agriculture and perform primary common processing on them for other system calls. The basic resource of basic terrain The library is used to store basic terrain data related to agriculture and perform primary common processing on it for other system calls. The agricultural enterprise basic resource library is used to store agricultural enterprise data and perform primary common processing on it for other system calls. The basic resource library of agricultural resources is used to store agricultural resources and carry out primary commonality Processing for mobilization by other systems, the image resource base resource library is used to store image resources related to agriculture and perform primary common processing on them for mobilization by other systems.
According to the smart agriculture AIOT distributed big data storage platform according to claim 1, it is characterized in that, the smart agriculture data center also includes a near-source acquisition database, and the near-source acquisition database is modeled according to the source system to keep as much as possible The original appearance of business data obtained from the data storage system.
According to the intelligent agriculture AIOT distributed big data storage platform according to claim 2, it is characterized in that, the shared resource library adopts a data service mode through a shared platform on the basis of ensuring information security, relying on the resources from the near source An external shared resource library established by collecting relevant data obtained from the database; the intelligent agricultural theme resource library uses ETL tools to process the relevant data obtained from the near-source collection database for general processing, application-oriented, and customized on demand; the basis The resource library performs primary common processing and refines common attributes on the related data obtained from the near-source collection database.
According to claim 1, the smart agriculture AIOT distributed big data storage platform is characterized in that, the structured data includes information from smart agricultural resource management system, smart agricultural production management system, smart agricultural supply chain management system, smart agricultural party building The management system obtains data and government affairs information from smart cities; the structured data is stored in structured databases and distributed databases, and can be transmitted in real time or offline through data interface protocols; the unstructured data includes satellite remote sensing image data, Spatial geographic data, intelligent Internet of Things sensor data, and frequency data use Hadoop clusters for distributed massive data storage; the real-time data includes sensors, remote sensing image data, and massive concurrent data.
According to claim 1, the smart agriculture AIOT distributed big data storage platform is characterized in that the business data is land, water, climate, population, agricultural economy and agricultural resource data and basic Soil nutrients, heavy metals and pesticide residues in farmland zoning and delimitation, standard farmland, second-round land contracting, agricultural decision-making expert knowledge base, cultivated land fertility survey and quality evaluation research; The table structure series formulates supporting attribute data collection standards and specifications at the same time; the architecture of the distributed massive spatial database adopts the distributed file management system based on the Hadoop system and the MPP database designed based on the MPP+Share-nothing technology.
The AIOT distributed big data storage platform for smart agriculture according to claim 1, wherein the data extraction of the data extraction module includes the following situations: if the database management systems between the business operation database and the data warehouse are completely the same, You only need to establish the corresponding connection relationship to use the ETL tool to directly access, or call the corresponding SQL statement or stored procedure; if the database management system of the data warehouse system and the business operation database are different, use the ETL tool to export it into a text file or Excel files, and then perform unified data extraction; if the amount of data to be extracted is very large, use incremental extraction in the form of marker bits or timestamps. Before each extraction, first determine whether it is to extract marker bits or the current latest time, and then extract the data from the data source.
The smart agriculture AIOT distributed big data storage platform according to claim 1, wherein the defect data selected by the data cleaning module includes repeated values, missing data, data errors, confusion of data ranges, existence of dirty data, and data inconsistencies In these cases, repeated values mean that the standard is not unique, and many values represent the same meaning; data range confusion means that the same value will be applied to different occasions and represent different meanings.
The smart agriculture AIOT distributed big data storage platform according to claim 6, wherein the data cleaning process of the data cleaning module comprises the following steps:

S01: Define business data sources, identify data sources that meet the requirements, and decide when to perform data cleaning;

S02: Analyze the business data source, analyze whether the data of the data source conforms to the rules and definitions of the business, and whether there is an abnormal data structure;

S03: Standardize the data, define the data in a standardized format, and convert it;

S04: Correct the wrong data through business rules, define the standard of whether it is correct data, and determine how to deal with the wrong data;

S05: Merge data, merge multiple data belonging to the same entity;

S06: Summarize the types of data errors, improve the integrity and correctness of the cleaning program by summarizing the types of data errors, thereby reducing the possibility of major data problems.
The smart agriculture AIOT distributed big data storage platform according to claim 1, wherein the conversion process of the data conversion module comprises the following steps:

S11: Handling of null values: If some fields have null values captured during the conversion process, the null values need to be replaced with certain data or loaded directly without any conversion;

S12: Standardize the data format: standardize and unify the data format according to the data type of each field in the business data source, for example, uniformly convert the numeric type into a string type;

S13: Split or merge fields according to business requirements;

S14: Replace missing data: replace missing data according to business requirements;

S15: Filter data according to business rules;

S16: Conversion of data uniqueness according to the coding table: data conversion is performed according to the business specification formulated by the coding table to realize the consistency of the internal data of the data warehouse system.
According to the intelligent agriculture AIOT distributed big data storage platform according to claim 1, it is characterized in that the data loading strategy of the data loading module includes a loading method of a timestamp, a loading method of a full table comparison, and a data loading method by reading a log table. The way of loading, the way of loading after deleting the entire table; the way of loading the timestamp is to add a timestamp field to the table of the source system, compare the current time of the system with the value of the timestamp, and determine which business data needs to be extracted , which can realize incremental loading of data, which is a relatively common loading method; the loading method of full table comparison is to compare each piece of data with all records of the target table before data loading, according to whether the primary key value is Similarly, it is judged whether the data is updated or inserted; the method of loading by reading the log table is to continuously update the information of the log table when the source data table changes, and use the information of the log table as a basis for data loading; after the entire table is deleted The way to load again is to delete all the data in the target table before loading the data, and then load all the data.