CN115665197A - Automatic warehousing system and method based on data acquisition of Internet of things - Google Patents
Automatic warehousing system and method based on data acquisition of Internet of things Download PDFInfo
- Publication number
- CN115665197A CN115665197A CN202211280959.6A CN202211280959A CN115665197A CN 115665197 A CN115665197 A CN 115665197A CN 202211280959 A CN202211280959 A CN 202211280959A CN 115665197 A CN115665197 A CN 115665197A
- Authority
- CN
- China
- Prior art keywords
- equipment
- data
- internet
- things
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an automatic warehousing system and method based on Internet of things data acquisition, wherein the system comprises an Internet of things acquisition service unit and a data warehousing and business object relationship mapping unit, wherein the Internet of things acquisition service unit is used for acquiring equipment data of different types and different protocol types; the data warehousing and business object relation mapping unit is used for performing visual arrangement relation mapping on different equipment type data, storing the data in the memory database and then distributing the data to the physical storage library for storage. By utilizing the automatic warehousing method for data acquisition of the Internet of things, development cycles and difficulties of related applications and systems of the Internet of things can be greatly saved, along with continuous improvement of acquisition engines, only configuration is needed to complete acquisition and warehousing of equipment data when related projects of the Internet of things are developed, and the development cycles and the cost are saved to the greatest extent.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to an automatic warehousing system and method based on data acquisition of the Internet of things.
Background
In the big data era of everything interconnection, the mobile sensing equipment supports social activities and changes of life styles of people, and the data acquisition modes and data types aiming at the mobile equipment are various. The method is characterized in that different manufacturers and different interface protocols are involved, the most common internet of things protocols comprise MQTT, OPC, modbus, BACnet, webservice, webAPI and the like, in the traditional internet of things scene application development process, the interfaces of different manufacturers and different protocol types need to be developed in a targeted mode, and meanwhile, equipment operation data needs to be collected to a platform to be stored in a unified mode and used for subsequent real-time monitoring of the operation state of equipment, fault alarming and operation data statistical analysis.
As shown in fig. 1 of the specification, a conventional data acquisition mode of the internet of things device is shown in a graph 1, and for different internet of things devices and different interface protocols, a data acquisition interface needs to be separately developed and then is analyzed according to a parameter format and a rule for storage. Meanwhile, the data is stored in different types of databases aiming at different data types, such as: the basic information MySQL relational database, heartbeat data of equipment operation are stored in Redis or MongoDB.
The traditional data acquisition mode of the internet of things leads to a large amount of interface development and docking work along with the increase of the number of devices and the types of the devices. Not only is the development cost of the project high and the period long, but also the development of the industry is severely restricted. There is a need to solve such problems by technical innovation means, thereby reducing the development cost and the period of the project.
Disclosure of Invention
The invention aims to provide an automatic warehousing system and method based on Internet of things data acquisition, which enable equipment data of different interface protocols to be conveniently and quickly accessed in actual use, realize automatic acquisition and warehousing and further improve the efficiency of Internet of things equipment data acquisition.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
an automatic warehousing system based on Internet of things data acquisition comprises an Internet of things acquisition service unit and a data warehousing and business object relationship mapping unit, wherein the Internet of things acquisition service unit is used for acquiring equipment data of different types and different protocol types; the data storage and business object relational mapping unit is used for carrying out visual arrangement relational mapping on different equipment type data, storing the data in the memory database and then distributing the data to the physical storage library for storage.
The invention also discloses an automatic warehousing method based on the data acquisition of the Internet of things, which comprises the following concrete warehousing methods:
step 1, data acquisition of Internet of things
According to the butt joint modes of different manufacturers and different equipment types, the acquisition engine automatically calls a communication component gateway in the engine to realize the north direction acquisition of equipment state and the south direction issue of the equipment data;
step 2, configuring data acquisition engine of Internet of things
Defining various types of internet-of-things equipment to be docked as products, defining basic attribute parameters, data indexes of the equipment and control indexes of the equipment contained in the equipment in the products, defining all newly added physical equipment as examples, inheriting the attributes and index contents of the products by the equipment examples, and individually expanding the attributes and indexes of the specific equipment examples according to project conditions;
and step 3: scheduling and monitoring of data acquisition engine of Internet of things
The device is used for controlling the frequency of device data acquisition in the process of interacting with the device and the log result of each data acquisition, so that whether the data acquisition process of the monitoring device normally operates or not can be conveniently mastered, and abnormal conditions can be captured and processed in time;
and 4, step 4: automatic data storage realized by equipment instantiation and business rule configuration
According to the equipment distribution point bit quantity of an actual service scene, instantiation initial configuration of equipment types is realized through a point bit table, and based on a service meaning explanation relation of switching value/analog quantity, service mapping rule configuration is completed, equipment original data are automatically converted and put in storage according to a service mapping rule, and equipment data service analysis is realized;
step 5, processing the data service of the Internet of things
The processing of the data service of the Internet of things is mainly used for completing the functions of SQL interface development and analysis, processing rule configuration, data preview, blood relationship analysis, automatic scheduling and log monitoring;
the configuration information comprises a target table, a target field, a source table, a source field, a processing mapping relation or SQL statement configuration automatic analysis, service processing scheduling frequency, audit rules and the like, and the full-process autonomous development function support from development, data preview, automatic scheduling, running log monitoring and audit log monitoring is completed;
step 6, configuring the external sharing service of the data unification
The method is mainly used for realizing application service sharing of the data of the Internet of things, monitoring visual display of internal support Internet of things equipment, data query of external support external systems and the like. The core capability automatically generates an API interface for the database table for front-end application and external service invocation.
In step 1, if the collected device is a standard mainstream internet of things protocol, the plug-in is directly loaded into the collection engine, and if the collected device is other non-standard protocols, the customized plug-in can be developed and loaded into the collection engine.
Further, the plug-ins are divided into three types, specifically, an internet of things communication protocol, a video communication protocol and a system Web API.
In step 2, the product includes a fresh air fan, a water pump or a temperature and humidity sensor, the basic attribute parameters include a product name, a type, a manufacturer or a docking protocol, the data indexes include a switching value or an analog value of the equipment, an equipment state or equipment running time, and the control indexes include indexes of control equipment, such as a switching, a temperature adjustment or a control mode.
Further optimization, in step 3, the scheduling and monitoring of the data acquisition engine of the internet of things specifically comprises the scheduling of the acquisition engine and the monitoring of the acquisition engine,
wherein the content of the first and second substances,
the acquisition engine scheduling is used for independently configuring acquisition frequency according to actual project requirements aiming at data indexes of each product, and an interface is taken to acquire data and write the data into a Redis database so as to facilitate the next step of data storage or the real-time display of a client for use;
the collection engine monitoring is used for data collection processes with different frequencies, and calling monitoring capability is provided in the engine, so that the condition of equipment collection at each time can be conveniently traced.
Further limiting, the step 4 specifically includes the following steps:
s401: device instance initialization
Combing equipment installation area information, equipment network allocation addresses, operation and maintenance responsible persons, installation responsible persons or installation time information according to construction/completion designs of different brands and models of equipment combined with actual service scenes, and then finishing equipment example basic information configuration work in a batch import or interface custom entry mode by combining basic attributes defined by equipment products in acquisition configuration;
s402: business rule mapping
The service mapping rule configuration comprises equipment point table interpretation information initialization configuration, point table binding configuration of an equipment instance, automatic table building configuration and field service mapping relation configuration;
initializing and configuring the device point table interpretation information: the initialization configuration of the equipment point table interpretation information mainly aims at the Chinese interpretation of the service meaning of the equipment point table entry,
point table binding configuration of device instance: carrying out initialization import, supporting a file upload import or interface input mode, and completing the relation binding of the point table items and the equipment examples by association matching by combining the initial information of the equipment examples, namely the relation between the equipment examples and the products, through the relation between the products and the point table items and by taking the products as external key conditions;
automatic table building configuration: the table building definition configuration is mainly based on the names of parameter items in the equipment point table;
and (3) field service mapping relation configuration: the field service mapping relation is mainly configured through the service interpretation meanings of the analog quantity and the switching quantity in the initialization of the equipment point table, the analog quantity is used as the value of the database service field by using a parameter value splicing parameter unit, and the switching quantity is mainly converted through the service mapping relation of the switching quantity value;
s403: automatic warehouse entry
The database storage mapping configuration mainly realizes the automatic data storage function after the business mapping conversion of the data collected by the equipment, the tool is integrated and supports the use of a mainstream database version (Oracle/Mysql/Postgre/Sqlserver), and the interface automatic storage configuration is realized.
In step S403, the configuration information includes configuration of login information of the storage database, automatic table establishment configuration of the database, automatic loading of mapping relationship between the target field and the acquired original field, and automatic loading of service meaning conversion relationship.
Compared with the prior art, the invention has the following beneficial effects:
in actual use, the method utilizes the internet of things collection service (DWD-IOT) to collect equipment data of different types and different protocol types, then utilizes data warehousing and business object relational mapping (DWD-ORM) to carry out visual arrangement relational mapping on the data of different equipment types, and the system warehouses according to data formats, data frequencies and the like, and finally realizes automatic warehousing of the internet of things equipment data collection;
by utilizing the automatic warehousing method for data acquisition of the Internet of things, the development period and difficulty of related applications and systems of the Internet of things can be greatly saved, along with the continuous improvement of an acquisition engine, the acquisition and warehousing of equipment data can be completed only by configuration when related projects of the Internet of things are developed in the future, and the development period and cost are saved to the greatest extent.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a flow chart of a conventional data acquisition method of the internet of things in the prior art.
Fig. 2 is a flow chart of a data acquisition mode of the internet of things.
Fig. 3 is an overall technical framework diagram of the present invention.
Fig. 4 is a diagram of a data collection engine of the internet of things according to the present invention.
FIG. 5 is a flow chart of the Web API interface configuration tool of the present invention.
FIG. 6 shows the E.R relationship between the product of the present invention and an example of the apparatus.
FIG. 7 is a flow chart of an implementation of an automatic warehousing for device instantiation and business rule configuration in accordance with the present invention.
Fig. 8 is a processing flow chart of the data service of the internet of things of the invention.
Fig. 9 is a block diagram of the overall process of the present invention.
Detailed Description
In the following, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the embodiments of the invention. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive, and embodiments of the invention are described in detail below with reference to the accompanying drawings.
Example one
The embodiment discloses an automatic data storage system based on Internet of things data acquisition, which comprises an Internet of things acquisition service unit and a data storage and business object relation mapping unit, wherein the Internet of things acquisition service unit is used for acquiring equipment data of different types and different protocol types; the data storage and business object relational mapping unit is used for carrying out visual arrangement relational mapping on different equipment type data, storing the data in the memory database and then distributing the data to the physical storage library for storage.
The method for automatically warehousing data acquisition based on the internet of things comprises the step of using the system for automatically warehousing data acquisition based on the internet of things, which is disclosed in claim 1, and the specific warehousing method comprises the following steps:
step 1, data acquisition of Internet of things
According to the butt joint modes of different manufacturers and different equipment types, the acquisition engine automatically calls a communication component gateway in the engine to realize the north direction acquisition of equipment state and the south direction issue of the equipment data;
step 2, configuring data acquisition engine of Internet of things
Defining various types of internet-of-things equipment to be butted as products, defining basic attribute parameters, equipment data indexes and equipment control indexes contained in the equipment in the products, defining all newly-added physical equipment as examples, inheriting the attributes and index contents of the products by the equipment examples, and individually expanding the attributes and indexes of the specific equipment examples according to the project conditions;
and 3, step 3: scheduling and monitoring of data acquisition engine of Internet of things
The device is used for controlling the frequency of device data acquisition in the process of interacting with the device and the log result of each data acquisition, so that whether the data acquisition process of the monitoring device normally operates or not can be conveniently mastered, and abnormal conditions can be captured and processed in time;
and 4, step 4: automatic data storage realized by equipment instantiation and business rule configuration
According to the equipment distribution point bit quantity of an actual service scene, instantiation initial configuration of equipment types is realized through a point bit table, and based on a service meaning explanation relation of switching value/analog quantity, service mapping rule configuration is completed, equipment original data are automatically converted and put in storage according to a service mapping rule, and equipment data service analysis is realized;
step 5, processing the data service of the Internet of things
The processing of the data service of the Internet of things is mainly used for completing the functions of SQL interface development and analysis, processing rule configuration, data preview, blood relationship analysis, automatic scheduling and log monitoring;
the configuration information comprises a target table, a target field, a source table, a source field, a processing mapping relation or SQL statement configuration automatic analysis, service processing scheduling frequency, audit rules and the like, and the full-process autonomous development function support from development, data preview, automatic scheduling, running log monitoring and audit log monitoring is completed;
step 6, configuring the external sharing service of the data unification
The method is mainly used for sharing application services of the data of the Internet of things, monitoring and visually displaying internal supporting Internet of things equipment, inquiring data of an external supporting external system and the like. The core capability automatically generates an API interface for the database table for front-end application and external service invocation.
In step 1, if the collected device is a standard mainstream internet of things protocol, the plug-in is directly loaded into the collection engine, and if the collected device is other non-standard protocols, the customized plug-in can be developed and loaded into the collection engine.
In this embodiment, the plug-in is divided into three types, specifically, an internet of things communication protocol, a video communication protocol, and a system Web API.
Further optimization, in step 2, the product includes a fresh air fan, a water pump or a temperature and humidity sensor, the basic attribute parameters include a product name, a type, a manufacturer or a docking protocol, the data indexes include switching values or analog values of equipment, the equipment state or equipment running time, and the control indexes include indexes of control equipment, such as a switching, temperature regulation or control mode.
Further limiting, in step 3, the scheduling and monitoring of the internet of things data acquisition engine specifically comprises the scheduling of the acquisition engine and the monitoring of the acquisition engine,
wherein the content of the first and second substances,
the acquisition engine scheduling is used for independently configuring acquisition frequency according to actual project requirements aiming at data indexes of each product, and an interface is taken to acquire data and write the data into a Redis database so as to facilitate the next step of data storage or the real-time display of a client for use;
the collection engine monitoring is used for data collection processes with different frequencies, and calling monitoring capability is provided in the engine, so that the condition of equipment collection at each time can be conveniently traced.
In this embodiment, step 4 specifically includes the following steps:
s401: device instance initialization
According to the implementation/completion design of different brands and models of equipment combined with the actual service scene, carding equipment installation area information, equipment network allocation addresses, operation and maintenance responsible persons, installation responsible persons or installation time information, defining basic attributes by combining equipment products in acquisition configuration, and completing the configuration work of equipment example basic information in a batch import or interface custom entry mode;
s402: business rule mapping
The service mapping rule configuration comprises equipment point table interpretation information initialization configuration, point table binding configuration of an equipment instance, automatic table building configuration and field service mapping relation configuration;
initializing and configuring the equipment point table interpretation information: the initialization configuration of the equipment point table interpretation information is mainly aiming at the Chinese interpretation of the service meaning of the equipment point table entry,
point table binding configuration of device instance: carrying out initialization import, supporting a file upload import or interface input mode, and completing the relation binding of the point table items and the equipment examples by association matching by combining the initial information of the equipment examples, namely the relation between the equipment examples and the products, through the relation between the products and the point table items and by taking the products as external key conditions;
automatic table building configuration: the table building definition configuration is mainly based on the names of parameter items in the equipment point table;
and (3) field service mapping relation configuration: the field service mapping relation is mainly configured through the service interpretation meanings of the analog quantity and the switching quantity in the initialization of the equipment point table, the analog quantity is used as the value of the database service field by using a parameter value splicing parameter unit, and the switching quantity is mainly converted through the service mapping relation of the switching quantity value;
s403: automatic warehouse entry
The database storage mapping configuration mainly realizes the automatic data storage function after the business mapping conversion of the data collected by the equipment, the tool is integrated and supports the use of a mainstream database version (Oracle/Mysql/Postgre/Sqlserver), and the interface automatic storage configuration is realized.
In step S403, the configuration information includes configuration of login information of the storage database, automatic table establishment configuration of the database, automatic loading of mapping relationship between the target field and the acquired original field, and automatic loading of service meaning conversion relationship.
In order to facilitate a further understanding of the invention for those skilled in the art, the invention is further described below in conjunction with specific examples.
Referring to fig. 1-9, the present embodiment mainly relates to two core content boards, specifically: and the Internet of things collection service (DWD-IOT), data storage and business object relational mapping (DWD-ORM).
The data acquisition mode of the internet of things is shown in fig. 2, and the main working idea is to use the internet of things acquisition service (DWD-IOT) to acquire device data of different types and different protocol types, then use data warehousing and business object relational mapping (DWD-ORM) to perform visual arrangement relational mapping on the data of different device types, and the system warehouses according to data format, data frequency and the like. And finally, automatic data acquisition and storage of the Internet of things equipment are realized.
The technical framework of the invention is shown in FIG. 3;
the internet of things collection service (DWD-IOT) is specifically as follows:
step 1: internet of things data acquisition (Internet of things data acquisition engine)
The communication gateway plug-in for managing various Internet of things protocols, video communication and system Web APIs is used as a bridge for communication interaction of software and physical equipment constructed by the invention. In actual use, according to the butt joint modes of different equipment types of different manufacturers, the communication component gateway in the engine is automatically called, and the north direction acquisition equipment state and the south direction issuing instruction of the equipment data are realized.
At present, the device supports mainstream Internet of things protocols, and if the collected equipment is other nonstandard protocols, customized plug-ins can be developed and loaded into the collection engine. The plug-in is divided into three types, namely an internet of things communication protocol, a video communication protocol and a system Web API, and the structure diagram is shown in FIG. 4.
Component calling and distributing: according to the communication protocol components configured for equipment management in the system, the system automatically calls the corresponding components; and opening the gateway through the communication rule to perform data and control interaction with the corresponding physical equipment.
The invention realizes hot plug type management of various communication protocols based on a java framework of the Spring Boot, creates an independent communication plug-in, and automatically calls java open source library components (such as Modbus4J, milo, BACnet4J and the like) in the communication technology of the Spring Boot according to the equipment communication mode definition of an application layer.
Thing networking communication protocol subassembly: according to the invention, main stream communication protocols of Modbus, OPC, BACnet and other equipment are integrated, and each type of equipment configures corresponding switching value and analog value conversion (please refer to the configuration flow of an Internet of things acquisition engine in the configuration process) on the basis of the standard protocol, so that interaction with corresponding physical equipment is realized;
1) In the butt joint of the Modbus communication protocol, the communication process of Modbus equipment is realized by adopting a Java-based Modbus4J open source library. The Modbus protocol core is used for reading and writing address variables and converting data types, and the Modbus4J open source library realizes the encapsulation of the address variable reading and writing method based on the Modbus protocol.
2) The invention relates to the interface of OPC communication protocol, in particular to OPC protocol referred to as new OPC standard-OPC UA (defined on a transport layer and supporting the new OPC standard of cross-platform). The invention adopts a Millo open source library based on JAVA (an item of the Internet of things under Eclipse flag is a high-performance OPC UA stack). In the Milo open source library, a group of client side and server side APIs are provided to support access to real-time data, monitoring, alarming, subscribing data, supporting events, accessing historical data and modeling data.
3) The invention relates to the docking of BACnet communication protocols, and a java open source library BACnet4j is used for realizing the calling of attributes (such as analog output, digital input and the like) and methods (such as reading attributes and the like) of BACnet equipment.
Video communication protocol components: including the mainstream video stream interfacing protocols to RTSP, ONVIF, HLS, GB28181, etc. The main stream video docking protocol is docked with various manufacturer video monitoring equipment, so that the functions of acquiring live video streams, distributing client sides, locally recording and broadcasting videos and controlling a holder are realized. In the invention, in order to realize the operations of stream pulling, format conversion and stream pushing of main stream Video stream docking protocols such as RTSP, ONVIF, HLS, GB28181 and the like, an AKStream open source soft NVR (soft NVR refers to NVR (Network Video decoder)) interface platform defined by software is used as a core Video communication component.
The AKStream fully supports H265/H264/AAC/G711/OPUS and other audio and video coding formats, supports Rtp (real time protocol) push stream of GB28181, GB28181-PTZ (real time protocol) control, HTTP (hyper text transport protocol) of a built-in streaming proxy, rtps (real time protocol) pull stream, RTMP pull stream (H264, H265/ACC/G711 support) and almost all forms of pull stream of ffmpeg streaming proxy (almost all formats and transcoding support), and converts the push and pull stream into almost all protocol interconversion of RTSP/RTMP/HLS/HTTP-FLV/WebSocket-FLV/GB 28181/HTTP-TS/WebSocket-TS/fMP-fMP 4/WebSocket-fMP4/MP4 and the like for third parties (APP, WEB, client and the like) to call and play.
The invention provides an interface configuration tool aiming at the butt joint mode of the Web API, and realizes the automatic analysis capability of the online calling and returning parameters of the interface, and a flow chart of the Web API interface configuration tool is shown in figure 5, and the concrete steps are as follows:
1) And adding a third-party Web API interface in the system, and adding an interface name and an interface URL address.
2) For the interface, the request parameters are added by referring to the third-party system Web API interface description. Each request parameter optional definition is deposited in the Header or Content (optional submission using form or application/json format) of the interface. The request parameters need to be configured to include data type, key, value and parameter description.
3) According to the configured interface address and the request parameter, the test interface initiates an interface request to the third-party system, and a corresponding interface result is returned when the request is successful. If the request fails, the interface address or request parameters are modified according to the interface return result.
4) The interface return parameters which are requested to be successfully returned are usually returned in a Json format, json of the returned result is subjected to format analysis (analyzed into a key-value form) by using a JsonPath tool, and target return parameters are selected to be stored (the data type, the key and the parameter description of the return parameters are stored).
5) And storing the interface, wherein the complete Web API interface stores the request and the return result in the interface configuration tool.
The method subjects an interface to remove customization development work, namely, defines an interface as an interface object, and the interface address, the request parameter and the return parameter contained in the interface are all attributes of the interface object. In the software application, the return parameters of the interface can be selected to be bound with the corresponding state property of the equipment, and the corresponding data can be automatically acquired.
Step 2: internet of things data acquisition engine configuration
Based on an object-oriented design idea, various types of internet-of-things equipment to be docked are defined as products (such as a fresh air fan, a water pump, a temperature and humidity sensor and the like), basic attribute parameters (such as product names, types, manufacturers, docking protocols and the like) contained in the equipment, data indexes (such as switching values or analog values of the equipment, equipment states, equipment running time and the like) of the equipment and control indexes (such as indexes of control equipment, such as switching, temperature regulation, control modes and the like) of the equipment are defined in the products. All newly added physical equipment is defined as an example, the equipment example inherits the attribute and index content of the product, and the attribute and index of the specific equipment example can be expanded in a personalized mode according to the project situation. In the following description, the temperature and humidity sensor is taken as an example, and the E · R relationship between the product and the device example is shown in fig. 6:
1. a device such as a temperature and humidity sensor is defined as a product comprising the following basic properties:
the docking mode is selected as a Web API, the reading control of the known temperature and humidity sensor equipment needs to be acquired from an environment monitoring system, and the corresponding interface of the system is configured and completed in advance in a Web API configuration tool. Other self-defined parameters are configured to be IP, and when the temperature and humidity sensor equipment is newly added, the IP address of each equipment needs to be recorded.
2. The data indexes of the temperature and humidity sensor are defined as follows:
3. the control indexes of the temperature and humidity sensor are defined as follows:
3. when the temperature and humidity sensor equipment is newly added, the basic attribute, the data index and the control index are inherited, and the corresponding interface can be correspondingly called to finish data reading and control interaction after the IP address of the equipment is initialized;
and step 3: scheduling and monitoring of data acquisition engine of Internet of things
The purpose of the functional design is to control the frequency of device data acquisition in the process of device interaction and log results of data acquisition at each time, so that whether the monitoring device data acquisition process runs normally or not can be conveniently mastered, and abnormal conditions can be captured and processed in time.
1. And (3) scheduling an acquisition engine:
the system can be directed at the data index of every product and dispose the collection frequency alone according to actual project demand, if to the environment in data computer lab center, the humiture directly influences whether normal operating of server in the data computer lab, need in time pay close attention to the computer lab humiture condition, and here is once every 5 seconds to the data index of temperature and humidity sensor product, temperature value and humidity value setting collection frequency. After setting, the acquisition engine automatically calls temperature and humidity acquisition interfaces to all temperature and humidity sensor equipment to acquire data and writes the data into a Redis database so as to facilitate the next step of data storage or real-time display of a client for use.
2. Monitoring by an acquisition engine:
aiming at data acquisition processes with different frequencies, the calling monitoring capability is provided in the engine, so that the condition of equipment acquisition at each time can be traced conveniently. The monitoring content comprises a product name, an equipment name, a data index name, calling time, a calling result (success and failure), input parameter content and return parameter content. And the above contents can also monitor whether the related configuration contents are correct in the equipment docking process, so as to influence the calling result.
The data warehousing and business object relational mapping (DWD-ORM) is specifically as follows:
and 4, step 4: automatic data storage realized by equipment instantiation and business rule configuration
The method comprises the steps of realizing instantiation initial configuration of equipment types through a point location table according to the equipment distribution point location quantity of an actual service scene, completing service mapping rule configuration based on the service meaning interpretation relation of switching value/analog quantity, automatically converting and warehousing equipment original data according to a service mapping rule, and realizing equipment data service analysis.
The functional core three elements comprise equipment point table interpretation information (used for describing specific Chinese meanings of equipment operation parameters, such as temp representing temperature, level representing gear, alarminfo representing early warning information and the like), an equipment point table (used for describing the deployment condition of equipment on site, including deployment distribution position, deployment network address and the like), database storage mapping (used for storing business mapping rules, equipment acquisition table storage elements and the like), and business relation mapping and database storage of returned data of equipment of different brands and models are realized. The overall process of implementing automatic warehousing by device instantiation and business rule configuration is shown in fig. 7:
1. device instance initialization:
the equipment point location table is mainly used for initializing equipment instances, combing equipment installation area information, equipment network allocation addresses, operation and maintenance responsible persons, installation time and other related information according to construction/completion design of different brands and models of equipment combined with actual service scenes, and then combining with equipment product definition basic attributes (such as equipment codes, equipment names, equipment types, equipment brands and equipment models) in collection configuration to complete equipment instance basic information configuration work in a batch import or interface custom entry mode.
The step realizes basic information management of field device deployment, forms a device instance information ledger, contains device codes, device names, device type codes, device type names, device brands, device deployment network addresses, device affiliated areas, device installation time, device manufacturers, operation and maintenance responsible persons and the like, and provides basic data support for warehousing the devices according to types and subsequent operation and maintenance inspection of the devices.
2. And (3) business rule mapping:
the service mapping rule configuration comprises equipment point table interpretation information initialization configuration, point table binding configuration of equipment instances, automatic table building configuration and field service mapping relationship configuration. The initialization configuration of the equipment point table interpretation information mainly aims at the Chinese interpretation of the service meaning of the equipment point table item (such as temp for temperature, level for gear, alarminfo for alarm information, and the like), performs initialization import, and supports a file uploading import or interface entry mode.
The binding configuration of the point table of the equipment instance is mainly characterized in that the binding of the point table and the equipment instance is completed through the correlation between a product (namely equipment type) and the point table, such as a temperature and humidity monitor, the point table is in a temperature, humidity and switch state, and the binding configuration combines initial information (the correlation between the equipment instance and the product) of the equipment instance, such as the temperature and humidity monitor installed in a field, wherein the area A is provided with a monitor No. 001, the area B is provided with a monitor No. 002 and 2 equipment instances, and the product is used as an external key condition.
The table building definition configuration is mainly based on parameter item names in the equipment point table, and in order to facilitate subsequent data anomaly problem positioning, logical relationship traceability and the like, the parameter item names are directly adopted as field names of the table base, and English character strings of products are coded into the table names.
The field service mapping relation is mainly configured through the service interpretation meaning of the analog quantity and the switching quantity in the initialization of the equipment point table. For the analog quantity, parameter value splicing parameter units are used as the field values of the database service (if the parameter values are 26, the parameter units are C, the field values are 26 ℃). The switching value is mainly converted through the service mapping relation (for example, 0/1 represents on/off, normal/abnormal, online/offline, etc.) of the switching value, and the specific type and the mapping rule are shown in the following table.
3. Automatic warehousing:
the database storage mapping configuration mainly realizes the automatic data storage function after the business mapping conversion of the data collected by the equipment, the tool is integrated and supports the version of the mainstream database to use Oracle/Mysql/Postgre/Sqlserver and the like), and the interface automatic storage configuration is realized. The configuration information comprises configuration (database link, user name and password) for storing database login information, configuration (table name, field type, index and field explanation) for automatically building a table for the database, mapping relation between a target field and an acquired original field, and automatic loading of business meaning conversion relation (conversion rule is configured in step one and loaded according to equipment type), and specific configuration items and configuration contents are as follows.
And 5, processing the data service of the Internet of things:
the step mainly ensures the processing function of an application layer for collecting and storing the equipment data, and realizes the application support of equipment monitoring data display, equipment data service statistical analysis, equipment operation and maintenance scene linkage decision management, equipment simulation training, data external sharing and the like.
The processing of the data service of the internet of things mainly completes the functions of SQL interface development and analysis, processing rule configuration, data preview, blood relationship analysis, automatic scheduling, log monitoring and the like. The configuration information comprises a target table, a target field, a source table, a source field, a processing mapping relation or SQL statement configuration automatic analysis, service processing scheduling frequency, audit rules and the like, the full-process independent development function support from development, data preview, automatic scheduling, operation log monitoring and audit log monitoring is completed, and the processing flow chart of the data service of the Internet of things is shown in FIG. 8.
Step 6, configuring external sharing services by data unification:
the module mainly realizes application service sharing of the data of the Internet of things, internal support Internet of things equipment monitoring visual display (including equipment operation monitoring, equipment operation statistics and the like), external support external system data query and the like. The core capability automatically generates an API interface for the database table for front-end application and external service invocation, and is specifically shown in the following table.
In this embodiment:
spring boot: the background configuration page is mainly used for parameter configuration of the data acquisition tool, arrangement of data storage relations and the like;
hibernate: the data persistence framework is mainly used for storing the acquired equipment data in a local database;
quartz: the data acquisition job scheduling framework is mainly used for controlling the frequency scheduling of equipment data acquisition;
jsonpath: the Json data format analysis framework is mainly used for analyzing the internal and external data service formats;
vue3, vuex, element-plus: background configuration pages and development of visual pages;
c + +, C #, python: the method is mainly used for developing the acquisition engine, and during the technical model selection process of engine development, the method is matched with the interface protocol and mode provided by equipment as much as possible, so that the technical compatibility and the efficiency of engine development are maintained to the greatest extent.
In practical use, the method relies on the internet of things and big data correlation technology to realize automatic collection and storage of the data of the internet of things equipment, and the data can be provided for the inside and the outside of the system for use after being stored.
The internal application scenario is specifically as follows:
based on the middlebox (service middlebox + data middlebox), a large number of service scenes can be enriched, such as: monitoring the operation of the facility equipment, and intelligently linking and controlling the facility equipment. Providing cross-industry and cross-field data fusion application for transportation, medical treatment, park, travel, building and the like
The external application scene is specifically as follows: the data is directly opened by using the data service to be used by external application, so that the cross-border integration of a subsequent system is facilitated.
In some embodiments of the present invention, the,
while preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, it should be noted that any modifications, equivalents and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (8)
1. The utility model provides a system based on thing networking data acquisition is automatic to be gone into storehouse which characterized in that: the system comprises an Internet of things acquisition service unit and a data warehousing and business object relation mapping unit, wherein the Internet of things acquisition service unit is used for acquiring equipment data of different types and different protocol types; the data warehousing and business object relation mapping unit is used for performing visual arrangement relation mapping on different equipment type data, storing the data in the memory database and then distributing the data to the physical storage library for storage.
2. A method for automatically warehousing data acquisition based on the Internet of things is characterized by comprising the following steps: the system for automatically warehousing data acquisition based on the internet of things comprises the following steps:
step 1, data acquisition of Internet of things
According to the butt joint modes of different manufacturers and different equipment types, the acquisition engine automatically calls a communication component gateway in the engine to realize the north direction acquisition of equipment state and the south direction issue of the equipment data;
step 2, configuring data acquisition engine of Internet of things
Defining various types of internet-of-things equipment to be docked as products, defining basic attribute parameters, data indexes of the equipment and control indexes of the equipment contained in the equipment in the products, defining all newly added physical equipment as examples, inheriting the attributes and index contents of the products by the equipment examples, and individually expanding the attributes and indexes of the specific equipment examples according to project conditions;
and step 3: scheduling and monitoring of data acquisition engine of Internet of things
The device is used for controlling the frequency of device data acquisition in the process of interacting with the device and the log result of each data acquisition, so that whether the data acquisition process of the monitoring device normally operates or not can be conveniently mastered, and abnormal conditions can be captured and processed in time;
and 4, step 4: automatic data storage realized by equipment instantiation and business rule configuration
According to the equipment distribution point bit quantity of an actual service scene, instantiation initial configuration of equipment types is realized through a point bit table, and the service meaning explanation relationship based on switching value/analog quantity is used for completing service mapping rule configuration, and equipment original data is automatically converted and put in storage according to the service mapping rule, so that equipment data service analysis is realized;
step 5, processing the data service of the Internet of things
The processing of the data service of the Internet of things is mainly used for completing the functions of SQL interface development and analysis, processing rule configuration, data preview, blood relationship analysis, automatic scheduling and log monitoring;
the configuration information comprises a target table, a target field, a source table, a source field, a processing mapping relation or SQL statement configuration automatic analysis, service processing scheduling frequency and audit rules, and the full-process autonomous development function support from development, data preview, automatic scheduling, running log monitoring and audit log monitoring is completed;
step 6, configuring the external sharing service of the data unification
The method is mainly used for sharing application services of the data of the Internet of things, monitoring and visually displaying internal support Internet of things equipment and inquiring data of an external support external system, and the core capability automatically generates an API (application programming interface) for a database table for calling front-end application and external services.
3. The method for automatic warehousing based on internet of things data acquisition as claimed in claim 2, wherein: in the step 1, if the collected equipment is the adopted standard mainstream internet of things protocol, the plug-in is directly loaded into the collection engine, and if the collected equipment is other non-standard protocols, the customized plug-in can be developed and loaded into the collection engine.
4. The method for automatic warehousing based on internet of things data acquisition as claimed in claim 3, wherein: the plug-in is divided into three types, specifically, an internet of things communication protocol, a video communication protocol and a system Web API.
5. The method for automatic warehousing based on internet of things data acquisition as claimed in claim 2, wherein: in step 2, the product comprises a fresh air fan, a water pump or a temperature and humidity sensor, the basic attribute parameters comprise a product name, a type, a manufacturer or a docking protocol, the data indexes comprise switching values or analog values of equipment, the equipment state or equipment running time, and the control indexes comprise indexes of control equipment, such as a switching, temperature regulation or control mode.
6. The method for automatic warehousing based on internet of things data acquisition as claimed in claim 2, wherein: in step 3, the scheduling and monitoring of the data acquisition engine of the internet of things specifically comprises the scheduling of the acquisition engine and the monitoring of the acquisition engine,
wherein the content of the first and second substances,
the acquisition engine scheduling is used for independently configuring acquisition frequency according to actual project requirements aiming at data indexes of each product, and an interface is taken to acquire data and write the data into a Redis database so as to facilitate the next step of data storage or the real-time display of a client for use;
the collection engine monitoring is used for data collection processes with different frequencies, and calling monitoring capability is provided in the engine, so that the condition of equipment collection at each time can be conveniently traced.
7. The method for automatically warehousing data acquisition based on the internet of things according to claim 2, wherein the step 4 specifically comprises the following steps:
s401: device instance initialization
Combing equipment installation area information, equipment network allocation addresses, operation and maintenance responsible persons, installation responsible persons or installation time information according to construction/completion designs of different brands and models of equipment combined with actual service scenes, and then finishing equipment example basic information configuration work in a batch import or interface custom entry mode by combining basic attributes defined by equipment products in acquisition configuration;
s402: business rule mapping
The service mapping rule configuration comprises equipment point table interpretation information initialization configuration, point table binding configuration of an equipment instance, automatic table building configuration and field service mapping relationship configuration;
initializing and configuring the equipment point table interpretation information: the initialization configuration of the equipment point table interpretation information is mainly aiming at the Chinese interpretation of the service meaning of the equipment point table entry,
point table binding configuration of device instance: carrying out initialization import, supporting a file upload import or interface input mode, and completing the relation binding of the point table items and the equipment examples by association matching by combining the initial information of the equipment examples, namely the relation between the equipment examples and the products, through the relation between the products and the point table items and by taking the products as external key conditions;
automatic table building configuration: the table building definition configuration is mainly based on the names of parameter items in the equipment point table;
and (3) field service mapping relation configuration: the field service mapping relation is mainly configured through the service interpretation meanings of the analog quantity and the switching quantity in the initialization of the equipment point table, the analog quantity is used as the value of the database service field of the analog quantity by using a parameter value splicing parameter unit, and the switching quantity is mainly converted through the service mapping relation of the switching quantity value;
s403: automatic warehouse entry
The database storage mapping configuration mainly realizes the automatic data storage function after the business mapping conversion of the data acquired by the equipment, and the mainstream database version is used (Oracle/Mysql/Postgre/Sqlserver), so that the interface automatic storage configuration is realized.
8. The method according to claim 7, wherein in step S403, the configuration information includes configuration of login information of a storage database, configuration of automatic table establishment of a database, mapping relationship between a target field and a collection original field, and automatic loading of service meaning conversion relationship.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211280959.6A CN115665197A (en) | 2022-10-19 | 2022-10-19 | Automatic warehousing system and method based on data acquisition of Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211280959.6A CN115665197A (en) | 2022-10-19 | 2022-10-19 | Automatic warehousing system and method based on data acquisition of Internet of things |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115665197A true CN115665197A (en) | 2023-01-31 |
Family
ID=84990184
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211280959.6A Pending CN115665197A (en) | 2022-10-19 | 2022-10-19 | Automatic warehousing system and method based on data acquisition of Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115665197A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116483909A (en) * | 2023-05-17 | 2023-07-25 | 杭州端点网络科技有限公司 | Big data integration system |
| CN116582582A (en) * | 2023-07-14 | 2023-08-11 | 中冶武勘智诚(武汉)工程技术有限公司 | Data acquisition method, system, equipment and medium for component type Internet of things |
| CN117708218A (en) * | 2024-02-05 | 2024-03-15 | 成都秦川物联网科技股份有限公司 | Industrial Internet of things data access method and database system based on service sub-platform |
-
2022
- 2022-10-19 CN CN202211280959.6A patent/CN115665197A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116483909A (en) * | 2023-05-17 | 2023-07-25 | 杭州端点网络科技有限公司 | Big data integration system |
| CN116582582A (en) * | 2023-07-14 | 2023-08-11 | 中冶武勘智诚(武汉)工程技术有限公司 | Data acquisition method, system, equipment and medium for component type Internet of things |
| CN116582582B (en) * | 2023-07-14 | 2023-10-13 | 中冶武勘智诚(武汉)工程技术有限公司 | Data acquisition method, system, equipment and medium for component type Internet of things |
| CN117708218A (en) * | 2024-02-05 | 2024-03-15 | 成都秦川物联网科技股份有限公司 | Industrial Internet of things data access method and database system based on service sub-platform |
| CN117708218B (en) * | 2024-02-05 | 2024-05-03 | 成都秦川物联网科技股份有限公司 | Industrial Internet of things data access method and database system based on service sub-platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115665197A (en) | Automatic warehousing system and method based on data acquisition of Internet of things | |
| US11120012B2 (en) | Web services platform with integration and interface of smart entities with enterprise applications | |
| US11522857B1 (en) | Video gateway for camera discovery and authentication | |
| US10410673B2 (en) | Embeddable video capturing, processing and conversion application | |
| KR102481906B1 (en) | Apparatus and method for providing a virtual device | |
| US6574655B1 (en) | Associative management of multimedia assets and associated resources using multi-domain agent-based communication between heterogeneous peers | |
| US8185427B2 (en) | Method and system for presenting user tasks for the control of electronic devices | |
| US10923157B2 (en) | System and method for event data collection and video alignment | |
| US8099313B2 (en) | Method and system for the orchestration of tasks on consumer electronics | |
| US11736412B1 (en) | Systems and methods for multi-tier resource and subsystem orchestration and adaptation | |
| Serrano et al. | Defining the stack for service delivery models and interoperability in the Internet of Things: A practical case with OpenIoT-VDK | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| CN102484740A (en) | Systems and methods for managing video data | |
| US11683579B1 (en) | Multistream camera architecture | |
| JP2008507014A (en) | Application partitioning for network edge computing | |
| US20060069602A1 (en) | Method and system for describing consumer electronics using separate task and device descriptions | |
| US20150317133A1 (en) | Cobol reference architecture | |
| US10951540B1 (en) | Capture and execution of provider network tasks | |
| FR2814569A1 (en) | METHOD AND SYSTEM FOR ANALYZING AND PROCESSING DATA ON THE INTERNET IN REAL TIME | |
| US20190362148A1 (en) | Method and system for enabling information in augmented reality applications | |
| US9372884B2 (en) | Extensible data query scenario definition and consumption | |
| CN115203172A (en) | Model construction method, model data subscription method, model construction device, model data subscription device, electronic equipment and medium | |
| US11152031B1 (en) | System and method to compress a time frame of one or more videos | |
| CN111108477A (en) | Universal virtual professional toolkit | |
| US10795925B2 (en) | Method, device and arrangement for measurement of app usage based on visual characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |