CN116126915A

CN116126915A - Intelligent gateway data storage deployment method, device, equipment and medium

Info

Publication number: CN116126915A
Application number: CN202310076330.8A
Authority: CN
Inventors: 唐仕斌; 彭府; 吴一鸣; 陈淑武; 吴世川
Original assignee: XIAMEN FOUR-FAITH COMMUNICATION TECHNOLOGY CO LTD
Current assignee: XIAMEN FOUR-FAITH COMMUNICATION TECHNOLOGY CO LTD
Priority date: 2023-01-17
Filing date: 2023-01-17
Publication date: 2023-05-16

Abstract

The invention provides an intelligent gateway data storage deployment method, device, equipment and medium, comprising the following steps: when the gateway process is started, a main thread and three sub threads are started, a current day data table in a database DB is obtained, an acquisition protocol is configured, and the gateway is triggered to perform an acquisition task; analyzing the acquired data through an interface to generate analysis data, and inserting the analysis data into a data table of the current day; when judging that the callback trigger period time of the reporting thread is reached, searching a message group packet for reporting, wherein the message group packet consists of recent data records in a preset period in a database DB; when judging that the packet is effective and the communication with the docking platform is abnormal, storing the packet into a cache table in a database DB; and when judging that the message group package is effective and the communication with the docking platform is normal, sending the message group package to the docking platform. In addition, the existing data processing scheme has the problems that files need to be split, the distribution number of handles is easy to trigger, and the like.

Description

Intelligent gateway data storage deployment method, device, equipment and medium

Technical Field

The invention relates to the technical field of intelligent gateway data storage, in particular to an intelligent gateway data storage deployment method, device, equipment and medium.

Background

With the advent of the big data age, protocol platforms such as mqtt, http and map need to collect and analyze data of terminal devices in multiple geographic positions, and present the data to users after conversion to realize efficient monitoring, but in the whole process, some important data need to be ensured not to be lost, in a zone where a network is not smooth enough, functions such as breakpoint continuous transmission and historical data reporting are necessary, and in order to meet efficient processing of the big data, the existing data processing scheme stores the acquired bottom data into a disk file through regular quantification, and then uses the acquired data according to different conditions. However, the text storage mode of the scheme is single, the fault tolerance rate is not high, and the data insertion mode is not chained storage, so that the efficiency of inquiring the large-capacity text is low; in short, the data stored in the data processing scheme in the current market can only be processed by a tail append mode, and the problem that the handle distribution number is insufficient when the number of files is enlarged to a certain degree is easily triggered because the file capacity reaches a certain threshold value and the efficiency of inquiring the data is required to be split to avoid low.

In view of this, the present application is presented.

Disclosure of Invention

In view of the above, the present invention aims to provide a method, an apparatus, a device, and a medium for deploying data storage in an intelligent gateway, which can effectively solve the problems that in the prior art, data stored in a data processing scheme can only be added in a tail portion mode, and when the file capacity reaches a certain threshold value, the efficiency of querying the data needs to be reduced, and when the number of files is enlarged to a certain extent, the number of handles is easily triggered to be insufficient.

The invention discloses an intelligent gateway data storage deployment method, which comprises the following steps:

when the gateway process is started, a main thread and three sub threads are started, a current day data table in a database DB is obtained, an acquisition protocol is configured, and the gateway is triggered to perform an acquisition task;

analyzing the acquired data through an interface to generate analysis data, and inserting the analysis data into the data table of the current day;

searching a message group packet to report after judging that the callback trigger period time of the reporting thread is reached, wherein the message group packet consists of recent data records in a preset period in the database DB;

when judging that the message group package is effective and the communication with the docking platform is abnormal, storing the message group package into a cache table in the database DB;

and when judging that the message group package is effective and the communication with the docking platform is normal, sending the message group package to the docking platform.

Preferably, the three sub-threads are a configuration thread, an acquisition thread and a display thread respectively, wherein the main thread is used for performing a report operation of the docking platform, the configuration thread is used for performing an automation operation, and the acquisition thread and the display thread are used for performing a database read-write operation.

Preferably, the data table of the current day in the database DB is obtained, and an acquisition protocol is configured to trigger the gateway to perform an acquisition task, specifically:

creating a database DB when the database DB in the storage device is judged to be absent;

when judging that the database DB exists in the storage device, judging whether the data table exists on the same day;

if not, creating a data table of the same day, and setting a timer in the endless loop of the display thread, wherein when the timer is ended at regular time, a new data table of the same day is automatically created, and when the number of the data tables reaches a preset value, the data table of the same day with the earliest date is deleted;

and configuring an acquisition protocol, and triggering a gateway to perform an acquisition task, wherein a data ID of the register data is used as a unique identifier.

Preferably, the collected data is parsed through an interface to generate parsed data, and the parsed data is inserted into the data table of the same day, specifically:

mapping a corresponding PLC protocol by defining a first interface body;

analyzing the acquired data through a second interface to generate analysis data;

displaying a show_stat thread corresponding to the analysis data to a web state page;

and calling a third interface to inquire a corresponding day data table from the database DB, and inserting the analysis data into the day data table.

Preferably, the packet is composed of recent data records within a preset period in the database DB, specifically:

creating a data linked list through a fourth interface, and extracting all data IDs to be reported from linked list nodes of the data linked list;

and matching the data in the data table of the same day by using the data ID and the current absolute timestamp as key values, and feeding back the screened data into a complete record for grouping to generate a message grouping.

Preferably, after the packet is stored in the cache table in the database DB when the packet is judged to be valid and communication with the docking platform is abnormal, the method further includes:

and when the communication on the docking platform is judged to be re-communicated, calling the third interface to search all unreported data in the cache table during disconnection, and reporting the unreported data to the docking platform one by one.

Preferably, when the packet is judged to be valid and the communication with the docking platform is normal, the method further comprises the steps of:

when judging that the sending of the message group packet is problematic, storing the message with failed sending into a cache table in the database DB;

and when the communication of the docking platform is communicated again, calling the third interface to search all unreported data in the cache table during disconnection, and reporting the unreported data to the docking platform one by one.

The invention also discloses an intelligent gateway data storage deployment device, which comprises:

the data acquisition unit is used for starting a main thread and three sub threads when the gateway process is started, acquiring a data table of the same day in the database DB, configuring an acquisition protocol, and triggering the gateway to perform an acquisition task;

the data insertion unit is used for analyzing the acquired data through an interface to generate analysis data and inserting the analysis data into the data table of the same day;

the message generating unit is used for searching a message group packet to report after judging that the callback trigger period time of the reporting thread is reached, wherein the message group packet consists of recent data records in a preset period in the database DB;

the message caching unit is used for storing the message group package into a cache table in the database DB when judging that the message group package is effective and the communication with the docking platform is abnormal;

and the message sending unit is used for sending the message group packet to the docking platform when judging that the message group packet is effective and the communication with the docking platform is normal.

The invention also discloses an intelligent gateway data storage deployment device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the intelligent gateway data storage deployment method is realized when the processor executes the computer program.

The invention also discloses a readable storage medium, which stores a computer program, the computer program can be executed by a processor of a device where the storage medium is located, so as to implement the intelligent gateway data storage deployment method according to any one of the above.

In summary, the method, the device, the equipment and the medium for deploying the intelligent gateway data storage provided in the embodiment are mainly used for solving the problem of data in the communication process between different protocol platforms and terminal equipment, and are realized by acquiring various time node historical data from a local terminal at any time, combining memory linked list storage variables, storing a sqlite database in real time, periodically traversing data table to update key values, clearing early data when different equipment reaches the upper limit threshold of storage capacity, and the like; the original data are mainly acquired from a lower-level PLC device through different protocols, unique marks are printed on one by one, then a record is formed according to the absolute value of acquisition time and relative time stamps, the record is stored in sqlite, and each 24 hours is stored in a table, so that the old data can be deleted in the whole table after a period of time, such as one week, so that sufficient storage capacity is ensured. Flexible storage can also be achieved by pruning a single data through a combination of time + identification. Therefore, the problems that data stored in a data processing scheme in the prior art can only be added in a tail part mode, the efficiency of inquiring data is low because the file capacity reaches a certain threshold value and the handle distribution number is not enough because the number of the files is enlarged to a certain degree are solved.

Drawings

Fig. 1 is a schematic flow chart of an intelligent gateway data storage deployment method according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of a database processing flow according to an embodiment of the present invention.

Fig. 3 is a diagram of a gateway software framework provided by an embodiment of the present invention.

Fig. 4 is a schematic block diagram of an intelligent gateway data storage deployment device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention.

Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

The invention discloses a method, a device, equipment and a medium for deploying intelligent gateway data storage, which at least solve the defects in the prior art to a certain extent.

Referring to fig. 1 to 3, a first embodiment of the present invention provides an intelligent gateway data storage deployment method, which may be executed by a storage deployment device (hereinafter referred to as a deployment device), and in particular, executed by one or more processors in the deployment device, to implement the following steps:

it should be noted that the deployment device may be a terminal device, which may implement wired or wireless communication with different protocol platforms, where the terminal device may be a smart phone, a smart computer, or other smart devices.

S101, when a gateway process is started, starting a main thread and three sub threads, acquiring a data table of the same day in a database DB, configuring an acquisition protocol, and triggering the gateway to perform an acquisition task;

specifically, step S101 includes: creating a database DB when the database DB in the storage device is judged to be absent;

Specifically, in this embodiment, the three sub-threads are a configuration thread, an acquisition thread and a display thread, where the main thread is used to perform a report operation of the docking platform, the configuration thread is used to perform an automation operation, and the acquisition thread and the display thread are used to perform a read-write operation of the database.

The existing data processing scheme is to store the acquired bottom data into a disk file by periodically and quantitatively, and then to acquire the data for use according to different conditions. However, the text storage mode of the scheme is single, the fault tolerance rate is not high, and the data insertion mode is not chained storage, so that the efficiency of inquiring the large-capacity text is low; in short, the data stored in the data processing scheme in the current market can only be processed by a tail append mode, and the problem that the handle distribution number is insufficient when the number of files is enlarged to a certain degree is easily triggered because the file capacity reaches a certain threshold value and the efficiency of inquiring the data is required to be split to avoid low.

In this embodiment, when the gateway process is started, three sub-threads, one is a configuration thread, one is an acquisition thread, and one is a display thread, where the main thread is used for reporting to the docking platform, the automation operations such as creating and destroying the database are all executed in the configuration thread, and the database read-write operations are executed in other threads. Then, judging whether a database DB in a storage device TF card or flash exists or not through a create_history_data_sqlite interface, and creating the database DB under the condition that the database DB does not exist; calling a create_history_data_table interface to judge whether a data table of the same day exists or not under the condition that a database DB exists, creating the data table of the same day under the condition that the data table of the same day does not exist, setting a timer of 0 early morning in the dead cycle of a display thread, and creating a new date table by self once the day is crossed; when a preset deadline of the web configuration, such as an upper limit of 7 days, is reached, the form on the earliest day is automatically cleared, and the destroy database interface can be invoked according to the web trigger button. And then configuring an acquisition protocol on a web page of the gateway, and triggering the gateway to perform an acquisition task by taking a data ID of the register data as a unique identifier and other flash parameters.

S102, analyzing the acquired data through an interface to generate analysis data, and inserting the analysis data into the data table of the same day;

specifically, step S102 includes: mapping a corresponding PLC protocol by defining a first interface body;

Specifically, in this embodiment, the gateway maps the corresponding PLC protocol through the defined plc_services interface body, for example, the acquisition modbusRTU protocol triggers the plc_mgmt_write interface to configure the acquisition parameters, and the mdbs_cmd_node_read interface performs framing acquisition. After the collected data can be analyzed through the mdbs_data_reporting interface, the show_stat thread displays the web state page, and meanwhile, the sqlite_write_history_data_table interface is called to search a data form of the date of the day from the built database, and the current data is inserted.

S103, searching a message group packet for reporting after judging that the callback trigger period time of the reporting thread is reached, wherein the message group packet consists of recent data records in a preset period in the database DB;

specifically, step S103 includes: creating a data linked list through a fourth interface, and extracting all data IDs to be reported from linked list nodes of the data linked list;

Specifically, in this embodiment, after the callback trigger period time of the reporting thread connected through mqtt is reached, the latest data record is searched for to form a json message to report by calling the sqlite_read_history_data_table interface. The part is a data link list created through list_for_each_safe, all data IDs needing to be reported are extracted from the link list nodes, then the data IDs are matched with the current absolute timestamp serving as a key value in the sqlite form, and the screened data are fed back into a complete record to be packaged.

S104, when judging that the message group package is effective and the communication with the docking platform is abnormal, storing the message group package into a cache table in the database DB;

in one possible embodiment of the present invention, after the packet is determined to be valid and communication with the docking platform is abnormal, storing the packet in a cache table in the database DB further includes: and when the communication on the docking platform is judged to be re-communicated, calling the third interface to search all unreported data in the cache table during disconnection, and reporting the unreported data to the docking platform one by one.

Specifically, in this embodiment, after the packet validity is determined, a determination is performed through a sending interface, for example, send_json is performed, if the communication with the platform is not normal, for example, if the connection is broken, the current message is stored in another Zhang Huancun table ff_bkdata_tx (X is serial numbers 1,2,3 of different data centers) of the database, and after the platform is re-accessed, a breakpoint continuous transmission is triggered, and likewise, all unreported records in the disconnection period are searched for and reported one by calling the sqlite_read_history_data_table interface.

And S105, when judging that the message group package is effective and the communication with the docking platform is normal, sending the message group package to the docking platform.

In one possible embodiment of the present invention, after the packet is determined to be valid and communication with the docking platform is normal, the method further includes: when judging that the sending of the message group packet is problematic, storing the message with failed sending into a cache table in the database DB;

Specifically, in this embodiment, after the validity of the group packet is determined, the sending interface, for example, send_json, is used to determine that if the docking platform is in normal communication, the mosquito_publish interface is called to send directly; the problem in the sending is generally to trigger the active disconnection, and then store the sending failure message into the database through the disconnect_callback_v25 callback interface to be processed by the interrupt continuous transmission mechanism.

In summary, the data storage and deployment method of the intelligent gateway is mainly used for solving the problem that the platform needs to acquire historical data for a certain time, but because the platform is limited by a storage space, a storage time period and release processing of used capacity need to be carefully configured, such as processing of deleting, transferring, compressing old data periodically. The key point of the intelligent gateway data storage deployment method is to improve the efficient use of data, the intelligent gateway data storage deployment method is stored in a sqlite database mode, and the prior art is stored in a text mode, wherein the difference between the two modes is as follows: firstly, the database storage mode is convenient and stable, a stack of redundant operations are not required to be carried out for each storage, only the processed interface and the sqlite instruction are required to be called, and the intelligent gateway data storage deployment method also needs to carry out a plurality of specific treatments according to different client requirements, so that the time is spent and the problem is easy to happen; secondly, a database mode is adopted to add and delete a certain piece or batch of data with high efficiency and accuracy, for example, the data is 1G, the database only takes a few nanoseconds to extract a piece of data from the database, and the intelligent gateway data storage deployment method needs tens of seconds, so that the difference is obvious; thirdly, the data stored in the database mode is not easy to lose and convenient to transfer, and the intelligent gateway data storage deployment method is opposite, and once synchronous processing is not performed, a large amount of data can be lost due to an abnormality.

In short, the intelligent gateway data storage deployment method can be used for structurally storing a large amount of PLC collection data information, is convenient for users to effectively search and access, can be used for classifying and storing data, and can provide quick inquiry. And the consistency and the integrity of data information can be effectively maintained, the data redundancy is reduced, the data stored in the sqlite database can be well ensured to be effective and not destroyed, and the database has the function of avoiding repeated data, so that the data redundancy is reduced. At the same time, it can meet the requirements of sharing and security of the application, and placing data in a database is also in many cases a safety concern, for example, if all data of different users are stored on disk files, then the confidentiality of data of some confidentiality authorities is not from the beginning: if the data is placed in the database, only the data of a particular user may be allowed to be queried and modified, thereby ensuring the security of the data.

Referring to fig. 4, a second embodiment of the present invention provides an intelligent gateway data storage deployment apparatus, including:

the data acquisition unit 201 is configured to start a main thread and three sub threads when a gateway process is started, acquire a current day data table in the database DB, configure an acquisition protocol, and trigger the gateway to perform an acquisition task;

the data inserting unit 202 is configured to parse the collected data through an interface to generate parsed data, and insert the parsed data into the data table of the current day;

the message generating unit 203 is configured to search a packet for reporting after determining that the reporting thread callback trigger period time reaches, where the packet is composed of recent data records in a preset period in the database DB;

a message buffering unit 204, configured to store the packet into a buffering table in the database DB when the packet is determined to be valid and communication with the docking platform is abnormal;

and the message sending unit 205 is configured to send the packet to the docking platform when the packet is determined to be valid and communication with the docking platform is normal.

A third embodiment of the present invention provides an intelligent gateway data storage deployment apparatus, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing an intelligent gateway data storage deployment method as described in any one of the above when executing the computer program.

A fourth embodiment of the present invention provides a readable storage medium storing a computer program executable by a processor of a device in which the storage medium is located to implement an intelligent gateway data storage deployment method according to any one of the above.

Illustratively, the computer programs described in the third and fourth embodiments of the present invention may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing particular functions to describe the execution of the computer program in the implementation of an intelligent gateway data storage deployment device. For example, the device described in the second embodiment of the present invention.

The processor may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is a control center of the intelligent gateway data storage deployment method, and the various interfaces and lines are used to connect the various parts of the entire implementation of the intelligent gateway data storage deployment method.

The memory may be used to store the computer program and/or the module, and the processor may implement various functions of an intelligent gateway data storage deployment method by running or executing the computer program and/or the module stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, a text conversion function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, text message data, etc.) created according to the use of the cellular phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.

Wherein the modules may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a stand alone product. Based on this understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of each method embodiment described above when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

It should be noted that the above-described apparatus embodiments are merely illustrative, and the units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the embodiment of the device provided by the invention, the connection relation between the modules represents that the modules have communication connection, and can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. The intelligent gateway data storage deployment method is characterized by comprising the following steps:

2. The intelligent gateway data storage deployment method according to claim 1, wherein three sub-threads are a configuration thread, an acquisition thread and a display thread, respectively, wherein the main thread is used for performing a docking platform reporting operation, the configuration thread is used for performing an automation operation, and the acquisition thread and the display thread are used for performing a database read-write operation.

3. The method for deploying the intelligent gateway data storage according to claim 2, wherein the method is characterized in that a data table of the current day in a database DB is obtained, and an acquisition protocol is configured, and the gateway is triggered to perform an acquisition task, specifically:

4. The method for storing and deploying the intelligent gateway data according to claim 1, wherein the analyzing the collected data through the interface to generate the analyzed data, and inserting the analyzed data into the data table of the current day specifically comprises:

mapping a corresponding PLC protocol by defining a first interface body;

5. The method for deploying intelligent gateway data storage according to claim 1, wherein the packet is composed of recent data records within a preset period in the database DB, specifically:

6. The intelligent gateway data storage deployment method according to claim 4, wherein after determining that the packet is valid and communication with the docking platform is abnormal, storing the packet in a cache table in the database DB, further comprises:

7. The method for deploying intelligent gateway data storage according to claim 6, wherein after determining that the packet is valid and the communication with the docking platform is normal, sending the packet to the docking platform further comprises:

8. An intelligent gateway data storage deployment apparatus, comprising:

9. An intelligent gateway data storage deployment apparatus comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing an intelligent gateway data storage deployment method as claimed in any one of claims 1 to 7 when the computer program is executed by the processor.

10. A readable storage medium storing a computer program executable by a processor of a device in which the storage medium is located to implement an intelligent gateway data storage deployment method according to any one of claims 1 to 7.