CN115858163A - Method and device for storing mass equipment instruction data of Internet of things platform - Google Patents
Method and device for storing mass equipment instruction data of Internet of things platform Download PDFInfo
- Publication number
- CN115858163A CN115858163A CN202211609056.8A CN202211609056A CN115858163A CN 115858163 A CN115858163 A CN 115858163A CN 202211609056 A CN202211609056 A CN 202211609056A CN 115858163 A CN115858163 A CN 115858163A
- Authority
- CN
- China
- Prior art keywords
- message
- instruction data
- module
- identification
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for storing mass equipment instruction data of an Internet of things platform, wherein the method comprises the following steps: the load balancer acquires the instruction data and realizes the balanced load of the instruction data writing request, and then sends the instruction data after balanced load to the corresponding message preprocessing module; the message preprocessing module is used for interpreting a tenant identifier, a product identifier, a device identifier and a first message body from a message request of instruction data and respectively sending the tenant identifier, the product identifier, the device identifier and the first message body to the message index encoding module and the message body compressing module; the message index coding module generates a message index code through the tenant identification, the product identification and the equipment identification; the message body compression module compresses the first message body to obtain a compression result; the message storage module encodes the message index as a row key index identification and writes the compression result as a second message body to the database. According to the invention, the instruction data are compressed, so that the efficient index storage of mass instruction data and the efficient utilization of cloud server resources are realized.
Description
Technical Field
The invention relates to the technical field of Internet of things, in particular to a method and a device for storing mass equipment instruction data of an Internet of things platform.
Background
With the scale development of the internet of things, the number of devices and applications accessed to the AIoT platform is continuously increased, the devices and applications can continuously trigger instruction issuing and query requests through the AIoT platform at different time points, the instruction data volume stored by the platform shows geometric growth, and how to realize efficient index storage of mass instruction data, efficient utilization of cloud server resources and organic balance of performance and cost is a major challenge to the scale development of the internet of things platform.
In the prior art, in order to save storage resources of a server, a distributed system based on HBASE usually starts a data compression characteristic of HBASE at a server, and this way has the main problems that occupation of a server CPU and a network transmission bandwidth is significantly increased when massive instruction data is accessed, hardware resource consumption and time delay are relatively high, and in order to obtain a relatively good compression ratio, a Block in HFile needs to be configured as large as possible, but since HBASE reads data in units of Block blocks, the larger the Block is, the worse random reading performance is, and a large amount of compression and frequent data decompression phenomena occur when a server processes a large amount of compression and requests; in addition, in the prior art, the message index code is formed by splicing a tenant identifier with 4 bytes, a product identifier with 4 bytes, a device identifier with 32 bytes and a message ID with 2 bytes, a row key index for optically storing 1 hundred million pieces of instruction data consumes about 4GB of memory space, and an excessively long row key index reduces the effective utilization rate of a system memory, cannot cache more data, and reduces the retrieval efficiency.
Disclosure of Invention
The invention provides a method and a device for storing mass equipment instruction data of an Internet of things platform, and aims to solve the problems of low data storage efficiency, low utilization rate of cloud server resources and the like in the prior art.
The embodiment of the invention provides a method for storing mass equipment instruction data of an Internet of things platform, which comprises the following steps:
the load balancer obtains the instruction data and realizes the balanced load of the instruction data write-in request;
the load balancer sends the instruction data after balancing the load to the corresponding message processing module; the message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module;
the message preprocessing module receives a message request written by the instruction data and interprets a tenant identification, a product identification, an equipment identification and a first message body from the message request;
the message preprocessing module sends the tenant identification, the product identification and the equipment identification to a message index coding module, and the message preprocessing module sends the first message body to a message body compression module;
the message index coding module generates a message index code through the tenant identification, the product identification and the equipment identification and sends the message index code to the message storage module;
the message body compression module compresses the first message body by adopting a compression algorithm to obtain a compression result and sends the compression result to the message storage module;
the message storage module takes the message index code as a line key index identification and writes the compression result into a database as a second message body.
Further, the load balancer for realizing the balanced load of the instruction data write request comprises:
and the load balancer adopts a K8SService mode to establish a virtual IP address for realizing the balanced load of the instruction data writing request.
Further, the generating, by the message index encoding module, a message index encoding through the tenant identification, the product identification, and the device identification includes:
performing feature calculation on the tenant identification, the product identification and the equipment identification to obtain a message feature value;
splicing, modulus taking and mapping the equipment identification to obtain a message ID value;
carrying out partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster to obtain a partition discrete value;
and splicing the message characteristic value, the message ID value and the partition discrete value to obtain the message index code.
Further, the performing feature calculation on the tenant identity, the product identity, and the device identity to obtain a message feature value includes: and performing character string splicing on the tenant identification, the product identification and the equipment identification by taking underlines as separators, then calculating the message characteristic value by adopting an MD5 algorithm, and converting the message characteristic value into 4-byte unsigned integer.
Further, the calculation formula of the MD5 algorithm is:
F1＝str2uint(md5(T1_P1_D1))
wherein F1 represents a message characteristic value; str2uint represents a string-to-unsigned integer function; md5 denotes the md5 function; t1 represents a tenant identity; a represents an underline; p1 represents a product identifier; d1 denotes a device identification.
Further, the splicing, modulus taking and mapping the device identifier to obtain the message ID value includes:
splicing the equipment identification and the custom character string prefix to obtain a key;
and generating a temporary message ID value by the key in a REDIS self-increment sequence mode, and mapping the temporary message ID value in a tenant address space range in a modulo mode to obtain a message ID value.
Further, the calculation formula for generating the message ID value by the key in the manner of the REDIS self-increment sequence is as follows:
I1＝redis.incr(seq:D1)mod65535
wherein I1 represents a message ID value; incr denotes the redis self-increasing sequence function; seq D1 represents a bond; d1 represents a device identification; seq: representing a custom string prefix; mod denotes modulo.
Further, the calculation formula for carrying out partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster is as follows:
Z1＝abs(F1modN1)
wherein Z1 represents a partition discrete value; abs represents a function for solving the absolute value of data; f1 represents a message characteristic value; mod represents the modulus; n1 represents the number of partitions of the HBASE cluster.
Further, the splicing the message characteristic value, the message ID value, and the partition discrete value includes:
and character string splicing is carried out on the message characteristic value, the message ID value and the partition discrete value by taking an underlining line as a separator to obtain the message index code.
The embodiment of the invention also provides a device for storing the instruction data of the mass equipment of the platform of the internet of things, which comprises the following components: the system comprises a load balancer, a plurality of message processing modules and a database for storing instruction data; each message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module;
the load balancer is used for acquiring the instruction data and realizing the balanced load of the instruction data write-in request;
the load balancer is also used for sending the instruction data after balancing the load to the corresponding message processing module;
the message preprocessing module is used for receiving a message request written by the instruction data and interpreting a tenant identifier, a product identifier, an equipment identifier and a first message body from the message request;
the message preprocessing module is further used for sending the tenant identification, the product identification and the equipment identification to a message index code, and the message preprocessing module sends the first message body to a message body compression module;
the message index coding module is used for generating a message index code through the tenant identification, the product identification and the equipment identification and sending the message index code to the message storage module;
the message body compression module is used for compressing the first message body by adopting a compression algorithm to obtain a compression result and sending the compression result to the message storage module;
the message storage module is used for coding the message index as a row key index identifier and writing the compression result as a second message body into a database.
The embodiment of the invention provides a method for storing mass equipment instruction data of an Internet of things platform, which comprises the following steps: the load balancer obtains the instruction data and realizes the balanced load of the instruction data write-in request; the load balancer sends the instruction data after balancing the load to the corresponding message processing module; the message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module; the message preprocessing module receives a message request written by the instruction data and interprets a tenant identification, a product identification, an equipment identification and a first message body from the message request; the message preprocessing module sends the tenant identification, the product identification and the equipment identification to a message index coding module, and the message preprocessing module sends the first message body to a message body compression module; the message index coding module generates a message index code through the tenant identification, the product identification and the equipment identification and sends the message index code to the message storage module; the message body compression module compresses the first message body by adopting a compression algorithm to obtain a compression result and sends the compression result to the message storage module; and the message storage module encodes the message index as a row key index identification and writes the compression result as a second message body into a database. The method is based on a micro-service architecture with data storage and calculation separated, and realizes the balance between the efficient index storage of mass instruction data, the efficient utilization of cloud server resources and the performance and cost by compressing the instruction data. The embodiment of the invention also provides a storage device for the instruction data of the mass equipment of the platform of the Internet of things, which has the beneficial effects and is not described herein again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a method for storing instruction data of mass devices of an internet of things platform according to an embodiment of the present invention;
fig. 2 is a sub-flow diagram of a method for storing instruction data of mass devices of an internet of things platform according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of an instruction data storage device of a mass device of an internet of things platform according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1, the embodiment provides a method for storing instruction data of mass devices of an internet of things platform, including:
s101: the load balancer obtains the instruction data and realizes the balanced load of the instruction data write-in request;
in this embodiment, a virtual IP address is created in a K8SService manner, so as to implement balanced load of an instruction data write request;
the working process comprises the following steps: and adopting a K8SService service to create a virtual IP address (clusterip), when a user accesses an interface written by instruction data, accessing the clusterip in the service, replacing the user by the service to find a corresponding pod, and forwarding the request to the corresponding pod for service processing.
S102: the load balancer sends the instruction data after balancing the load to the corresponding message processing module; the message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module;
after the load balancer sends the instruction data after balancing the load to the corresponding message processing module, the message preprocessing module in the message processing module preprocesses the instruction data after balancing the load and then sends the instruction data to the message index coding module and the message body compression module in the message processing module.
S103: the message preprocessing module receives a message request written by the instruction data and interprets a tenant identification, a product identification, an equipment identification and a first message body from the message request;
the tenant identification, the product identification and the equipment identification are respectively global unique identifications distributed by the Internet of things platform when a user registers an account number, creates a product and adds equipment on the Internet of things platform; the tenant identification and the product identification are 4-byte integer, and the device identification is a 32-byte UUID character string.
S104: the message preprocessing module sends the tenant identification, the product identification and the equipment identification to a message index coding module, and the message preprocessing module sends the first message body to a message body compression module;
s105: the message index coding module generates a message index code through the tenant identification, the product identification and the equipment identification and sends the message index code to the message storage module;
referring to fig. 2, the message index encoding module generates the message index encoding by the tenant identification, the product identification and the device identification, and includes:
s201: performing feature calculation on the tenant identification, the product identification and the equipment identification to obtain a message feature value;
specifically, character string splicing is carried out on the tenant identification, the product identification and the equipment identification by taking underlining lines as separators, then an MD5 algorithm is adopted to calculate a message characteristic value, and the message characteristic value is converted into 4-byte unsigned integer.
The calculation formula of the MD5 algorithm is as follows:
F1＝str2uint(md5(T1_P1_D1))
wherein F1 represents a message characteristic value; str2uint represents a string-to-unsigned integer function; md5 denotes the md5 function; t1 represents a tenant identity; a represents an underline; p1 represents a product identifier; d1 denotes a device identification.
In one embodiment, character string splicing is performed on a tenant identifier T1, a product identifier P1 and an equipment identifier D1 by using underlining lines as separators to obtain T1_ P1_ D1, then an MD5 algorithm is adopted to calculate the T1_ P1_ D1 to obtain a message characteristic value F1, and then the message characteristic value F1 is converted into 4-byte unsigned integer.
S202: splicing, modulus taking and mapping the equipment identification to obtain a message ID value;
specifically, splicing the equipment identifier and the user-defined character string prefix to obtain a key; and generating a temporary message ID value by the key in a REDIS self-increment sequence mode, and mapping the temporary message ID value in a tenant address space range in a modulo mode to obtain a message ID value.
The calculation formula for generating the message ID value by the key in the REDIS self-increment sequence mode is as follows:
I1＝redis.incr(seq:D1)mod 65535
wherein I1 represents a message ID value; incr denotes the redis self-increasing sequence function; seq D1 represents a bond; d1 represents a device identification; seq: representing a custom string prefix; mod denotes modulo.
In one embodiment, the device identifier D1 and a custom character string prefix (seq:) are spliced to obtain a key named seq: D1; and then generating a temporary message ID value by the key in a REDIS self-increment sequence mode, and mapping the temporary message ID value in a tenant address space range in a modulo mode to obtain a message ID value I1.
The message ID value occupies 2 bytes, is used for uniquely determining the address space index position of a certain device under the corresponding tenant name when instruction data is written, and supports cyclic addressing utilization from 0 to 65535.
S203: carrying out partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster to obtain a partition discrete value;
the calculation formula for carrying out partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster is as follows:
Z1＝abs(F1 modN1)
wherein Z1 represents a partition discrete value; abs represents a function for solving the absolute value of data; f1 represents a message feature value; mod represents the modulus; n1 represents the number of partitions of the HBASE cluster.
The purpose of partition dispersion in this embodiment is to break up the row key index identifier of the HBASE, so that the instruction data can be sufficiently dispersed to each partition server in the HBASE cluster when being stored, and the resources of each partition server in the HBASE cluster can be sufficiently scheduled and utilized.
S204: and splicing the message characteristic value, the message ID value and the partition discrete value to obtain the message index code.
Specifically, character string splicing is performed on the message characteristic value, the message ID value and the partition discrete value by using the underlining as a separator to obtain the message index code.
In one embodiment, the following scribe lines are used as separators to perform string splicing on the message characteristic value F1, the message ID value I1 and the partition discrete value Z1, so as to obtain a message index code: k1= Z1_ F1_ I1, where K1 denotes a message index coding.
S106: the message body compression module compresses the first message body by adopting a compression algorithm to obtain a compression result and sends the compression result to the message storage module;
in order to reduce the occupation of the row key index memory, the data storage space and the network communication burden, the embodiment compresses the first message body M1 by using a compression algorithm to obtain a compression result M2.
The SMAZ is a small object-based compression algorithm library and is mainly applied to a Redis database at present. Compared with the traditional compression algorithm, the SMAZ is very suitable for compressing small objects, such as several bytes to dozens of bytes, the average test can achieve about 40% -50% of compression effect, the SMAZ has extremely high compression efficiency and decompression efficiency, and the SMAZ is very suitable for a processing scene of small object storage, namely Internet of things platform instruction data, so that the SMAZ compression algorithm is adopted to compress the first message body M1 to obtain the compression result M2.
S107: and the message storage module encodes the message index as a row key index identification and writes the compression result as a second message body into a database.
In one embodiment, the message index code K1 is used as the row key index identifier, and the compression result M2 is used as the second message body to write the instruction data into the HBASE distributed database.
In the embodiment, the HBASE distributed database cluster is designed to be only responsible for storage management of mass instruction data, and responsibilities (such as compression, decompression and the like) except for other storage are processed by an external stateless micro-service, so that the pressure of the HBASE storage cluster is reduced from the source, and the dependence on a high-performance server is reduced; secondly, message index coding is carried out on the HBASE row key before the instruction data are delivered, the length of the coded row key index is only 16% of that of the original row key index, the first message body is compressed by adopting an SMAZ algorithm, the compression ratio can reach 50% to the maximum extent, and the occupation of a row key index memory, the data storage space and the network communication burden are greatly reduced; by inserting the partition discrete identification into the message index code, the resources of each partition server in the HBASE cluster are fully scheduled and utilized, and the data read-write performance is greatly improved. The stateless micro-service support adopts a dynamic capacity expansion and contraction mode to distribute the number of the operation instances as required, realizes the efficient utilization of calculation, storage and network resources and the stable operation of the system, and is lighter in weight compared with the capacity expansion of an HBASE server. In conclusion, the invention reduces the pressure of HBASE storage clusters from the source, reduces the dependency on high-performance servers, has good expandability and has good popularization and use values.
Referring to fig. 3, the embodiment further provides an instruction data storage device for mass devices of an internet of things platform, including: a load balancer 3100, a plurality of message processing modules 3200, and a database 3300 for storing instruction data; each message processing module 3200 comprises a message preprocessing module 3210, a message index coding module 3220, a message body compression module 3230 and a message storage module 3240;
the load balancer 3100 is used for obtaining the instruction data and implementing the balanced load of the instruction data write request;
the load balancer 3100 is further configured to send the instruction data after balancing the load to the corresponding message processing module 3200;
the message preprocessing module 3210 is configured to receive a message request for instruction data writing, and interpret a tenant identifier, a product identifier, an equipment identifier, and a first message body from the message request;
the message preprocessing module 3210 is further configured to send the tenant identifier, the product identifier, and the device identifier to the message index encoding module 3220, and the message preprocessing module 3210 sends the first message body to the message body compression module 3230;
the message index encoding module 3220 is configured to generate a message index encoding through the tenant identifier, the product identifier, and the device identifier, and send the message index encoding to the message storage module 3240;
the message body compression module 3230 is configured to compress the first message body by using a compression algorithm, obtain a compression result, and send the compression result to the message storage module 3240;
the message storage module 3240 is used to encode the message index as a row key index identification and write the compression result as a second message body to the database 3300.
Further, the load balancer 3100 is further configured to create a virtual IP address in a K8SService manner, so as to achieve balanced load of the instruction data write request.
Further, the message index encoding module 3220 includes:
the message characteristic calculating unit 3221 is configured to perform characteristic calculation on the tenant identifier, the product identifier, and the device identifier to obtain a message characteristic value;
a message ID addressing unit 3222, configured to perform splicing, modular extraction, and mapping processing on the device identifier to obtain a message ID value;
a partition discrete calculation unit 3223, configured to perform partition discrete calculation on the message characteristic value and the number of partitions of the HBASE cluster to obtain a partition discrete value;
a message index generating unit 3224 is configured to perform splicing processing on the message characteristic value, the message ID value, and the partition discrete value to obtain the message index code.
Further, the message characteristic calculating unit 3221 is further configured to perform character string splicing on the tenant identifier, the product identifier, and the device identifier by using a following line as a separator, and then calculate the message characteristic value by using an MD5 algorithm and convert the message characteristic value into a 4-byte unsigned integer.
Further, the message characteristic calculating unit 3221 is further configured to calculate a message characteristic value according to the following formula:
F1＝str2uint(md5(T1_P1_D1))
wherein F1 represents a message characteristic value; str2uint represents a string-to-unsigned integer function; md5 denotes the md5 function; t1 represents a tenant identity; a represents an underline; p1 represents a product identifier; d1 denotes a device identification.
Further, the message ID addressing unit 3222 includes:
the splicing subunit is used for splicing the equipment identifier and the custom character string prefix to obtain a key;
and the generating subunit is used for generating the temporary message ID value from the key by adopting a REDIS self-increment sequence mode and mapping the temporary message ID value in a tenant address space range in a modulo mode to obtain the message ID value.
Further, the generating subunit is further configured to generate the message ID value as follows:
I1＝redis.incr(seq:D1)mod65535
wherein I1 represents a message ID value; incr denotes the redis self-increasing sequence function; seq D1 represents a bond; mod denotes modulo.
Further, the partition dispersion calculating unit 3223 is further configured to calculate the partition dispersion value according to the following formula:
Z1＝abs(F1modN1)
wherein Z1 represents a partition discrete value; abs represents a function for solving the absolute value of data; f1 represents a message characteristic value; mod represents the modulus; n1 represents the number of partitions of the HBASE cluster.
Further, the message index generating unit 3224 is further configured to perform character string splicing on the message feature value, the message ID value, and the partition discrete value by using an underline as a separator to obtain the message index code.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A storage method for mass equipment instruction data of an Internet of things platform is characterized by comprising the following steps:
the load balancer obtains the instruction data and realizes the balanced load of the instruction data write-in request;
the load balancer sends the instruction data after balancing the load to the corresponding message processing module; the message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module;
the message preprocessing module receives a message request written by the instruction data and interprets a tenant identification, a product identification, an equipment identification and a first message body from the message request;
the message preprocessing module sends the tenant identification, the product identification and the equipment identification to a message index coding module, and the message preprocessing module sends the first message body to a message body compression module;
the message index coding module generates a message index code through the tenant identification, the product identification and the equipment identification and sends the message index code to the message storage module;
the message body compression module compresses the first message body by adopting a compression algorithm to obtain a compression result and sends the compression result to the message storage module;
and the message storage module encodes the message index as a row key index identification and writes the compression result as a second message body into a database.
2. The method for storing the instruction data of the mass device on the platform of the internet of things according to claim 1, wherein the load balancer for balancing the load of the instruction data write request comprises:
and the load balancer adopts a K8SService mode to create a virtual IP address for realizing the balanced load of the instruction data write-in request.
3. The method for storing the instruction data of the mass device on the platform of the internet of things according to claim 1, wherein the generating of the message index code by the message index coding module through the tenant identifier, the product identifier and the device identifier comprises:
performing feature calculation on the tenant identification, the product identification and the equipment identification to obtain a message feature value;
splicing, modulus taking and mapping the equipment identification to obtain a message ID value;
carrying out partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster to obtain a partition discrete value;
and splicing the message characteristic value, the message ID value and the partition discrete value to obtain the message index code.
4. The method for storing the instruction data of the mass equipment on the platform of the internet of things according to claim 3, wherein the step of performing feature calculation on the tenant identifier, the product identifier and the equipment identifier to obtain a message feature value comprises the steps of: and performing character string splicing on the tenant identification, the product identification and the equipment identification by taking underlines as separators, then calculating the message characteristic value by adopting an MD5 algorithm, and converting the message characteristic value into 4-byte unsigned integer.
5. The Internet of things platform mass device instruction data storage method according to claim 4, wherein the calculation formula of the MD5 algorithm is as follows:
F1＝str2uint(md5(T1_P1_D1))
wherein F1 represents a message characteristic value; str2uint represents a string-to-unsigned integer function; md5 denotes the md5 function; t1 represents a tenant identity; a represents an underline; p1 represents a product identifier; d1 denotes a device identification.
6. The method for storing the instruction data of the mass equipment on the platform of the internet of things according to claim 3, wherein the splicing, the modulus extraction and the mapping of the equipment identifier are performed to obtain the message ID value, and the method comprises the following steps:
splicing the equipment identification and the custom character string prefix to obtain a key;
and generating an interim message ID value by the key in a REDIS self-increment sequence mode, and mapping the interim message ID value in a tenant address space range in a modulo mode to obtain a message ID value.
7. The method for storing the instruction data of the mass device on the platform of the internet of things according to claim 6, wherein the calculation formula for generating the message ID value of the key by using the REDIS self-increment sequence mode is as follows:
I1＝redis.incr(seq:D1)mod 65535
wherein I1 represents a message ID value; incr denotes the redis self-increasing sequence function; seq D1 represents a bond; d1 represents a device identification; seq: representing a custom string prefix; mod denotes modulo.
8. The method for storing the instruction data of the mass device on the platform of the internet of things as claimed in claim 3, wherein the calculation formula for performing the partition discrete calculation on the message characteristic value and the partition number of the HBASE cluster is as follows:
Z1＝abs(F1 modN1)
wherein Z1 represents a partition dispersion value; abs represents a function for solving the absolute value of data; f1 represents a message characteristic value; mod represents the modulus; n1 represents the number of partitions of the HBASE cluster.
9. The method for storing the instruction data of the mass device on the platform of the internet of things according to claim 3, wherein the splicing the message characteristic value, the message ID value and the partition discrete value comprises:
and character string splicing is carried out on the message characteristic value, the message ID value and the partition discrete value by taking an underlining line as a separator to obtain the message index code.
10. The utility model provides an thing networking platform mass equipment instruction data storage device which characterized in that includes: the system comprises a load balancer, a plurality of message processing modules and a database for storing instruction data; each message processing module comprises a message preprocessing module, a message index coding module, a message body compression module and a message storage module;
the load balancer is used for acquiring the instruction data and realizing the balanced load of the instruction data write-in request;
the load balancer is also used for sending the instruction data after balancing the load to the corresponding message processing module;
the message preprocessing module is used for receiving a message request written by the instruction data and interpreting a tenant identifier, a product identifier, an equipment identifier and a first message body from the message request;
the message preprocessing module is further used for sending the tenant identification, the product identification and the equipment identification to a message index coding module, and the message preprocessing module sends the first message body to a message body compression module;
the message index coding module is used for generating a message index code through the tenant identification, the product identification and the equipment identification and sending the message index code to the message storage module;
the message body compression module is used for compressing the first message body by adopting a compression algorithm to obtain a compression result and sending the compression result to the message storage module;
the message storage module is used for coding the message index as a row key index identification and writing the compression result as a second message body into a database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211609056.8A CN115858163A (en) | 2022-12-14 | 2022-12-14 | Method and device for storing mass equipment instruction data of Internet of things platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211609056.8A CN115858163A (en) | 2022-12-14 | 2022-12-14 | Method and device for storing mass equipment instruction data of Internet of things platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115858163A true CN115858163A (en) | 2023-03-28 |
Family
ID=85672934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211609056.8A Pending CN115858163A (en) | 2022-12-14 | 2022-12-14 | Method and device for storing mass equipment instruction data of Internet of things platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115858163A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117220686A (en) * | 2023-09-18 | 2023-12-12 | 青岛展诚科技有限公司 | Parasitic parameter compression and extraction system and method |
-
2022
- 2022-12-14 CN CN202211609056.8A patent/CN115858163A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117220686A (en) * | 2023-09-18 | 2023-12-12 | 青岛展诚科技有限公司 | Parasitic parameter compression and extraction system and method |
| CN117220686B (en) * | 2023-09-18 | 2024-02-23 | 青岛展诚科技有限公司 | Parasitic parameter compression and extraction system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9477682B1 (en) | Parallel compression of data chunks of a shared data object using a log-structured file system | |
| CN110222048B (en) | Sequence generation method, device, computer equipment and storage medium | |
| CN103348334B (en) | Cloud system and the compressing file in cloud system and transfer approach | |
| TWI735545B (en) | Model training method and device | |
| CN107196998B (en) | Mobile Web request processing method, equipment and system based on data deduplication | |
| CN111611225A (en) | Data storage management method, query method, device, electronic equipment and medium | |
| CN103873438A (en) | Compression packet uploading and duplication-removing system and method | |
| CN106874348A (en) | File is stored and the method for indexing means, device and reading file | |
| CN115858163A (en) | Method and device for storing mass equipment instruction data of Internet of things platform | |
| CN102469142A (en) | Data transmission method for data deduplication program | |
| CN108108247A (en) | Distributed picture storage service system and method | |
| CN109325089A (en) | A kind of non-pointing object querying method, device, terminal device and storage medium | |
| Upadhyay et al. | Deduplication and compression techniques in cloud design | |
| CN102214170B (en) | Methods and systems for compressing and decompressing extensible markup language (XML) data | |
| CN101483844B (en) | Method and system for indexing mobile telephone number | |
| CN106959997A (en) | Document handling method and device based on cluster file system | |
| CN114125071A (en) | Data compression transmission method and device | |
| KR20120087225A (en) | Method for tile-map service using solid compression, and system for the same | |
| CN110311754B (en) | Data receiving method and device, storage medium and electronic equipment | |
| CN115438046B (en) | Distributed storage method based on HTAP database | |
| CN111259013A (en) | Method and device for storing data | |
| CN108234552B (en) | Data storage method and device | |
| KR102236521B1 (en) | Method and apparatus for processing data | |
| CN114238264A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN111697973B (en) | Compression method and compression system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |