CN112347125B

CN112347125B - Equipment data processing method and Internet of things data processing method

Info

Publication number: CN112347125B
Application number: CN202011282844.1A
Authority: CN
Inventors: 蒋宗文
Original assignee: Individual
Current assignee: Individual
Priority date: 2020-11-16
Filing date: 2020-11-16
Publication date: 2024-06-11
Anticipated expiration: 2040-11-16
Also published as: CN112347125A

Abstract

The invention provides a method for processing equipment data and a method for processing data of the Internet of things, comprising the following steps: treating a data column which cannot meet the 1-format in the data in the device specific format as a character string with an indefinite length or a column in a binary data format as a structured data column meeting the 1-format; a schema attribute is added to the data columns which cannot satisfy the 1-mode, so that the extended compiler can understand the specific formats of the data columns which cannot satisfy the 1-mode and process the data. The scheme provided by the invention has the following beneficial effects: the method solves the problems of insufficient SQL language expansion capability, low performance, high requirements on users, unstable system and difficult interactive auxiliary design in the data processing process, greatly improves the data processing performance, has great expansion capability and system stability, and improves the cost performance.

Description

Equipment data processing method and Internet of things data processing method

Technical Field

The present invention relates to the field of information technologies, and in particular, to a method for processing device data, and a method and apparatus for processing data in the internet of things.

Background

With the massive application of the internet of things, various data processing systems related to the internet of things appear, and the common characteristics of the systems are that data needs to be collected, converted in format, filtered, statistically analyzed, linked, sent, displayed or stored, wherein the key points are format conversion, filtration, statistically analyzed and linked. Typical application systems implement these data processing processes mainly using program-curing rules, and also partly using scripting languages. The prior art either uses program-cured rules to implement data processing, which limits the ability to connect only a few devices that were considered when the program rules were initially written, without expansion capabilities. Or the script language is adopted for processing, and higher hardware is required for processing the equivalent data due to the low processing performance of the script language; moreover, the script language processing program can be written by special skills, and a common data engineer cannot use the script language processing program; moreover, the script language is easy to make mistakes, so that the stability of the system is poor; the scripting language is more arbitrary and is not conducive to assisting in generation through an interactive interface.

Interpretation of related terms:

Internet of things equipment: such as various sensors, alarm devices, etc., interconnected with other devices via various networking protocols.

1 Formula (1 NF): a specification of a data type. The data is required to be divided into a fixed number of columns, and the data type of the columns is simple and determined, but can be only limited to a numerical value type, a string type, a date and time type, a binary byte type, and the like.

Structured data: also called relationship data. The 1-range specification must be satisfied.

Semi-structured data: the data may have any number of columns, and a column may be a combination of other data such as an Array (ordered combination of a plurality of data) or an Object (unordered combination of a plurality of attribute data) in addition to the simple data types described above. The data within the array and object may also continue to be semi-structured data.

Structured Query Language (SQL): is an international standard for structured database query languages.

JSON: javascript Object Notation (object representation format of the web programming language Javascript), is the most common format for structured and semi-structured data transfer between application systems, in particular internet application systems.

Compiling: one computing language is translated into another computer language.

Scripting language: the generic term for lightweight computer programming languages for execution is explained.

A command: and special data received by the Internet of things equipment, wherein the special data indicate the equipment to perform specific actions. For example, when the door access device receives a door opening command, the door lock managed by the door access device is opened.

The existing treatment methods have the following defects:

the internet of things devices are of a wide variety, and each device has its own unique data/command format and communication protocol, and these data/commands are typically semi-structured data.

The problems are that:

1) The method has the advantages that a standardized, simple and easy and efficient way is not provided for filtering and converting the data/command format of one device into the data/command format of another device, and special writing programs are needed; there is no standardized, simple and easy to implement efficient way to describe the processing procedures of filtering, converting, statistical analysis, etc. of data, and special writing procedures are required. The prior art has complex data conversion and processing procedures and needs to consume huge manpower, material resources and financial resources.

2) The prior art can only connect a few devices that were considered when writing program rules, without expansion capabilities.

The development of a general, convenient and low-cost method or system capable of identifying and processing data of different types of equipment is a technical problem to be solved.

Disclosure of Invention

The invention aims to overcome the defects of the prior art, and provides an SQL language extension method, an Internet of things data processing device and an Internet of things data processing system, so as to solve the technical problem that the prior art cannot be simply and easily connected with different types of Internet of things equipment, collect data of the Internet of things equipment and analyze and count the data.

The invention aims to provide a method for processing equipment data, which treats a data column which cannot meet 1-format in equipment-specific format data as a character string with an indefinite length or a column with a binary data format, and processes the character string or the column as a structured data column which meets 1-format; a schema attribute is added to the data columns which cannot satisfy the 1-mode, so that the extended compiler can understand the specific formats of the data columns which cannot satisfy the 1-mode and process the data.

Preferably, the extended compiler generates data format conversion program code based on the schema attribute appended to the data columns that fail to satisfy the 1-gram.

Preferably, the data format conversion program code is invoked by an extended SQL executor and/or a data format decoding component and/or a data format encoding component.

More preferably, the data in the specific format includes: xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr format.

Preferably, the extension compiler may support one or more of xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr formats.

Preferably, the syntax of the extended SQL includes additional device types, device communication addresses, and device-specific options after creating the table statement to reference the additional devices to continuously generate data streams in a specific format.

Preferably, a string or binary type column of the table statement is created, followed by data format and pattern information indicating the structure information inside the data column.

Preferably, the extension compiler uses the schema information to map the device data stream into a JSON data structure.

More preferably, format extensions of data format and mode information are added to format clauses of functions of json_exists and/or json_value and/or json_query and/or json_table input JSON format to process data columns of multiple formats.

More preferably, a format extension of data format and mode information is added to a format clause of a function outputting JSON format in json_object and/or json_array and/or json_ OBJECTAGG and/or json_ ARRAYTAG to output multiple format data columns.

The invention also comprises an Internet of things data processing method, wherein the extended SQL compiler receives an extended SQL instruction, extracts the mode information of the data column according to the table name and the format clause in the extended SQL instruction, transmits the mode information to the decoding component and/or the encoding component, and generates the data processing program code required by the executor according to the mode information.

More preferably, the system further comprises a component manager, which records which communication protocol type and/or which data format is supported by each data format decoding component and data format encoding component, and matches the corresponding decoding component or encoding component according to the communication protocol type and/or data format information.

Preferably, the decoding component converts the data in the specific format of the device into unified structured data meeting the 1-norm according to the mode information of the input data transmitted by the extension compiler, and transmits the structured data to the compiled executor for processing.

Preferably, the compiled executor processes the structured data meeting the 1-scale according to the data processing program code generated by the extended SQL compiler, and sends the processed result data to the encoding component for processing.

Preferably, the coding component codes unified structured data meeting 1-range into data in a specific format of the device according to the mode information of the input data transmitted by the expansion compiler, and the data is transmitted to the device data output component for processing.

The invention also includes an internet of things data processing device, comprising: and the extended SQL compiler is used for receiving the extended SQL instruction, extracting the mode information of the data column according to the table name and the format clause in the extended SQL instruction, transmitting the mode information to the decoding component or the encoding component, and generating the data processing program code required by the executor according to the mode information.

More preferably, the system further comprises a component manager, configured to record which communication protocol type and/or which data format is supported by each data format decoding component and data format encoding component, and match the corresponding decoding component or encoding component according to the communication protocol type and/or data format information.

Preferably, the decoding component is configured to convert the data in the device specific format into unified structured data meeting the 1-format according to the mode information of the input data transmitted from the extension compiler, and transmit the structured data to the compiled executor for processing.

Preferably, the compiled executor is configured to process the structured data meeting the 1-range according to the data processing program code generated by the extended SQL compiler, and to deliver the processed result data to the encoding component for processing.

Preferably, the encoding component is configured to encode the unified structured data satisfying the 1-range format into data in a specific format according to the mode information of the input data transmitted from the extension compiler, and transmit the data to the device data output component for processing.

The invention also includes an internet of things data processing system, comprising: the internet of things data processing device of any one of the above.

The invention also comprises a computer program product for data processing of the internet of things, comprising program code means adapted to be on a computer, executing the SQL language extension method for device data processing according to any of the above at run-time.

The invention also comprises a computer readable storage medium storing a computer program according to any one of claims 1-15, the computer program according to any one of the above methods being executable by one or more processors to implement a method of processing a data column of a device specific format that cannot satisfy a 1-format as a column of a variable length string or binary data format as a structured data column that satisfies a 1-format.

The scheme of the invention has the beneficial effects that:

1) The SQL language expansion method uses the format of the grammar expansion column and the mode information of the description column to process (including conversion, filtering, cleaning and statistical analysis) structured and semi-structured data in any format in a unified standardized way.

2) The method for compiling SQL language to efficiently process data collected in real time can directly utilize compiled SQL sentences to process the data without storing and warehousing and collecting the data in a memory, and the processed data can be directly sent to a next data processing unit in the memory.

3) The widely used SQL is expanded to be used as a basic data processing language, and can be applied to conversion, filtering, cleaning, statistical analysis and linkage processing of any structured and semi-structured data format; further, the compiling technology is adopted, the data processing performance can be greatly improved, and more data can be processed by the same computing capacity.

4) The invention further expands the JSON data processing part of the international standard SQL-2016 language to support data processing of any format to support a plurality of structured and semi-structured data, and compiles the expanded SQL instruction into a machine language to improve the performance of data processing.

5) By using the technical scheme of the invention, the structured and semi-structured data in any format of any equipment can be processed, and the invention has great data processing capacity and expansion capacity.

6) The technical scheme of the invention greatly reduces the skill requirement of users and reduces the operation and maintenance cost of the data processing system based on the technology.

7) The extended SQL instructions can be compiled into machine instructions which are executed efficiently, so that the data processing performance is greatly improved. The data processing system based on the technology can process more data under the same hardware capacity, and improves the cost performance.

8) The technical scheme of the invention increases the stability of the system.

Drawings

The application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The embodiments are not limiting, in which like numerals represent like structures, wherein:

FIG. 1 is a schematic diagram of an Internet of things data processing apparatus;

FIG. 2 is a schematic diagram of an extended SQL compiler processing extended SQL instructions;

FIG. 3 is a schematic diagram of an extended SQL executor, a data format decoding component, and a data format encoding component according to the processed data.

Detailed Description

In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings that are required to be used in the description of the embodiments will be briefly described below. It is apparent that the drawings in the following description are only some examples or embodiments of the present application, and technical features of the respective embodiments may be combined with each other to form a practical solution for achieving the object of the present application, and it is also possible for one of ordinary skill in the art to apply the present application to other similar situations according to these drawings without inventive efforts. Unless otherwise apparent from the context of the language or otherwise specified, like reference numerals in the figures refer to like structures or operations.

It will be appreciated that "system," "apparatus," "unit" and/or "module" as used herein is one method for distinguishing between different components, elements, parts, portions or assemblies of different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

As used in the specification and in the claims, the terms "a," "an," "the," and/or "the" are not specific to a singular, but may include a plurality, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.

Embodiment one:

A method for processing data of a device regards a data column which cannot meet 1-format in data of a device specific format as a string with an indefinite length or a column with a binary data format, and processes the string as a structured data column which meets 1-format; a schema attribute is added to the data columns which cannot satisfy the 1-mode, so that the extended compiler can understand the specific formats of the data columns which cannot satisfy the 1-mode and process the data.

The invention treats all kinds of data which cannot meet 1-format (1 NF) in different device-specific format data, including semi-structured data columns as columns of an indefinite length character string (string) type or binary byte data (binary) type, so that the data can be input into the system for processing as structured data columns meeting 1-format. This allows a continuous stream of device data in the same format to be considered a table of SQL. In order for the extended SQL compiler and various data format decoding and encoding components to understand the specific format of these columns, the invention extends the syntax of the SQL Create Table, requiring that the string and binary columns that contain semi-structured data be attached with a schema (schema) attribute, examples of which are as follows:

Create Table GateDevice

for gate device

at‘http://192.168.1.1:2010/gate’

options '< other device options >'.

(

gate_id integer,

gate_status binary format xml‘<XML-DTD-schema>’encoding utf8

)

Three clauses for, at and options from for to the left bracket in the above-described SQL instruction are extensions of the invention to SQL2016, the for clause indicating that the table is actually a gate access device, the at clause indicating that the access device is accessible via an http protocol address, and the options indicating other device-specific options when the device is connected via a communications protocol. In the above instruction, the column gate_status is a column in XML format, and its specific format is specified by the following '< XML-DTD-schema >' schema. The syntax part starting from format is also an extended part of the invention (excluding XML-DTD-schema, which is specified by the XML standard) for providing the schema information of the columns.

Once the extended SQL compiler has this schema information, it can generate efficient data format conversion program code for the corresponding column compilation. Compilation is referred to herein as compiling the SQL language into a machine language. So that structured and semi-structured data in any format can be processed (including conversion, filtering, cleaning, statistical analysis) using a unified standardized manner. Greatly improving the data processing performance, and the same computing power can process more data. The operation and maintenance cost of the data processing system based on the technology is reduced.

The same functionality can be accomplished if the extensions are also based on SQL, including but not limited to modifying the grammar keys listed herein, changing the order and format of SQL language grammar clauses, etc., without making significant changes to the information structure. All falling within the scope of the invention.

Embodiment two:

a method of device data processing, in accordance with a first embodiment,

Further, the extension compiler generates data format conversion program code according to the mode attribute attached to the data columns which cannot satisfy the 1-mode.

Further, the data format conversion program code is invoked by an extended SQL executor and/or a data format decoding component and/or a data format encoding component.

Further, the data in the specific format includes, but is not limited to: xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr, etc.

The extended SQL compiler can support a wide variety of data formats and modes thereof by analyzing the schema information through different data format decoding and encoding components. In a particular application, the format may be xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr, or the like.

Further, the extension compiler may support one or more of formats including, but not limited to xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr.

Further, the syntax of extended SQL includes additional device types, device communication addresses, and device-specific options after creating the table statement to reference the additional devices to continuously generate data streams in a specific format.

The table here means a semi-structured data stream of a specific format that the device will continuously generate, and the continuous device data stream of the same format is treated as a table of SQL.

The syntax of extended SQL specifically includes appending device types, device communication addresses, and device-specific options to the data table when creating a new data table, in the format of: for < device type > at < device communication address > options < device specific options >; and attaching a data format and mode information to each extended data column, wherein the format is format < format name > < mode information >, and the mode information is structure information in the data column. The present invention can be achieved without being limited to this format, and other formats, as long as a data sequence which cannot satisfy the 1-format in the data of the device-specific format is treated as a string of an indefinite length or a sequence of a binary data format, and is treated as a structured data sequence which satisfies the 1-format. The above data formats are the same.

This is an extension of the SQL-2016 standard of the present invention, where the tables are actually interpreted as a specially formatted semi-structured data stream that the device will continuously produce, rather than referring to the tables of a conventional database.

A format clause is appended to the string or binary type column of the Create Table sentence in the format of format < format name such as xml > < schema information >. The pattern information of the different formats should conform to the relevant standards of the format. This extension clause is optional for SQL-2016 compatibility, and if it is not present, it is a generic column.

Further, data format and pattern information indicating structure information inside the data columns are attached to the columns of the character string or binary type of the created table sentence.

Further, the extension compiler maps the relevant data columns into JSON data structures using the schema information.

The schema information also specifies the structural information within those semi-structured data columns, on the basis of which the extended SQL compiler can map into a JSON structure. The invention expands the access grammar of SQL2016 to JSON format data at the same time, so that the user can access the contained data through the JSON format grammar of SQL2016 standard. Such as:

The SQL instruction filters the real-time traffic record in the xml format of the access control equipment, and codes information into a format of thorft to be transmitted to the alarm display equipment for display once people pass through 1:00 to 5:00 in the morning. The format clause part in the instruction is an extension of the present invention to indicate the mode information of the output data column.

Further, format extensions of data format and mode information are added to format clauses of functions of json_exists and/or json_value and/or json_query and/or json_table input JSON format to process data columns of multiple formats. The format is format < format name > < schema information >.

So as to be used not only for processing JSON formatted data, but also for processing semi-structured data columns of arbitrary format.

Further, format extensions of data format and mode information are added to format clauses of json_object and/or json_array and/or json_ OBJECTAGG and/or json_ ARRAYTAG output JSON format functions to output multiple format data columns.

Format extensions of data format and schema information are also added to format clauses of functions outputting JSON format such as JSON OBJECT/JSON ARRAY/JSON OBJECTAGG/JSON ARRAYTAG in the format of format < format name > < schema information >.

So as to be used not only for outputting data in JSON format, but also for outputting any semi-structured data columns.

The technical solutions in the second embodiment may be combined with each other, or may be combined with the technical solutions in the first embodiment separately or after combination.

The method for compiling SQL language to efficiently process data collected in real time can directly utilize compiled SQL sentences to process the data without storing and warehousing and collecting the data in a memory, and the processed data can be directly sent to a next data processing unit in the memory.

The widely used SQL is expanded to be used as a basic data processing language, and can be applied to conversion, filtering, cleaning, statistical analysis and linkage processing of any structured and semi-structured data format; further, the compiling technology is adopted, the data processing performance can be greatly improved, and more data can be processed by the same computing capacity.

Embodiment III:

In the data processing method of the Internet of things, as shown in fig. 1, an extended SQL compiler receives an extended SQL instruction, extracts mode information of a data column according to a table name and a format clause in the extended SQL instruction, transmits the mode information to a decoding component and/or an encoding component, and generates data processing program codes required by an executor according to the mode information.

And the extended SQL compiler receives an extended SQL instruction which is input by a user according to the extended SQL grammar and is processed by data, and the extended SQL instruction is compiled and then is handed to a compiled extended SQL executor and a data format decoding component or a data format encoder component to process the data.

As shown in fig. 2, the extended SQL compiler receives the extended SQL instruction and compiles according to the following procedure:

(1) Analyzing the extended SQL instruction, and extracting corresponding table names, namely names of data streams of input equipment or output equipment, and format extension clauses from the extended SQL instruction;

(2) Extracting mode information of corresponding data columns according to table names and/or format extension clauses;

(3) Searching a corresponding data format decoding component or encoding component according to the mode information;

(4) Passing the mode information to a data format decoding or encoding component;

(5) Program code required for expanding the SQL executor is generated according to the mode information and the rest part (standard SQL part except the mode information) of the SQL instruction.

The compiling and code generating process is only executed once after the user inputs the SQL command, and the generated high-efficiency data processing code is repeatedly invoked by the extended SQL executor, the data format decoding component and the data format encoding component when processing the input data. Further, the compiling technology is adopted, the data processing performance can be greatly improved, and more data can be processed by the same computing capacity.

And using the extended SQL grammar to input an extended SQL instruction of data processing, treating a data column which cannot meet a 1-format in data in a specific format of equipment as a column of an indefinite length character string or binary data, and treating the data column as a structured data column which meets the 1-format.

Further, the system also comprises a component manager, which records which communication protocol type and/or which data format is supported by each data format decoding component and data format encoding component, and matches the corresponding decoding component or encoding component according to the communication protocol type and/or data format information.

Furthermore, the decoding component converts the data in the specific format of the equipment into unified structured data meeting the 1-norm according to the mode information of the input data transmitted by the expansion compiler, and transmits the structured data to the compiled executor for processing.

Further, the compiled executor processes the structured data meeting the 1-scale according to the data processing program code generated by the extended SQL compiler, and sends the processed result data to the encoding component for processing.

Furthermore, the coding component codes unified structured data meeting 1-range into data in a specific format of the device according to the mode information of the input data transmitted by the expansion compiler, and the data is transmitted to the device data output component for processing.

The technical scheme of the invention comprises one or a combination of a plurality of the following components, as shown in figure 1: the device comprises a device data access component, a data format decoding component, an extended SQL compiler, a compiled extended SQL executor, a data format encoding component, a device data output component and a component manager.

The user inputs data conversion, filtering, cleaning and statistical analysis extended SQL instructions according to the extended SQL grammar, and the extended SQL instructions are compiled by an extended SQL compiler and then are used by a compiled extended SQL executor and various data format decoding or encoder components.

All components, i.e. the individual data format decoding components and/or data format encoding components, register with the component manager which communication protocol or which data format they support, so that the system can only find the corresponding processing component from the protocol type or data format information.

There are a number of device data access components, each of which is responsible for a different device communication protocol. The device data access component parses the communication protocol and then takes the data in a specific format (typically semi-structured data) to the input device for processing by the data format decoding component.

There are a variety of data format decoding components, each of which is responsible for decoding a different data format. The data format decoding component decodes the specific format of the device according to the requirement of the extended SQL input by the user, converts the specific format into unified structured data meeting the 1-norm, and transmits the structured data to a compiled extended SQL executor for processing.

The compiled extended SQL executor converts, filters, cleans and statistically analyzes the structured data satisfying the 1-format according to the semantics of SQL, and delivers the processed result data to various data format encoding components for processing.

There are a variety of data format encoding components, each of which is responsible for encoding a different data format. The data format encoding component encodes unified structured data meeting 1-format into data (generally semi-structured data) in a device specific format according to the requirement of extended SQL input by a user, and the data is transmitted to the device data output component for processing.

There are a variety of device data output components, each of which is responsible for a different device communication protocol. The device data output component transmits data in a device specific format to the output device in accordance with a specified communication protocol.

After the output device takes the data, it may be stored or may perform some action (e.g. "open door", "play a piece of speech", etc.) according to the requirements of the communication protocol.

As shown in fig. 3, after input data is generated, the extended SQL executor, the data format decoding component and the data format encoding component execute according to the following procedures:

(1) The data format decoder receives a piece of data record from the input component of the device and loops each record;

(2) The data format decoder decodes the data according to the mode information of the input data transmitted by the extended SQL compiler to generate structured data which is convenient for the extended SQL executor to process and meets the 1-model and has a unified format;

(3) The extended SQL executor filters, cleans, performs statistical analysis and the like on the data generated before according to SQL processing semantics to generate a plurality of structured data meeting a 1-norm;

(4) The data format encoder circulates a plurality of structured data records meeting the 1-range generated by the extended SQL executor;

(5) The data format encoder encodes each generated structured record according to the mode information of the output data transmitted by the extended SQL compiler and then transmits the encoded structured record to the data output component;

(6) If there is an output data transfer (4);

(7) If there is more input data (1).

By using the technical scheme of the invention, the structured and semi-structured data in any format of any equipment can be processed, and the invention has great expansion capability. The technical scheme of the invention greatly reduces the skill requirement of users and reduces the operation and maintenance cost of the data processing system based on the technology. The extended SQL instructions can be compiled into machine instructions which are executed efficiently, so that the data processing performance is greatly improved. The data processing system based on the technology can process more data under the same hardware capacity, and improves the cost performance. The technical scheme of the invention increases the stability of the system.

Embodiment four:

An internet of things data processing device, as shown in fig. 1, corresponds to the third embodiment one by one, and is not described herein. Comprising the following steps: and the extended SQL compiler is used for receiving the extended SQL instruction, extracting the mode information of the data column according to the table name and the format clause in the extended SQL instruction, transmitting the mode information to the decoding component or the encoding component, and generating the data processing program code required by the executor according to the mode information.

The system also comprises a component manager, a data format decoding component and a data format encoding component, wherein the component manager is used for recording which communication protocol type and/or which data format is supported by each data format decoding component and data format encoding component, and matching the corresponding decoding component or encoding component according to the communication protocol type and/or data format information.

The decoding component is used for converting the data in the specific format of the equipment into unified structured data meeting the 1-norm according to the mode information of the input data transmitted by the expansion compiler and transmitting the structured data to the compiled executor for processing.

The compiled executor is used for processing the structured data meeting the 1-range according to the data processing program code generated by the extended SQL compiler, and processing the processed result data by the encoding component.

The coding component is used for coding unified structured data meeting 1-range into data with a specific format of equipment according to the mode information of the input data transmitted by the expansion compiler, and the data is transmitted to the equipment data output component for processing.

Fifth embodiment:

an internet of things data processing system, which comprises the method of the third embodiment or the internet of things data processing device of the fourth embodiment.

The specific description is shown in the third and fourth embodiments.

Example six:

A computer program product for data processing of the internet of things, comprising program code means adapted to execute the SQL language extension method for device data processing according to any one of the first, second and third embodiments when running on a computer.

Embodiment seven:

A computer readable storage medium storing any one or more computer programs according to the claims and according to any one of the first, second and third embodiments, the computer program being executable by one or more processors to implement a method of processing a data string that cannot satisfy a 1-format among data in a device-specific format as a string of an indefinite length or a string of a binary data format, as a structured data string that satisfies a 1-format.

With a great deal of applications of the internet of things, various data processing systems related to the internet of things appear, including but not limited to a linkage alarm system (continuously collecting data of devices, sending a command such as "ringing alarm" to other devices when judging that the data of some devices are abnormal), a data storage system (filtering and converting the data generated by the devices according to a certain rule and storing the data), a data access conversion cleaning system (collecting the data of the devices and carrying out format conversion on the data and sending the data to other systems after finishing the data of the devices), and a data statistical analysis display system (analyzing the data of the devices to obtain various statistical data and displaying the data of the devices). The common characteristic of these systems is the need to collect, format convert, filter, statistically analyze, link, send, display or store the data, wherein the key is format convert, filter, statistically analyze, link.

The invention aims at the problems of insufficient expansion capability, low performance, higher requirements on users, unstable system and difficult interactive auxiliary design in the data processing process in the prior art. The semi-structured data processing is divided into five parts, namely data input, format decoding, structured data processing, format encoding and data output, and unified interface standards are formulated for the data input, the data output and the format decoding encoding so as to be connected into equipment of various communication protocols in a componentized mode, and the semi-structured data and the structured data in various formats are mutually converted; the SQL language standard is extended, and some grammars are added to extend the format of the columns and to describe the pattern information of the columns in order to process (including converting, filtering, cleaning, statistical analysis) structured and semi-structured data in any format using a unified standardized way.

The SQL language is compiled to efficiently process the data collected in real time, and the data can be directly processed by using the compiled SQL sentence after being collected in the memory without being stored and put in storage, and the processed data can be directly sent to the next data processing unit in the memory.

Expanding widely used SQL as a basic data processing language, and applying the SQL to conversion, filtering, cleaning, statistical analysis and linkage processing of any structured and semi-structured data format; further, the compiling technology is adopted, the data processing performance can be greatly improved, and more data can be processed by the same computing capacity.

The invention further expands the JSON data processing part of the international standard SQL-2016 language to support data processing of any format to support a plurality of structured and semi-structured data, and compiles the expanded SQL instruction into a machine language to improve the performance of data processing.

The possible beneficial effects of the embodiment of the application include but are not limited to:

Based on the present invention, data engineers can manipulate structured and semi-structured data in any format from any device through their essentially all SQL data processing language, with tremendous expansion capabilities.

The skill requirements for the user are greatly reduced. The SQL instructions for completing the functions of data format conversion, filtering, cleaning, statistical analysis, alarming and the like can be written easily only by knowing the extended optimized compiled SQL of different data formats without writing complex script program codes, and the work is simple and convenient. This tends to reduce the operational maintenance costs of data processing systems based on this technology.

These SQL instructions are compiled into machine instructions that execute efficiently, greatly improving the performance of data processing. The data processing system based on the technology can process more data under the same hardware capacity, and improves the cost performance.

The SQL language is a language which is hardly in error, and a data engineer is difficult to write out processing instructions which cause the error of the system, so that the stability of the system is improved.

Because the SQL language has been widely used for decades, numerous interactive-assisted SQL authoring tools have been available to assist individuals with little programming in completing data processing. This further reduces the demands on the user and increases the range of applications of the data processing system based on this technology.

A flowchart is used in the present application to describe the operations performed by a system according to embodiments of the present application. It should be appreciated that the preceding or following operations are not necessarily performed in order precisely. Rather, the steps may be processed in reverse order or simultaneously. Also, other operations may be added to or removed from these processes.

While the basic concepts have been described above, it will be apparent to those skilled in the art that the foregoing detailed disclosure is by way of example only and is not intended to be limiting. Although not explicitly described herein, various modifications, improvements and adaptations of the application may occur to one skilled in the art. Such modifications, improvements, and modifications are intended to be suggested within the present disclosure, and therefore, such modifications, improvements, and adaptations are intended to be within the spirit and scope of the exemplary embodiments of the present disclosure.

Furthermore, those skilled in the art will appreciate that the various aspects of the application are illustrated and described in the context of a number of patentable categories or circumstances, including any novel and useful procedures, machines, products, or materials, or any novel and useful modifications thereof. Accordingly, aspects of the application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "device," unit, "" component, "or" system. Furthermore, aspects of the application may take the form of a computer product, comprising computer-readable program code, embodied in one or more computer-readable media. "and/or" includes both "and" or "both.

The computer storage medium may contain a propagated data signal with the computer program code embodied therein, for example, on a baseband or as part of a carrier wave. The propagated signal may take on a variety of forms, including electro-magnetic, optical, etc., or any suitable combination thereof. A computer storage medium may be any computer readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer storage medium may be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or a combination of any of the foregoing.

The computer program code may execute entirely on the user's computer or as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any form of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or the use of services such as software as a service (SaaS) in a cloud computing environment.

Furthermore, the order in which the elements and sequences are presented, the use of numerical letters, or other designations are used in the application is not intended to limit the sequence of the processes and methods unless specifically recited in the claims. While certain presently useful inventive embodiments have been discussed in the foregoing disclosure, by way of example, it is to be understood that such details are merely illustrative and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements included within the spirit and scope of the embodiments of the application. For example, while the system components described above may be implemented by hardware devices, they may also be implemented solely by software solutions, such as installing the described system on an existing server or mobile device.

In some embodiments, numbers describing the components, number of attributes are used, it being understood that such numbers being used in the description of embodiments are modified in some examples by the modifier "about," approximately, "or" substantially. Unless otherwise indicated, "about," "approximately," or "substantially" indicate that the number allows for a 20% variation. Accordingly, in some embodiments, numerical parameters set forth in the specification and claims are approximations that may vary depending upon the desired properties sought to be obtained by the individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and employ a method for preserving the general number of digits. Although the numerical ranges and parameters set forth herein are approximations in some embodiments for use in determining the breadth of the range, in particular embodiments, the numerical values set forth herein are as precisely as possible.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present application. Other variations are also possible within the scope of the application. Thus, by way of example, and not limitation, alternative configurations of embodiments of the application may be considered in keeping with the teachings of the application. Accordingly, the embodiments of the present application are not limited to the embodiments explicitly described and depicted herein.

Claims

1. A method for processing data of equipment is characterized in that a data column which cannot meet 1-format in data of a specific format of the equipment is regarded as a string with an indefinite length or a column with a binary data format, and the data column is processed as a structured data column which meets 1-format; a mode attribute is added to the data columns which cannot meet the 1-mode, so that an expansion compiler can understand the specific formats of the data columns which cannot meet the 1-mode and process the data;

the expansion compiler generates data format conversion program codes according to the mode attribute attached to the data columns which cannot meet the 1-mode; the data format conversion program codes are called by an extended SQL executor and/or a data format decoding component and/or a data format encoding component; extending the syntax of SQL includes appending device types, device communication addresses, and device-specific options after creating the table statement, the device continually generating a data stream of a specific format; data format and pattern information indicating structure information inside the data columns are appended after the strings or binary type columns of the table statement are created.

2. The method of claim 1, wherein the data in the particular format comprises: xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr format.

3. The method of claim 1, wherein the extension compiler supports one or more of xml, CSV, JSON, protobuf, thrift, avro, BSON, messagepack, rpcxdr formats.

4. The method of claim 1, wherein the extended compiler maps data columns into JSON data structures using the schema information.

5. A method according to claim 2, characterized in that format extensions of data format and schema information are added to the format clauses of the function of json_exists and/or json_value and/or json_query and/or json_table input JSON format for processing data columns of multiple formats.

6. A method according to claim 2, characterized in that a format extension of data format and mode information is added to a format clause of a function outputting JSON format in json_object and/or json_array and/or json_ OBJECTAGG and/or json_ ARRAYTAG for outputting multiple format data columns.

7. The data processing method of the Internet of things is characterized in that an extended SQL compiler receives an extended SQL instruction, extracts mode information of a data column according to a table name and a format clause in the extended SQL instruction, transmits the mode information to a decoding component and/or an encoding component, and generates data processing program codes required by an executor according to the mode information; the extended SQL compiler generates data format conversion program codes according to the mode attribute attached to the data columns which cannot meet the 1-format, and attaches data format and mode information to each extended data column, wherein the mode information is structural information in the data columns, extended SQL instructions for data processing are input by using extended SQL grammar, and the data columns which cannot meet the 1-format in the data of the device specific format are regarded as the columns of the indefinite length character strings or binary data and are processed as the structured data columns which meet the 1-format.

8. The method of claim 7, further comprising a component manager that records which communication protocol type and/or which data format is supported by each of the data format decoding component and the data format encoding component, and matches the corresponding decoding component or encoding component based on the communication protocol type and/or data format information.

9. The method of claim 7, wherein the decoding component converts the device-specific format data into unified structured data satisfying 1-paradigm according to the schema information of the input data from the extended SQL compiler, and passes the structured data to the compiled executor for processing.

10. A method according to claim 7 or 9, wherein the compiled executor processes the structured data satisfying the 1 st-th-order in accordance with the data processing program code generated by the extended SQL compiler, and passes the processed result data to the encoding component for processing.

11. The method of claim 7, wherein the encoding component encodes the unified structured data satisfying 1-format into data of a device-specific format according to the schema information of the input data transmitted from the extended SQL compiler, and transmits the data to the device data output component for processing.

12. The utility model provides an thing networking data processing device which characterized in that includes: the extended SQL compiler is used for receiving the extended SQL instruction, extracting the mode information of the data column according to the table name and the format clause in the extended SQL instruction, transmitting the mode information to the decoding component or the encoding component, and generating the data processing program code required by the executor according to the mode information; the extended SQL compiler generates data format conversion program codes according to the mode attribute attached to the data columns which cannot meet the 1-format, and attaches data format and mode information to each extended data column, wherein the mode information is structural information in the data columns, extended SQL instructions for data processing are input by using extended SQL grammar, and the data columns which cannot meet the 1-format in the data of the device specific format are regarded as the columns of the indefinite length character strings or binary data and are processed as the structured data columns which meet the 1-format.

13. The apparatus of claim 12, further comprising a component manager for recording which communication protocol type and/or which data format is supported by each of the data format decoding component and the data format encoding component, and matching the corresponding decoding component or encoding component according to the communication protocol type and/or data format information.

14. The apparatus of claim 12, wherein the decoding component is configured to convert the device-specific format data into unified structured data satisfying 1-format according to the schema information of the input data transmitted from the extended SQL compiler, and to transmit the structured data to the compiled executor for processing.

15. The apparatus according to claim 12 or 14, wherein the compiled executor is configured to process the structured data satisfying the 1-way expression according to the data processing program code generated by the extended SQL compiler, and to pass the processed result data to the encoding component for processing.

16. The apparatus of claim 12, wherein the encoding component is configured to encode the unified structured data satisfying the 1-format into data in a device-specific format according to the schema information of the input data transmitted from the extended SQL compiler, and to process the data in the device-specific format by the device data output component.

17. An internet of things data processing system, comprising: the internet of things data processing apparatus of any of claims 12-16.

18. A computer program product for internet of things data processing, comprising program code means adapted to be on a computer for performing the method of device data processing according to any of claims 1-6 at run-time.

19. A computer readable storage medium storing a computer program executable by one or more processors to implement a method of treating a data column in a device-specific format that cannot satisfy a 1-gram as a column in an indefinite length string or binary data format in any of claims 1-11 as a structured data column that satisfies a 1-gram.