WO2022134878A1

WO2022134878A1 - Data processing method and apparatus, data querying method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022134878A1
Application number: PCT/CN2021/128503
Authority: WO
Inventors: 吉斌
Original assignee: 中兴通讯股份有限公司
Priority date: 2020-12-21
Filing date: 2021-11-03
Publication date: 2022-06-30
Also published as: CN114647660A

Abstract

The embodiments of the present application relate to the technical field of data processing. Provided are a data processing method and apparatus, a data querying method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring data from a data source according to preconfigured data source information; acquiring descriptive information of the data according to a format definition in the data source information; according to the descriptive information, storing the data in a preset storage space as uniform structured data; and after a query instruction is received, performing data querying in the storage space.

Description

Data processing, query method, device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number "202011519522.4" and the filing date is December 21, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference. Application.

technical field

The embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing and query method, apparatus, electronic device, and storage medium.

Background technique

With the development of the Internet, various software and systems used in daily life and commercial production will generate and use a large amount of data, among which, the data generated and used by different software and systems have different data formats.

In related data processing methods, different data sources store data in different data formats. When querying two or more data sources, only different query languages can be used in each data source. , query each data source one by one.

Therefore, there are the following problems in the related data processing methods: two or more data sources cannot be correlated and queried, and the query efficiency is low.

SUMMARY OF THE INVENTION

An embodiment of the present application provides a data processing method, including: obtaining data from a data source according to preconfigured data source information; obtaining description information of the data according to a format definition in the data source information; The unified structured data is stored in the preset storage space; after receiving the query command, the data query is performed in the storage space.

The embodiment of the present application also provides a data query method, which includes: after receiving a semi-structured query instruction, converting the query instruction into a structured query instruction; The data query is performed in the storage space of structured data; the structured query result obtained by the data query is converted into a semi-structured query result.

An embodiment of the present application further provides a data processing device, including: an acquisition module, configured to acquire data from a data source according to preconfigured data source information; a description module, used to acquire data according to a format definition in the data source information The description information of the data; the conversion module is used to store the data in the preset storage space as unified structured data according to the description information; the query module is used to query the data in the storage space after receiving the query instruction.

The embodiment of the present application also provides a data query device, including: a query conversion module, which is used to convert the query instruction into a structured query instruction after receiving a semi-structured query instruction; The query instruction is used to perform data query in the preset storage space for storing unified structured data; the result conversion module is used to convert the structured query result obtained by the data query into a semi-structured query result.

An embodiment of the present application further provides an electronic device, including: at least one processor; a memory connected in communication with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, So that at least one processor can execute the above-mentioned data processing method, or the above-mentioned data query method.

Embodiments of the present application also provide a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the above-mentioned data processing method or the above-mentioned data query method is implemented.

Description of drawings

1 is a flowchart of a data processing method provided according to a first embodiment of the present application;

2 is a schematic diagram of an overall architecture provided according to the first embodiment of the present application;

3 is a flowchart of a data processing method provided according to a second embodiment of the present application;

4 is a flowchart of a data query method provided according to a third embodiment of the present application;

5 is a schematic diagram of a data processing apparatus provided according to a fourth embodiment of the present application;

6 is a schematic diagram of a data query device provided according to a fifth embodiment of the present application;

FIG. 7 is a schematic diagram of an electronic device provided according to a sixth embodiment of the present application.

Detailed ways

The main purpose of the embodiments of the present application is to provide a data processing and query method, apparatus, electronic device and storage medium, so as to improve the efficiency of data query.

In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that, in each embodiment of the present application, many technical details are provided for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized. The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.

The first embodiment of the present application relates to a data processing method, and the specific process is shown in Figure 1:

Step 101: Acquire data from a data source according to preconfigured data source information;

Step 102, according to the format definition in the data source information, obtain the description information of the data;

Step 103, according to the description information, store the data in a preset storage space with unified structured data;

Step 104, after receiving the query instruction, perform data query in the storage space.

The data processing method of this embodiment is applied to a data processing apparatus, for example, an electronic device capable of data storage and data query. When using electronic equipment such as servers to perform data analysis on big data, due to the wide range of data sources, different data formats, and huge data volume of big data, it is impossible to query at least two data sources with different data formats at the same time. For complex queries such as these, you can only query data from each data source separately, and obtaining the data required for analysis requires high manpower and material resources. Based on the data processing method of the present application, the data in the data of each data source can be stored in the preset storage space as unified structured data, so that the user can perform complex query on the unified structured data, and then query the obtained data. data for data analysis. The electronic device of the present application includes three main parts: a data format conversion device, a structured data storage engine, and a query execution engine. The overall architecture of the data processing method of the present application is shown in FIG. 2 .

The implementation details of the data processing method in this embodiment will be specifically described below, and the following contents are only provided for the convenience of understanding, and are not necessary for implementing this solution.

In step 101, the electronic device acquires data from a data source according to preconfigured data source information. Among them, the data source can be the file system, Representational State Transfer REST (Representational State Transfer, referred to as "REST") interface, WebService interface and other channels for storing or obtaining data, and the data obtained by the electronic device from the file system and the REST interface is The data file stored in the file system or pointed to by the REST interface. The data file stores JavaScript Object Notation JSON (JavaScript Object Notation, referred to as "JSON"), Extensible Markup Language XML (eXtensible Markup Language, referred to as "XML"), comma-separated Semi-structured data in the format of CSV (Comma-Separated Values, referred to as "CSV"). The preconfigured data source information may be in the form of a configuration instruction file, written or uploaded by the user, and stored in the electronic device. Specifically, the electronic device can sequentially acquire each data object in the data source, or can acquire all data objects in one data source at a time, and the electronic device can select different ways to acquire data objects according to the data format of the data source. The electronic device can obtain data from the data source by invoking the interface method and receiving the data returned by the interface.

Specifically, the electronic device can generate a configuration instruction file in advance. There is one or more configuration items in the configuration instruction file. The user fills in the configuration items. Each configuration item is the data source information. Configure the instruction file to obtain data source information.

In one example, the data source information may include the path of the data source, for example, the storage path of the file system in the data source electronic device, the URL of the Uniform Resource Locator (Uniform Resource Locator, referred to as "URL") of the REST interface, etc. . The electronic device obtains data from each data source by reading the path of each data source in the data source information. Wherein, the data source electronic device may be the same device as the electronic device performing data query in this application, or may be another device. After reading the data source information, the electronic device can immediately obtain data from the data source according to the path of the data source.

In one example, the data source information includes a parameter name and a parameter value type, and the electronic device obtains data matching the parameter name and parameter value type from the data source according to the data source information. Specifically, the configuration instruction file may contain parameter items, and the user sets the parameter name and parameter value type for the parameter item, and the electronic device uses this parameter to filter the data in the data source. For example, if the parameter name in the data source information is name and the parameter value type is string, the electronic device obtains the part of the data structure description name and the data value type string from the data source. If a data has three parts, and the structure description name and data value type of the three parts are: name, character string; gender, character string; birthday, date, the electronic device only obtains the data of the name part of the data.

In this embodiment, by acquiring data from the data source that conforms to the preset parameter name and the preset parameter value type, when acquiring data from the data source, the data is filtered, only the required data is acquired, and the storage space is reduced. The amount of data is increased, so as to speed up the operation speed of the device during data query and improve the efficiency of data query.

In one example, the configuration instruction file may further include a time range item, the user configures the time range item, and the electronic device uses the time range item to filter the data in the data source. The time range item can contain two parameters: start time and end time, and time refers to the update time of the data. For example, if the user sets the start time to 02/11/2020 and the end time to 12/11/2020, the electronic device will obtain the update time from the data source according to the time range from 02/11/2020 to 12/11/2020 data in it.

In one example, the configuration instruction file may contain parameter items and time range items, and the electronic device obtains the preset parameters within the time range from the data source according to the parameter name, parameter value type and time range in the data source information. Data of name and preset parameter value type to further filter the data in the data source.

In step 102, the electronic device acquires description information of the data according to the format definition in the data source information, where the description information is a structure description of the data object acquired by the electronic device. If the electronic device obtains a single data object one by one, it needs to obtain the description information of the data object once every time it obtains a data object. If the electronic device obtains all the data objects in the data source at one time, it only needs to obtain the data object once. The description information of all data objects is enough.

In step 103, the electronic device stores the data in a preset storage space as unified structured data according to the description information. Among them, the unified structured data can be directly stored in the data file, and can also be stored through the database. Specifically, the electronic device converts the data object into unified structured data according to the acquired structure description of the data object, and stores the unified structured data as a result set through a structured storage engine.

In one example, the electronic device can generate an operation statement for operating the unified structured data, for example, an operation statement for inserting a row of data into a data file, an insert statement for inserting data into a data table in a database, etc. Unified structured data store.

In one example, the storage space may be a memory space, and the electronic device converts the data into unified structured data and stores it in the memory space.

In one example, the electronic device also stores and persists the unified structured data as a result set, such as storing the structured data in the memory space in a data file or a database. In this embodiment, by persistently storing the structured data, the structured data can be further processed after the query.

In step 104, after receiving the query instruction, the electronic device performs data query in the storage space. Specifically, after receiving the query instruction input by the user, the electronic device sends the query instruction to the query engine for execution, and the query engine performs data query on the structured data in the storage space to obtain the query result. The query statement may include query parameters and query parameter values, and the electronic device may mark or store the query results as a data set.

In one example, the electronic device can receive complex query statements input by the user, and perform complex queries on two-dimensional table data. For example, the query statement can be any query statement supported by standard SQL statements, and can implement condition filtering, group statistics, and aggregate statistics. (For example, the COUNT function counts the number of rows, the SUM function obtains the total value of a single column), window statistics, etc. The query statement may also include parameters and parameter values, and the electronic device uses the parameter values to replace the parameters in the statement, and the query execution engine executes the query to query the data in the result set.

In one example, the storage space is a memory space, and after receiving the query instruction, the electronic device performs data query on the result set in the memory space. Since the data storage of the result set and the execution process of the query are all in the memory, the data query efficiency is very high.

In one example, after a data query is performed in the storage space of the electronic device, the query result is output in JSON format, for example, in a JSON array or a single number.

In one example, the electronic device also encapsulates the query result of the query, that is, a data set, as a message, and sends it to a message queue, or performs visual processing on the query result, such as displaying it as report data on a line graph or a bar graph of the report, etc. .

In one example, after receiving the data source information, the electronic device may acquire data from the data source at a preset period according to the path of the data source. For example, in the data source information, the data acquisition period can be configured to be 1 hour. If the electronic device acquires data from the data source x at 10:00a.m, then at 11:00a.m, the electronic device will retrieve data from the data source x again. Get data from data source x. In this embodiment, by acquiring data from the data source at a preset period according to the path of the data source, updated data can be acquired from the data source, thereby achieving synchronization between the acquired data and the data in the data source.

In one example, the electronic device may also acquire data from the data source by receiving data sent by the data source. Specifically, after receiving the data sent by the data source, the electronic device can determine the data source for sending the data according to the path in the data source information. When the data source sends data to the electronic device for the first time, it sends all the data in the data source, and the data sent to the electronic device subsequently may be all data or updated data. The electronic device receives the data sent by the data source, and updates the received data into the storage space. Among them, when the data source sends the updated data, it can directly send the updated data to the electronic device. After the electronic device converts the updated data into unified structured data, it can be compared with the data sent before the data source already stored in the storage space. Carry out comparison, confirm the update type of the updated data according to the comparison result, such as new data, modified data, etc., and then perform corresponding operations on the updated data according to the update type, for example, add the new data to the storage space, The modified data corresponds to the original data before the update in the storage space.

In one example, when the data source sends update data to the electronic device, it can also send update information to the electronic device, where the update information may include the update type of the update data. It also contains information about the data before modification corresponding to this modified data.

In this embodiment, the data is obtained from the data source according to the pre-configured data source information, the data description information is obtained according to the format definition, and the data is stored in the preset storage space as unified structured data. After the query command, the data query is performed in the storage space. Since the description information is obtained according to the format definition of the data source information, the format of the data obtained from the data source can be changed according to the description information and converted into structured data with a unified format. Therefore, it is possible to perform an associated query on structured data with a unified format, use a unified query language, and query data in multiple data sources at the same time, thereby improving query efficiency.

The steps of the above various methods are divided only for the purpose of describing clearly. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this patent. ;Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

The second embodiment of the present application relates to a data processing method. The second embodiment is substantially the same as the first embodiment, and the main difference is that: in the second embodiment of the present application, the data is specifically stored in the form of two-dimensional table data.

This embodiment relates to a data processing method. The specific process is shown in Figure 3:

Step 301: Acquire data from a data source according to preconfigured data source information;

Step 302, according to the format definition in the data source information, obtain the description information of the data; wherein, the description information includes: the key name and the key value type of the data;

Step 303, take the key name as the column name in the two-dimensional table, take the key value type as the data type of the column corresponding to the key name, and fill in the key value of the data in the column corresponding to the key name; wherein, the two-dimensional table Stored in the preset storage space;

Step 304, after receiving the query instruction, perform data query in the storage space.

Wherein, step 301 and step 304 are substantially the same as step 101 and step 104 in the first embodiment, and will not be repeated here.

In step 302, the data in the data source is semi-structured data stored in the form of key-value pairs. The description information of the data can be the key name and key value type of the data. When the electronic device acquires data, it also acquires the key name and key value type of the data.

In one example, the data source information includes a parameter name and a parameter value type, and the electronic device obtains data matching the parameter name and parameter value type from the data source according to the data source information. For example, if the parameter name in the data source information is name and the parameter value type is string, the electronic device only obtains the part of all data in the data source whose key name is name and whose key value type is string. Key-value pair, in which the key name of one key-value pair is name, the key-value type is string, and the key-value is Zhang San, and the key-value pair is gender, the key-value type is string, and the key-value is male. , the key name of a key-value pair is birthday, the key-value type is date, and the key-value is 02/11/1991, then the data obtained by the electronic device from the data source is a key-value pair, and the key-value pair’s key name is The name and key value type are strings, and the key value is Zhang San.

In step 303, the unified structured data is two-dimensional table data, the electronic device uses the key name as the column name in the two-dimensional table, the key value type is used as the data type of the column corresponding to the key name, and the key value of the data is used as the data type of the column corresponding to the key name. Fill in the column corresponding to the key name. Specifically, after the electronic device obtains the data, it determines whether there is a column with the same name as the key in the two-dimensional table. If there is a column with the same name as the key, the key value of the data is filled in. If there is no column with the same name as the key. , then create a column with the same name as the key in the two-dimensional table, and then fill in the key value of the data.

In one example, the two-dimensional table in the storage space corresponds to the data source one-to-one. After the electronic device obtains data from the data source, it also determines whether there is a two-dimensional table corresponding to the data source in the storage space. If there is a two-dimensional table corresponding to the data source, a two-dimensional table is created in the storage space.

In this embodiment, after acquiring the data from the data source, a two-dimensional table corresponding to the data source is created in the storage space, which facilitates the query and management of the unified structured data in the storage space, and can simplify the structured data. Data management, and improve query efficiency.

In one example, the electronic device uses the data processing method of the present application to filter the real-time data in JSON format, and then sends it to a message queue as a message body for use by other application systems, and is implemented according to the following steps: 1. The configured data source information, where the path of the data source is the URL of the REST interface: /realtime-data, the parameter name of the parameter of this interface is moOid, which is used to filter the identity of the monitoring object oid (object identity document, referred to as "oid" for short). ”), the data in the data source is in JSON format, and the description information obtained by the electronic device is the key name and key value type of the key-value pair with the key names moOid (oid of monitoring object) and smpOid (oid of standard measurement point). 2. The electronic device creates a database table in the memory database, and calls the interface /realtime-data of the real-time data JSON data source to obtain real-time data in JSON format. 3. The electronic device converts the real-time data in JSON format into an insert statement in SQL language, inserts the real-time data into the memory database table, generates a two-dimensional data table corresponding to this data source, and marks the data in the table as a result set. 4. The electronic device obtains the SQL statement input by the user to query the result set, which includes query parameters and query parameter values corresponding to moOid and smpOid. The electronic device executes the SQL statement to query moOid and smpOid, and the filtering does not meet the query requirements data, and output the resulting dataset in JSON format. 5. The electronic device encapsulates the result data set as a message and sends it to the message queue.

In one example, the electronic device uses the data processing method of the present application to collect statistics on historical data in JSON format and display it in a report, and implement the following steps: 1. Obtain pre-configured data source information, wherein the path of the data source is It is the URL of the REST interface: /history-data. This interface has three preset parameters: oid, start time, and end time of the monitoring object. The data in the data source is in JSON format, and the description information obtained by the electronic device is the key name moOid (oid of monitoring object), smpOid (oid of standard measuring point), timeRange (time range), curValue (current value of time range) key-value pair. 2. The electronic device creates a database table in the memory database, and calls the interface /history-data of the historical data JSON data source to obtain historical data in JSON format. 3. The electronic device converts the historical data in JSON format into an insert statement in SQL language, inserts the historical data into the memory database table, generates a two-dimensional data table corresponding to this data source, and marks the data in the table as a result set. 4. The electronic device obtains the SQL statement input by the user to query the result set, which includes query parameters and query parameter values corresponding to timeRange, moOid, and smpOid. The electronic device executes the SQL statement, queries the data in the result set according to the query parameters and the query parameter values, filters the data that does not meet the query requirements, and outputs the result data set in JSON format. 5. The electronic device displays the resulting dataset in the report.

In one example, the electronic device uses the data processing method of the present application to obtain the parent-child node relationship of the logical grouping in JSON format, and displays it in a tree control, and implements it according to the following steps: 1. Obtaining the pre-configured logical grouping parent-child node relationship JSON data source information, where the path of the data source is the URL of the REST interface: /logic-group-tree, the data in the data source is in JSON format, and the description information obtained by the electronic device is the key named oid (logical group oid) , name (logical group name), parentOid (logical group parent node oid) key-value pair. 2. The electronic device creates a database table in the memory database, and calls the interface /logic-group-tree of the JSON data source of the logically grouped parent and child nodes to obtain the logically grouped parent and child node data in JSON format. 3. The electronic device converts the logically grouped parent-child node data in JSON format into an insert statement in SQL language, inserts the logically grouped parent-child node data into the memory database table, generates a two-dimensional data table corresponding to this data source, and converts the data in the table. Data is marked as a result set. 4. The electronic device obtains the SQL statement input by the user to query the result set, which includes query parameters and query parameter values corresponding to parentOid. The electronic device executes the SQL statement, queries the data in the result set according to the query parameters and the query parameter values, filters the data that does not meet the query requirements, and outputs the result data set in JSON format. 5. The electronic device displays the result data set on the tree control.

In this embodiment, the key name is used as the column name in the two-dimensional table, and the key value type is used as the data type of the column corresponding to the key name, and the key value of the data is filled in the column corresponding to the key name. The dimension table data is unified structured data, which can realize the conversion of the data obtained from the data source into unified structured data; at the same time, by converting the data in the data source into two-dimensional table data, it can be realized based on the two-dimensional table structure. Various complex queries, so the data processing method of the present application can improve the query efficiency.

The third embodiment of the present application relates to a data query method, and the specific process is shown in FIG. 4 , including:

Step 401, after receiving the semi-structured query instruction, convert the query instruction into a structured query instruction;

Step 402, according to the structured query instruction, perform data query in a preset storage space for storing unified structured data;

Step 403: Convert the structured query result obtained by the data query into a semi-structured query result.

The data query method of this embodiment is applied to a data query device, for example, an electronic device that can perform data query. In particular, the data query apparatus may also be the query execution engine in the first embodiment. When an electronic device such as a server is used to perform data analysis on the big data, the data query method in this embodiment can be used to obtain the required data from the big data.

In step 401, after receiving the semi-structured query instruction, the electronic device converts the query instruction into a structured query instruction corresponding to unified structured data. The semi-structured query instruction is a query statement entered by the user and represented by a semi-structured data query format. A query statement for querying structured data.

Specifically, the electronic device can call the first conversion function that converts the semi-structured query instruction correspondingly into the structured query instruction, and fills the query information in the semi-structured query instruction into the pre-generated structured query statement template to obtain the structure query command. Wherein, the first conversion function can be generated according to the description information.

In step 402, the electronic device performs a data query in a preset storage space for storing unified structured data according to the structured query instruction, and obtains a query structure in a structured data format, that is, a structured query result.

In step 403, the electronic device converts the structured query result obtained by the data query into a semi-structured query result. Specifically, the electronic device can call a second conversion function that converts structured data into semi-structured data, and correspondingly fill in the data in the structured query result into the pre-generated semi-structured data template to obtain the semi-structured query result . Wherein, the second conversion function can be generated according to the description information.

In an example, the electronic device uses the data query method of this embodiment to perform data query, wherein the semi-structured query instruction is a query statement in JSON format, the structured query instruction is an SQL statement, and a storage space for storing unified structured data for memory space. After receiving the query statement in JSON format input by the user, the electronic device converts the query statement in JSON format into SQL statement, and uses the converted SQL statement to query the two-dimensional table format data in the memory in the memory space. After obtaining the data query result, the electronic device converts the query result in the two-dimensional table format into JSON format, and outputs it to the user.

In this embodiment, by converting the query instruction into a structured query instruction, according to the structured query instruction, data query is performed in a preset storage space for storing unified structured data, and the structured query obtained by the data query is performed. As a result, it can be converted into semi-structured query results, and in semi-structured application scenarios, the query language in semi-structured format can be used to perform related queries on data sources in various data formats, and a semi-structured directly applicable semi-structured query can be obtained. data, so as to improve the query efficiency and better meet the actual application needs.

Since the above-mentioned first and second embodiments correspond to this embodiment, this embodiment can be implemented in cooperation with the above-mentioned embodiments. The relevant technical details mentioned in the foregoing embodiment are still valid in this embodiment, and the technical effects that can be achieved in the foregoing embodiment can also be realized in this embodiment, and in order to reduce repetitions, details are not repeated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiments.

The fourth embodiment of the present application relates to a data processing apparatus, as shown in FIG. 5 , including:

an acquisition module 501, configured to acquire data from a data source according to preconfigured data source information;

A description module 502, configured to obtain description information of the data according to the format definition in the data source information;

The conversion module 503 is used to store the data in a preset storage space with unified structured data according to the description information;

The query module 504 is configured to perform data query in the storage space after receiving the query instruction.

In one example, the description information includes: the key name and key value type of the data; the unified structured data is two-dimensional table data; the conversion module 503 is specifically configured to use the key name as the column name in the two-dimensional table, and use the key name as the column name in the two-dimensional table. The value type is used as the data type of the column corresponding to the key name, and the key value of the data is filled in the column corresponding to the key name; wherein, the two-dimensional table is stored in the preset storage space.

In one example, the two-dimensional table is in one-to-one correspondence with the data source; the data query device further includes a creation module for obtaining data from the data source, if there is no two-dimensional table corresponding to the data source in the storage space, then Create a two-dimensional table in the storage space.

In one example, the storage space includes a memory space; the obtaining module 501 is specifically configured to obtain data from a data source that conforms to a preset parameter name and a preset parameter value type; the data source information includes the parameter name and parameter value type.

In one example, the data source information includes: the path of the data source; the obtaining module 501 is specifically configured to obtain data from the data source at a preset period according to the path of the data source.

In one example, the query module 504 is further configured to encapsulate the query result of the query into a message, send it to a message queue, or perform visual processing on the query result.

It is not difficult to find that this embodiment is a device embodiment corresponding to the first and second embodiments, and this embodiment can be implemented in cooperation with the first and second embodiments. The relevant technical details mentioned in the first and second embodiments are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the related technical details mentioned in this embodiment can also be applied to the first and second embodiments.

It is worth mentioning that all the modules involved in this embodiment are logical modules. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. A composite implementation of the unit. In addition, in order to highlight the innovative part of the present application, this embodiment does not introduce units that are not very closely related to solving the technical problem proposed by the present application, but this does not mean that there are no other units in this embodiment.

The fifth embodiment of the present application relates to a data query device, as shown in FIG. 6 , including:

The query conversion module 601 is configured to convert the query instruction into a structured query instruction after receiving the semi-structured query instruction;

The structure query module 602 is configured to perform data query in the preset storage space for storing unified structured data according to the structured query instruction;

The result conversion module 603 is configured to convert the structured query result obtained by the data query into a semi-structured query result.

It is not difficult to find that this embodiment is a device embodiment corresponding to the third embodiment, and this embodiment can be implemented in cooperation with the third embodiment. The related technical details mentioned in the third embodiment are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied in the third embodiment.

It is worth mentioning that all the modules involved in this embodiment are logical modules. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. A composite implementation of the unit. In addition, in order to highlight the innovative part of the present application, this embodiment does not introduce units that are not closely related to solving the technical problem raised by the present application, but this does not mean that there are no other units in this embodiment.

The sixth embodiment of the present application relates to an electronic device, as shown in FIG. 7 , comprising: at least one processor 701; The executed instruction, the instruction is executed by at least one processor 701 to execute the above-mentioned data processing method or the above-mentioned data query method.

The memory 702 and the processor 701 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 701 and various circuits of the memory 702 together. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The information processed by the processor 701 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the information and transmits the information to the processor 701 .

Processor 701 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory 702 may be used to store information used by the processor in performing operations.

The seventh embodiment of the present application relates to a computer-readable storage medium storing a computer program. The above method embodiments are implemented when the computer program is executed by the processor.

That is, those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, Read-Only Memory (ROM for short), Random Access Memory (RAM for short), magnetic disk or optical disk, etc. medium of program code.

Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for realizing the present application, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present application. scope.

Claims

A data processing method comprising:

Acquire data from the data source according to preconfigured data source information;

Obtain the description information of the data according to the format definition in the data source information;

According to the description information, store the data in a preset storage space as unified structured data;

After receiving the query instruction, perform data query in the storage space.
The data processing method according to claim 1, wherein the description information includes: a key name and a key value type of the data; the unified structured data is two-dimensional table data;

The storing of the data in the preset storage space as unified structured data according to the description information includes:

Use the key name as the column name in the two-dimensional table, use the key value type as the data type of the column corresponding to the key name, and fill in the key value of the data into the column corresponding to the key name wherein, the two-dimensional table is stored in the preset storage space.
The data processing method according to claim 2, wherein the two-dimensional table is in one-to-one correspondence with the data source; after acquiring the data from the data source, the method further comprises:

If the two-dimensional table corresponding to the data source does not exist in the storage space, the two-dimensional table is created in the storage space.
The data processing method according to any one of claims 1 to 3, wherein the storage space comprises a memory space;

The acquiring data from the data source includes:

Acquire data conforming to a preset parameter name and a preset parameter value type from the data source; the data source information includes the parameter name and the parameter value type.
The data processing method according to claim 4, wherein the data source information further includes a time range; and the acquiring data conforming to a preset parameter name and a preset parameter value type from the data source includes:

The data that conforms to the preset parameter name and the preset parameter value type within the time range is obtained from the data source.
The data processing method according to any one of claims 1 to 5, wherein the data source information comprises: a path of the data source;

The obtaining data from the data source according to the pre-configured data source information includes:

According to the path of the data source, data is obtained from the data source at a preset period.
The data processing method according to any one of claims 1 to 6, wherein after performing the data query in the storage space, the method further comprises:

The query result of the query is encapsulated into a message and sent to a message queue, or the query result is visualized.
A data query method, including:

After receiving the semi-structured query instruction, convert the query instruction into a structured query instruction;

According to the structured query instruction, perform data query in a preset storage space for storing unified structured data;

The structured query result obtained by the data query is converted into a semi-structured query result.
A data processing device, comprising:

an acquisition module, configured to acquire data from the data source according to preconfigured data source information;

a description module, configured to obtain description information of the data according to the format definition in the data source information;

a conversion module, configured to store the data in a preset storage space as unified structured data according to the description information;

The query module is configured to perform data query in the storage space after receiving the query instruction.
A data query device, comprising:

a query conversion module, configured to convert the query instruction into a structured query instruction after receiving the semi-structured query instruction;

a structural query module, configured to perform data query in a preset storage space for storing unified structured data according to the structured query instruction;

The result conversion module is used for converting the structured query result obtained by the data query into a semi-structured query result.
An electronic device comprising:

at least one processor;

a memory communicatively coupled to the at least one processor;

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any one of claims 1 to 7 The data processing method described in claim 8, or the data query method described in claim 8.
A computer-readable storage medium storing a computer program, when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7, or the data query method according to claim 8 is realized .