WO2022141287A1

WO2022141287A1 - Data management method and apparatus, data query method and apparatus, service mesh system, and computing device

Info

Publication number: WO2022141287A1
Application number: PCT/CN2020/141773
Authority: WO
Inventors: 于逸凡; 高亮
Original assignee: 西门子（中国）有限公司
Priority date: 2020-12-30
Filing date: 2020-12-30
Publication date: 2022-07-07
Also published as: CN116802628A

Abstract

A data management method and apparatus, a data query method and apparatus, a service mesh system, and a computing device. The data management method comprises: adding a sidecar module for each data source, the edge vehicle module at least comprising a registration submodule, the registration submodule comprising identification information of the data source (S102); parsing the registration module to obtain the identification information of the data source (S104); and registering the identification information of the data source in a service registration center (S106).

Description

Data management and query method, apparatus, service grid system, computing device

technical field

The present disclosure generally relates to the field of database technology, and more particularly, to a data management and query method, apparatus, service grid system, and computing device.

Background technique

The concept of data as a service (DaaS) helps the unified management and application of enterprise data assets, and greatly improves the efficiency. For enterprise data assets, there are various data storage forms in the market according to different needs. On the other hand, service mesh is a hot technology used in microservice scenarios, and its sidecar model helps to discover and register services.

However, there is currently no technical solution for automatically registering and discovering data as services.

SUMMARY OF THE INVENTION

The following presents a brief summary of the present invention in order to provide a basic understanding of certain aspects of the invention. It should be understood that this summary is not an exhaustive overview of the invention. It is not intended to identify key or essential parts of the invention nor to limit the scope of the invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.

In view of this, the present invention proposes a technical solution of applying a service mesh (Service Mesh) technology to data, and realizing DaaS by constructing a unified data specification.

According to an aspect of the present disclosure, a data management method is provided, including: adding a sidecar module for each data source, the sidecar module including at least a registration submodule, and the registration submodule includes identification information of the data source; Parsing the registration module to obtain identification information of the data source; and registering the identification information of the data source in a service registration center.

By adding a sidecar structure to the data source, the automatic registration of the data source is realized.

Optionally, in an example of the above aspect, the sidecar module further includes: a discovery submodule, a flow control submodule, and a routing submodule, where the discovery submodule is used to realize mutual discovery between different data sources; The flow control sub-module is used to control the flow of the data source; the routing sub-module is used to implement data routing of the data source.

In this way, the automatic discovery of data sources and the functions of flow control and routing are further realized.

According to another aspect of the present disclosure, a data management apparatus is provided, comprising: a sidecar adding unit configured to add a sidecar module for each data source, the sidecar module including at least a registration submodule, the registration submodule including the identification information of the data source; a parsing unit, configured to parse the registration sub-module to determine the identification information of the data source; a registration unit, configured to register the identification information of the data source in the service registration center for registration.

According to another aspect of the present disclosure, a data query method is provided, comprising: receiving a user's query request for a data source; a driver browsing a registered data source in a service registration center; The user returns prompt information, and if the data source requested by the user is found in the service registration center, the driver establishes a connection with the data source by using the information in the sidecar module.

Optionally, in an example of the above aspect, the method further includes: mapping the queried data into a standard format based on a pre-established standard interface specification, and returning the data in the standard format to the user

In this way, data sources in different formats can be converted into a unified standard format and returned to the user, so that the user can obtain data in a unified standard format without caring about the storage method of the underlying data.

According to another aspect of the present disclosure, there is provided a service mesh system including a data unit and a control unit, the data unit including a plurality of data sources, each data source having a sidecar module; and the control unit including a service registry a center and a data source parser, where the data source parser is used for parsing the sidecar module and registering the identification information of the data source obtained by parsing in the service registration center.

Optionally, in an example of the above aspect, the system further includes: a driving unit, where the driving unit includes a driver for the multiple data sources and a service discovery module, the service discovery module is configured to When the driving unit receives a query request for the data source from the user, it browses the service registration center to determine whether the data source is registered in the service registration center.

According to another aspect of the present disclosure, there is provided a computing device comprising: at least one processor; and a memory coupled to the at least one processor, the memory for storing instructions, when the instructions are executed by the at least one processor When executed by the processor, the processor is caused to execute the method as described above.

According to another aspect of the present disclosure, there is provided a non-transitory machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the method as described above.

According to another aspect of the present disclosure, there is provided a computer program comprising computer-executable instructions which, when executed, cause at least one processor to perform the method as described above.

According to another aspect of the present disclosure, there is provided a computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed, cause at least one A processor executes the method as described above.

According to the heterogeneous data management method, the data query method and the service grid system of the present invention, by adding a sidecar structure to the data source, the functions of automatic registration and discovery of the data source, flow control and routing are realized. On the other hand, by defining standard data format specifications and APIs, raw data can be mapped into standard formats, so that standard data query results can be provided.

Description of drawings

The above and other objects, features and advantages of the present invention will be more easily understood with reference to the following description of the embodiments of the present invention in conjunction with the accompanying drawings. The components in the drawings are merely intended to illustrate the principles of the invention. In the drawings, the same or similar technical features or components will be denoted by the same or similar reference numerals. In the attached picture:

1 is a flowchart of an exemplary process of a data management method according to a first aspect of an embodiment of the present invention;

2 is a block diagram of an exemplary configuration of a data management apparatus according to an embodiment of the present invention;

3 is a flowchart of an exemplary process of a data query method according to a second aspect of an embodiment of the present invention;

FIG. 4 shows an exemplary architecture diagram of a service mesh system according to an embodiment of the present invention;

5 illustrates a block diagram of a computing device implementing data management and data querying according to an embodiment of the present disclosure.

Among them, the reference numerals are as follows:

100: Data management methods S102, S104, S106, S302, S304, S306,

S308: Steps

200: Data management device 202: Sidecar addition unit

204: Parsing unit 206: Registration unit

300: Data query method 400: Service grid system

401: Data unit 4011: Data source

4012: Sidecar module 402: Control unit

4021: Service Registry 4022: Data Source Resolver

403: Drive unit 4031: Drive

4032: Service Discovery Module 404: Gateway

405: Data User 406: API

500: Computing Equipment 502: Processor

504: memory

Detailed ways

The subject matter described herein will now be discussed with reference to example implementations. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and not to limit the scope of protection, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as desired. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with respect to some examples may also be combined in other examples.

As used herein, the term "including" and variations thereof represent open-ended terms meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment." The term "another embodiment" means "at least one other embodiment." The terms "first", "second", etc. may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. The definition of a term is consistent throughout the specification unless the context clearly dictates otherwise.

First, a few terms mentioned in this specification are briefly explained below.

DaaS: Data is provided as a service. When data management is more centralized, upper-level data consumers do not need to focus on issues such as acquiring underlying data, and can focus entirely on data usage.

Heterogeneous data sources: Different types of databases, such as JDBC type, restful type, etc., different types of databases have different forms of data storage and data connection configuration.

Service Mesh: Service mesh is a hot technology for microservices and other scenarios in recent years. Its core is to achieve rapid service discovery and registration by decoupling the data platform and the control platform. The sidecar structure provides proxies that implement service discovery, load balancing, and other functions.

The present invention applies service mesh (Service Mesh) technology to data, and realizes DaaS by constructing a unified data specification.

The data management method and apparatus according to the embodiments of the present disclosure will be described below first with reference to the accompanying drawings.

FIG. 1 is a flowchart of an exemplary process of a data management method 100 according to the first aspect of the embodiment of the present invention.

First, in step S102, a sidecar module is added for each data source, and the sidecar module includes at least a registration sub-module, and the registration sub-module includes identification information of the data source;

According to the data management method of the embodiment of the present invention, combined with the service mesh (Service Mesh) technology, a plurality of different data sources constitute a service mesh, and it is hoped that the data source can be discovered and registered as a separate service. The data sources here can be heterogeneous, such as stored in different types of databases, such as JDBC, Restful, etc., or stored on a website. Therefore, the data management method according to the present invention can realize the management of heterogeneous data. Specifically, a sidecar module is added for each data source, and the sidecar module can implement the service registration function of the data source. The sidecar module includes a registration sub-module that can extract identification information from the data source. The identification information refers to information that can be used to uniquely identify a specific data source, such as an IP address and/or a port address. The present invention does not limit the specific type of identification information of the data source.

In one example, the sidecar module may further include: a discovery sub-module, a flow control sub-module, and a routing sub-module. The discovery sub-module is used to realize mutual discovery between different data sources; the flow control sub-module is used to control the flow of the data source; the routing sub-module is used to realize the data routing of the data source.

Next, in step S104, the registration sub-module is parsed to obtain identification information of the data source.

By parsing the registration sub-module of the sidecar module, the identification information of the data source can be determined.

Next, in step S106, the identification information of the data source is registered in the service registration center.

Based on the service mesh concept, the data source can be called the data plane, and the service registry that parses the registration sub-modules and registers the data source is located in the control plane. In the present invention, the data plane and the control plane may be located on the same server, or may be located on different servers in the same cluster.

In the data management method according to this embodiment, a sidecar structure is added for each data source, so that the data plane and the control plane are decoupled, and rapid service discovery and registration are realized. The sidecar structure can provide functions such as service registration, service discovery, routing, and flow control to the control plane. The sidecar structure decouples these services from the storage of the data source, making these services independent of the logic of the data source itself.

Corresponding to the first aspect, an embodiment of the present invention further provides a data management apparatus. FIG. 2 is a block diagram of an exemplary configuration of a data management apparatus 200 according to an embodiment of the present invention.

As shown in FIG. 2 , the data management apparatus 200 includes a sidecar adding unit 202 , a parsing unit 204 and a registration unit 206 .

The sidecar adding unit 202 is configured to add a sidecar module for each data source, the sidecar module at least includes a registration sub-module, and the registration sub-module includes identification information of the data source.

The parsing unit 204 is configured to parse the registration submodule to obtain identification information of the data source.

The registration unit 206 is configured to register the identification information of the data source in the service registration center.

In the embodiment of the present invention, the details of the operation and function of each part of the data management apparatus 200 may be the same as or similar to the relevant parts of the embodiment of the data management method 100 of the present invention described with reference to FIG. 1 , and will not be described in detail here. .

FIG. 3 is a flowchart of an exemplary process of a data query method 300 according to the second aspect of the embodiment of the present invention.

The data query method according to this embodiment is to query the database that adds a sidecar module to the data source and registers in the service registration center according to the data management method described with reference to FIG. 1 .

First, in step S302, a query request for a data source from a user is received.

Next, in step S304, the driver browses the registered data sources in the service registration center.

In step S306, if the data source requested by the query is not found, it means that the data source is not registered in the service registration center, and a prompt message is returned to the user; if the data source requested by the user is found in the service registration center, then The driver uses the information in the sidecar module to establish a connection with the data source.

The registration sub-module in the sidecar module includes the identification information of the data source, so the driver can directly establish a connection with the data source based on this information.

In an example, the data query method 300 may further include step S308 : mapping the queried data into a standard format based on a pre-established standard interface specification, and returning the data in the standard format to the user.

It can be understood that different types of databases have different data storage forms. For example, in Restful database, data is stored in json format, not in the form of tables. In the data query method according to the embodiment of the present invention, a standard data storage form may be defined, for example, a set of standard specifications may be specified, and metadata information related to the data source may be defined in the specifications. For example, for JDBC type data sources, you can define user names, passwords, and other information, and for Rest data sources, you can define URL proxy, schema mapping information, etc.; and then convert data sources in different formats into a unified standard The format is returned to the user, so that the user can obtain data in a unified standard format without caring about the storage method of the underlying data. In the present invention, the standard specification of the data can be formulated by the user as required, and the specific format of the data is not limited.

Specifically, the driver can determine the mapping rule between the data format of the data source and the standard data format, and add an API interface to the sidecar module or add an API interface to the data source to realize the above-mentioned operation of converting the format of the data source. In this way, the data converted into the standard format is first returned to the driver, and the driver provides the data in the standard format to the user via the gateway.

FIG. 4 shows an exemplary architecture diagram of a service mesh system 400 according to an embodiment of the present invention, including: a data unit 401 and a control unit 402 .

The data unit 401 includes a plurality of data sources 4011, each data source 4011 having a sidecar module 4012. Data sources 4011 may be heterogeneous data stored on different types of databases or on different websites.

The control unit 402 includes a service registration center 4021 and a data source parser 4022. The data source parser 4022 is used to parse the sidecar module 4012 of the data source 4011, and the identification information of the data source obtained by the analysis is stored in the service registration center. 4021 to register.

In one example, the service mesh system 400 further includes a driving unit 403, and the driving unit 403 includes a driver 4031 and a service discovery module 4032 for different data sources, respectively. The service discovery module 4032 is configured to browse the service registration center 4021 to determine whether the data source 4011 is registered in the service registration center when the driving unit 403 receives the user's query request for the data source 4011 4021.

The data unit 401, the control unit 402 and the drive unit 403 may be located on the same server, or may be located on different servers in the same cluster.

An exemplary process for a user to query data in a service grid system according to an embodiment of the present invention will be described below with reference to FIG. 4 .

In addition to the data unit 401, the control unit 402 and the drive unit 403, FIG. 4 also includes a gateway 404, a data user 405, and an API 406 (application program interface) between the user and the gateway. These components are to help explain the specific process of data query, and are not necessary components of the service grid system according to the embodiment of the present invention.

In the service mesh system 400, a plurality of data sources 4011 have been registered in the service registration center 4021 according to the data management method described above with reference to FIG. 1 .

First, when the user 405 initiates a query request for the data source 4011 to the driver 403 through the gateway 404, the service discovery module 4032 of the driver 403 browses the identification information of the registered data source in the service registration center 4021; if the requested query data is not found If the requested data source is found in the service registration center 4021, the driver 403 uses the information in the sidecar module 4012 to establish with the data source 4011 connect.

After establishing the connection with the data source 4011, the driver determines the mapping rule between the data format of the data source and the standard format based on the pre-established standard specification, and sends the mapping rule to the API interface of the data source, and the API interface will query the data It is mapped into a standard format, and the data in the standard format is returned to the driver 403 , and the driver 403 then returns the data query result in a unified format to the user 405 through the gateway 404 . As mentioned above, the operation of converting the format of the data source can be realized by adding an API interface in the sidecar module 4012 or adding an API interface in the data source 40211, which will not be described in detail here.

A specific application scenario of the data management method and the data query method according to the embodiment of the present disclosure will be described below. In a complex production system, for example, in the industrial production process, the data sources and storage methods are diverse, the data sources may be different instruments and equipment, and the data formats collected from these equipment are also different. Equipment suppliers can choose different databases to store data according to the characteristics of the equipment. When dealing with complex situations, such as modeling and simulating a set of devices, upper-layer applications need to connect multiple data sources. By using the method according to the present invention, the deployed data sources can be registered automatically. Different data sources can also discover each other. This reduces development effort for data sources from different vendors and improves scalability. The added sidecar structure integrates functions unrelated to the data source logic itself, such as registration, discovery, flow control, and routing. Data storage and control functions can be decoupled through the sidecar, which reduces the cost of DevOps (development operations and maintenance). For example, when multiple data sources are stored in the system, the flow control function can solve the congestion problem of network transmission.

According to the data management method, data query method and service grid system of the present invention, a new DaaS solution based on service grid is proposed. By adding a sidecar structure to the data source, the automatic registration and discovery of the data source, as well as the flow control and routing functions, are realized. On the other hand, by defining standard data format specifications and APIs, raw data can be mapped into standard formats, so that standard data query results can be provided. In addition, the connection information of the data source can also be standardized according to the data source type template. When the data source is deployed in the cluster, using the sidecar, the data source that can provide the standard connection information will be automatically registered in the registry.

According to the data management method and the data query method of the present invention, the sidecar structure is used without the need for configuration files, thereby reducing the deployment time of the data source and having better expansibility.

When data is obtained from the underlying database, it will be in a standardized form as a service. From the data consumer's perspective, this eliminates the need to pay attention to how the underlying data is stored.

As above, with reference to FIGS. 1 to 4 , the data management method, the data query method, and the service mesh system according to the embodiments of the present disclosure are described. The above-mentioned data management apparatus and each unit of the service grid system may be implemented by hardware, or may be implemented by software or a combination of hardware and software.

5 illustrates a block diagram of a computing device 500 implementing data management and data querying according to an embodiment of the present disclosure. According to one embodiment, computing device 500 may include at least one processor 502 that executes at least one computer-readable instruction (ie, the above-described in software form) stored or encoded in a computer-readable storage medium (ie, memory 504 ). implemented elements).

It should be understood that the computer-executable instructions stored in memory 504, when executed, cause at least one processor 502 to perform various operations and functions described above in connection with FIGS. 1-4 in various embodiments of the present disclosure.

According to one embodiment, a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may have machine-executable instructions (ie, the above-described elements implemented in software) that, when executed by a machine, cause the machine to perform various embodiments of the present disclosure above in conjunction with FIGS. 1-6 Various operations and functions are described.

According to one embodiment, there is provided a computer program comprising computer-executable instructions that, when executed, cause at least one processor to perform each of the various embodiments of the present disclosure described above in connection with FIGS. 1-6 operations and functions.

According to one embodiment, there is provided a computer program product comprising computer-executable instructions that, when executed, cause at least one processor to perform the various embodiments of the present disclosure described above in connection with FIGS. 1-6 Various operations and functions.

It should be understood that, the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the description of other embodiments. the difference. For example, for the above-mentioned embodiments about the apparatus, the embodiment about the computing device, and the embodiment about the machine-readable storage medium, since they are basically similar to the method embodiments, the descriptions are relatively simple, and the relevant details refer to the method implementation Part of the example can be explained.

Specific embodiments of the present specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Not all steps and units in the above processes and system structure diagrams are necessary, and some steps or units may be omitted according to actual needs. The device structure described in the above embodiments may be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities respectively, or may be implemented by multiple physical entities. Some components in separate devices are implemented together.

The detailed description set forth above in connection with the accompanying drawings describes exemplary embodiments and does not represent all embodiments that may be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not mean "preferred" or "advantage" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described technology. However, these techniques may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.

The above description of the present disclosure is provided to enable any person of ordinary skill in the art to make or use the present disclosure. Various modifications to this disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of this disclosure . Thus, the present disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims

Data management methods, including:

A sidecar module is added for each data source, and the sidecar module includes at least a registration submodule, and the registration submodule includes identification information of the data source;

Parsing the registration module to obtain identification information of the data source; and

The identification information of the data source is registered in the service registration center.
The method of claim 1, wherein the sidecar module further comprises: a discovery sub-module, a flow control sub-module and a routing sub-module, the discovery sub-module is used to realize mutual discovery between different data sources; The flow control sub-module is used to control the flow of the data source; the routing sub-module is used to implement data routing of the data source.
Data management device, including:

a sidecar adding unit, configured to add a sidecar module for each data source, the sidecar module including at least a registration sub-module, and the registration sub-module includes identification information of the data source;

a parsing unit, configured to parse the registration submodule to determine the identification information of the data source;

The registration unit is configured to register the identification information of the data source in the service registration center.
Data query methods, including:

Receive user's query request for data source;

The driver browses the registered data sources in the service registry;

If the data source requested to be queried is not found, a prompt message is returned to the user, and if the data source requested by the user is found in the service registration center, the driver uses the information in the sidecar module to establish a connection with the data source .
The method of claim 4, further comprising:

The queried data is mapped into a standard format based on a pre-established standard interface specification, and the data in the standard format is returned to the user.
Service mesh system, including data unit and control unit,

the data unit includes a plurality of data sources, each data source having a sidecar module; and

The control unit includes a service registration center and a data source resolver, and the data source resolver is used to parse the sidecar module and register the identification information of the data source obtained by the analysis in the service registration center.
The system of claim 6, further comprising: a driving unit, the driving unit comprising a driver and a service discovery module for the plurality of data sources, respectively, the service discovery module for receiving the user information at the driving unit When querying the data source, browse the service registration center to determine whether the data source is registered in the service registration center.
A computing device (500) comprising:

at least one processor (502); and

a memory (504) coupled to the at least one processor (502) for storing instructions that, when executed by the at least one processor (502), cause the processor (502) ) to perform a method as claimed in any one of claims 1 to 2 and 4 to 5.
A non-transitory machine-readable storage medium storing executable instructions which, when executed, cause the machine to perform the method of any one of claims 1-2 and 4-5.
A computer program comprising computer-executable instructions which, when executed, cause at least one processor to perform the method of any one of claims 1-2 and 4-5.
A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions which, when executed, cause at least one processor to perform the execution according to claims 1 to 2 and the method of any one of 4 to 5.