CN114924881A

CN114924881A - Method and device for querying state data result

Info

Publication number: CN114924881A
Application number: CN202210569533.6A
Authority: CN
Inventors: 张政淇; 付海涛
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2022-05-24
Filing date: 2022-05-24
Publication date: 2022-08-19

Abstract

The invention discloses a method and a device for inquiring a state data result, and relates to the technical field of big data. One embodiment of the method comprises: receiving a state data query request, wherein the state data query request carries an operation identifier, a state data name and a query main key; matching at least one task computing node according to the operation identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node; matching the corresponding relation between each hash index interval and each task computing node information according to the name of the state data, and sending a first query request to a first task computing node, wherein the first query request carries a query main key and the corresponding relation between each hash index interval and each task computing node address information; and receiving a state data result returned by the first task computing node, and returning the state data result. The implementation method can solve the technical problem that the handwritten code and the query parameters are difficult to obtain.

Description

Method and device for querying state data result

Technical Field

The invention relates to the technical field of big data, in particular to a method and a device for inquiring a state data result.

Background

In the current latest version design of Flink, runtime state query is based on a client/server mode, and a user needs to manually introduce client-related dependencies and write customized query logic codes. Because different states (used for storing intermediate results or metadata of nodes in a computing process, and the like, which are called state data) often need different query parameters, the query codes are generally difficult to reuse, and the query codes are required to be revised every time different states are queried; and query parameters are difficult to obtain.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for querying status data results, so as to solve the technical problem that handwritten codes and query parameters are difficult to obtain.

To achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a method for querying status data results, applied to a job management node, including:

receiving a state data query request, wherein the state data query request carries an operation identifier, a state data name and a query main key;

matching at least one task computing node according to the operation identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node;

matching the corresponding relation between each hash index interval and each task computing node information according to the state data name, and sending a first query request to the first task computing node, wherein the first query request carries the query key and the corresponding relation between each hash index interval and each task computing node address information;

and receiving a state data result returned by the first task computing node, and returning the state data result.

Optionally, before receiving the status data query request, the method further includes:

for each task computing node, receiving a job identifier, a task computing node identifier, a state data name, a hash index interval and task computing node address information reported by the task computing node;

and storing the corresponding relation among the job identification, the task computing node identification, the state data name, the Hash index interval and the task computing node address information of each task computing node in a local memory.

Optionally, receiving a status data query request includes:

receiving a state data query request forwarded by a website background; the state data query request is sent to the website background by a client;

returning the state data result, including:

and returning the state data result to the website background so that the website background forwards the state data result to the client.

In addition, according to another aspect of the embodiments of the present invention, there is provided a method for querying status data results, applied to a first task computing node, including:

receiving a first query request sent by a job management node, wherein the first query request carries a query main key and a corresponding relation between each hash index interval and each task computing node address information;

calculating a hash index corresponding to the query main key according to the query main key, and determining second task computing node address information corresponding to the query main key according to the corresponding relation between each hash index interval and each task computing node address information;

and sending a second query request to the second task computing node, wherein the second query request carries the query primary key and the hash index corresponding to the query primary key.

Optionally, calculating a hash index corresponding to the query main key according to the query main key, so as to determine, according to a correspondence between each hash index interval and each task computing node address information, second task computing node address information corresponding to the query main key, including:

converting the query primary key into a query primary key object instance; wherein the query primary key is in a json form;

calculating the query primary key object instance by adopting a hash algorithm to obtain a query primary key hash index, thereby determining a hash index interval to which the query primary key hash index belongs;

and matching second task computing node address information corresponding to the query main key based on the corresponding relation between each hash index interval and each task computing node address information.

Optionally, converting the query primary key into a query primary key instance object includes:

acquiring a primary key sequencer to generate primary key type information;

and converting the query primary key and the primary key type information into a query primary key instance object by adopting a json tool library.

Optionally, after the address information of the second task computing node corresponding to the query primary key is matched based on the corresponding relationship between each hash index interval and the address information of each task computing node, the method further includes:

acquiring a primary key sequencer;

generating a query primary key in a byte array form by calling a local function according to the query primary key object instance and the primary key sequencer;

and the query primary key carried by the second query request is in a byte array form.

Optionally, after sending the second query request to the second task computing node, the method further includes:

receiving a state data query result in a byte array form returned by the second task computing node;

converting the state data query result into a state data query result object instance, and converting the state data query result object instance into a json-form state data result;

and forwarding the state data result to the operation management node.

Optionally, converting the state data query result into a state data query result object instance, and converting the state data query result object instance into a json-form state data result, includes:

acquiring a state data sequencer;

generating a state data query result instance object by calling a local function according to the state data query result and the state data sequencer;

and converting the state data query result instance object into a json-form state data result by adopting a json tool library.

Optionally, before receiving the first query request sent by the job management node, the method further includes:

and reporting the operation identification, the task computing node identification, the state data name, the Hash index interval and the task computing node address information of the first task computing node to an operation management node.

In addition, according to another aspect of the embodiments of the present invention, there is provided an apparatus for querying a status data result, which is disposed on a job management node, and includes:

the system comprises a first receiving module, a second receiving module and a sending module, wherein the first receiving module is used for receiving a state data query request, and the state data query request carries a job identifier, a state data name and a query main key;

the selecting module is used for matching at least one task computing node according to the operation identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node;

the first sending module is used for matching the corresponding relation between each hash index interval and each task computing node information according to the state data name and sending a first query request to the first task computing node, wherein the first query request carries the query main key and the corresponding relation between each hash index interval and each task computing node address information;

the first receiving module is further configured to receive a status data result returned by the first task computing node; the first sending module is further configured to return the status data result.

Optionally, the first receiving module is further configured to:

before receiving a state data query request, receiving a job identifier, a task computing node identifier, a state data name, a Hash index interval and task computing node address information which are reported by task computing nodes for each task computing node;

Optionally, the first receiving module is further configured to:

the first sending module is further configured to:

In addition, according to another aspect of the embodiments of the present invention, there is provided an apparatus for querying status data results, disposed in a first task computing node, including:

the second receiving module is used for receiving a first query request sent by the job management node, wherein the first query request carries a query main key and the corresponding relation between each hash index interval and the address information of each task computing node;

the matching module is used for calculating a hash index corresponding to the query main key according to the query main key, so that second task computing node address information corresponding to the query main key is determined according to the corresponding relation between each hash index interval and each task computing node address information;

and the second sending module is used for sending a second query request to the second task computing node, wherein the second query request carries the query primary key and the hash index corresponding to the query primary key.

Optionally, the matching module is further configured to:

acquiring a primary key sequencer to generate primary key type information;

Optionally, the matching module is further configured to:

after the address information of the second task computing node corresponding to the query primary key is matched based on the corresponding relation between each hash index interval and the address information of each task computing node,

acquiring a primary key sequencer;

Optionally, the second receiving module is further configured to:

after a second query request is sent to the second task computing node, receiving a state data query result in a byte array form returned by the second task computing node;

the device also comprises a conversion module, a query module and a query module, wherein the conversion module is used for converting the state data query result into a state data query result object instance and converting the state data query result object instance into a json-form state data result; and forwarding the state data result to the operation management node.

Optionally, the conversion module is further configured to:

acquiring a state data sequencer;

Optionally, the apparatus further includes a reporting module, configured to:

before receiving a first query request sent by a job management node, reporting a job identifier, a task computing node identifier, a state data name, a hash index interval and task computing node address information of the first task computing node to the job management node.

According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the embodiments described above.

According to another aspect of the embodiments of the present invention, there is also provided a computer readable medium, on which a computer program is stored, the program, when executed by a processor, implementing the method according to any of the embodiments described above.

According to another aspect of the embodiments of the present invention, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the above embodiments.

One embodiment of the above invention has the following advantages or benefits: the technical means that at least one task computing node is matched according to the operation identification and the state data name, so that a first task computing node is randomly selected, the corresponding relation between each hash index interval and each task computing node information is matched according to the state data name, a first query request carrying a query main key and the corresponding relation is sent to the first task computing node, and a state data result returned by the first task computing node is received is adopted, so that the technical problem that codes and query parameters are required to be handwritten and are difficult to obtain in the prior art is solved. The embodiment of the invention integrates and encapsulates the work of parameter acquisition, construction, transmission, state query forwarding, result retrieval, serialization and the like in the state query into the kernel code, thereby greatly reducing the use threshold of a user and improving the usability and convenience. Therefore, the user can inquire the runtime state without writing codes, the inquiry parameters are simplified, the user does not need to be concerned about parameter acquisition, and the local inquiry of the online cluster state can be realized, so that the use threshold of the user is obviously reduced.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts. Wherein:

FIG. 1 is a schematic diagram of the main flow of a method of querying status data results, according to one embodiment of the invention;

FIG. 2 is a system framework diagram of querying status data results, according to an embodiment of the invention;

fig. 3 is a schematic diagram of reporting from each task computing node to a job management node according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a main flow of a method for querying status data results according to one exemplary embodiment of the present invention;

FIG. 5 is a schematic diagram of the main flow of a method of querying status data results, according to another embodiment of the invention;

FIG. 6 is a schematic flow chart of computing a query primary key hash index according to an embodiment of the present invention;

fig. 7 is a main flowchart illustrating the processing of the queried state according to an embodiment of the present invention;

FIG. 8 is a diagram illustrating the main flow of a method of querying status data results according to another referenceable embodiment of the present invention;

FIG. 9 is a schematic diagram of the main modules of an apparatus for querying status data results, according to one embodiment of the present invention;

FIG. 10 is a schematic diagram of the main modules of an apparatus for querying status data results according to another embodiment of the present invention;

FIG. 11 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 12 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the current latest version design of Flink, runtime state query is based on a client/server mode, and a user needs to manually introduce client-related dependencies and write customized query logic codes. Since different states often require different query parameters, such query codes are generally difficult to reuse, and the query codes are revised each time a different state is queried. For users, the use is not convenient enough, and multiple times of encoding and packing operation are needed.

In an official design, a server exposes two port service ports (server ports) and a proxy port (proxy port), the server port queries the state stored in the server port, and the proxy port forwards a query request to other task computing nodes. When constructing a query request, an address parameter is required to be transmitted, and the parameter is composed of ip and proxy ports of a task computing node. The state to be inquired must exist on the task computing node corresponding to the ip, and the appointed state is only distributed on part of the task computing nodes, so that a user must select the ip of a correct task computing node by observing methods such as a flight UI operation diagram, and the like, and in addition, the proxy port can only be acquired by looking up a log. The above operation is complicated and complicated, and is not user-friendly.

The client-side query mode is used for querying by directly communicating a certain task computing node in the cluster, and the local nodes cannot be directly communicated under the online production environment, so that a user cannot simply run a program locally for querying, and the query program needs to be put into a bastion machine or packaged into the production environment for running. Because the program is finished after the client end finishes querying each time, the online query program needs to be packaged for many times if the state needs to be queried for many times. This limitation due to the isolation of the production environment increases the use threshold of the user.

In summary, the prior art mainly has the following disadvantages:

a code needs to be handwritten; query parameters are difficult to obtain; the production environment is isolated, and the local inquiry cannot be carried out; each query requires modification of the code and packaging of the lines.

FIG. 1 is a schematic diagram of the main flow of a method of querying status data results, according to one embodiment of the invention. As an embodiment of the present invention, as shown in fig. 1, the method for querying a status data result is applied to a job management node, and may include:

step 101, receiving a status data query request, where the status data query request carries a job identifier, a status data name, and a query main key.

When the query state function is used, a user fills in necessary parameters on a client page, mainly including a job identifier (JobID), a state name and a query main key (key) of a current state to be queried, and then clicks a control to initiate a state query request.

Optionally, receiving a status data query request includes: receiving a state data query request forwarded by a website background; and sending the state data query request to the website background by the client. As shown in fig. 2, a client sends a status data query request to a web backend, where the status data query request carries a job identifier, a status data name, and a query main key, which are input by a user, and after receiving the status data query request sent by the client, the web backend forwards the status data query request to a job management node (a management node in the JobManager, which is responsible for task scheduling in the Flink cluster) in the Flink cluster. Specifically, the web background initiates a REST request to the job management node, the request carries a job identifier, a status data name and a query key, and the job management node starts a series of processes of state query after receiving the request. The embodiment of the invention abandons a client/server mode in official design, and uses REST API to access the state query service, thereby realizing local query of the state of the on-line cluster. The embodiment of the invention provides a method for querying the state through the REST API by modifying the kernel code on the premise of being compatible with the original state query mode of the Flink.

Optionally, before step 101, further comprising: for each task computing node (computing node in the TaskManager, Flink which is responsible for running the task), receiving a job identifier, a task computing node identifier, a state data name, a Hash index interval and task computing node address information which are reported by the task computing node; and storing the corresponding relation among the job identification, the task computing node identification, the state data name, the Hash index interval and the task computing node address information of each task computing node in a local memory. When a Flink program of a client side is initialized and operated, the initialization work related to the state on each task computing node in the Flink cluster is started, at this time, the task computing node stores description information of the state (original name of the state, data type of the key in the state, and the like), a state sequencer and a sequencer of the key in the state in a local state maintenance component of the task computing node, then packs and reports the unique identifiers of the JobID, the state name and the TaskManager to the JobManager, and the JobManager stores the information in a local memory, so that the TaskManager can subsequently inquire which TaskManager the state is stored in according to the JobID and the state name provided in the state inquiry request.

It should be noted that one state may be distributed over multiple taskmanagers, and the state description information is fixed and will not change.

According to the embodiment of the invention, the state description information, the sequencer and other related information are stored or registered in the Flink framework in a mode of actively reporting at the state initialization stage, so that a user does not need to provide the parameters manually when inquiring the state, and how to transmit the parameters to the Flink is not considered.

In order for the first task computing node to be able to match the second task computing node, each task computing node also needs to report the binding relationship between the hash index interval and the address information of the task computing node to the job management node, and the job management node stores the corresponding relationship between each hash index interval and the address information of each task computing node in a local memory so as to forward the corresponding relationship to the first task computing node.

Each task computing node stores a state of a certain hash index interval, as shown in fig. 3, the first task computing node stores a state with a hash index interval of 1-100, so that when the state is initialized at the first task computing node, the first task computing node reports the hash index interval and information (such as an ip and a service port number) of the first task computing node to the operation management node; for another example, the second task computing node stores the state with the hash index interval of 101-200, so when the state is initialized at the second task computing node, the second task computing node reports the hash index interval and the information (such as ip and service port number) of the second task computing node to the job managing node.

And 102, matching at least one task computing node according to the job identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node.

Because each task settlement node reports the binding relationship of the JobID, the state name and the task computing node identifier to the job management node, the job management node can match the corresponding task computing node based on the JobID and the state name carried in the state data query request.

Since one state can be distributed on a plurality of task computing nodes, a plurality of task computing nodes can be matched, and then one task computing node is randomly selected from the matched task computing nodes and used as a first task computing node.

Step 103, matching the corresponding relation between each hash index interval and each task computing node information according to the state data name, and sending a first query request to the first task computing node, where the first query request carries the query key and the corresponding relation between each hash index interval and each task computing node address information.

After the job management node randomly selects the first task computing node, the job management node initiates an RPC request (RPC: remote procedure call, call non-local remote function, and implement effect similar to local call) to the first task computing node, queries a corresponding state result, wherein the request carries a key input by a client and a corresponding relation between each hash index interval and address information (such as IP and service port number) of each task computing node. Optionally, the first query request further carries the job identifier and the status data name, so that the first task computing node matches a task computing node (i.e., a second task computing node) that actually stores the query primary key.

And 104, receiving a state data result returned by the first task computing node, and returning the state data result.

As shown in fig. 2, in the embodiment of the present invention, the job management node (JobManager) and each task computing node (TaskManager) communicate using an RPC protocol, and specifically, a gateway (gateway) is responsible for accepting an RPC request from the JobManager, then calls a local method of the TaskManager, obtains a state result, and returns the state result to the JobManager through the RPC protocol.

Optionally, returning the status data result includes: and returning the state data result to the website background so that the website background forwards the state data result to the client. As shown in fig. 2, after receiving the state result returned by the first task computing node, the job management node returns the state result to the web background, and the web background forwards the state result to the client, so that the user can see the state result on the browser page.

In the state query process, the request forwarding and transmission are completed by RPC calling between bottom layer Flink components, and a user does not need to pay attention to how to acquire address information such as ip and service port numbers. After the state is inquired, the work of serialization transmission and the like is also finished in the Flink framework, and the work is completely transparent to the user.

According to the various embodiments described above, it can be seen that the technical means of the embodiments of the present invention, by matching at least one task computing node according to the job identifier and the state data name, randomly selecting a first task computing node, matching the corresponding relationship between each hash index interval and each task computing node information according to the state data name, sending a first query request carrying a query main key and the corresponding relationship to the first task computing node, and receiving a state data result returned by the first task computing node, solves the technical problem in the prior art that a handwritten code and a query parameter are difficult to obtain. The embodiment of the invention integrates and encapsulates the work of parameter acquisition, construction, transmission, state query forwarding, result retrieval, serialization and the like in the state query into the kernel code, thereby greatly reducing the use threshold of a user and improving the usability and convenience. Therefore, the user can inquire the runtime state without writing codes, the inquiry parameters are simplified, the user does not need to care about parameter acquisition, and the local inquiry of the online cluster state can be realized, so that the use threshold of the user is obviously reduced.

Fig. 4 is a schematic diagram of a main flow of a method of querying a status data result according to one referential embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 3, the method for querying status data results is applied to a job management node, and may include:

step 401, for each task computing node, receiving a job identifier, a task computing node identifier, a state data name, a hash index interval, and task computing node address information reported by the task computing node.

Step 402, storing the corresponding relation among the job identification, the task computing node identification, the state data name, the hash index interval and the task computing node address information of each task computing node in a local memory.

Step 403, receiving a status data query request forwarded by a website background; the state data query request is sent to the website background by the client, and the state data query request carries an operation identifier, a state data name and a query main key.

And 404, matching at least one task computing node according to the job identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node.

Step 405, matching the corresponding relationship between each hash index interval and each task computing node information according to the name of the state data, and sending a first query request to the first task computing node, where the first query request carries the query key and the corresponding relationship between each hash index interval and each task computing node address information.

Step 406, receiving the status data result returned by the first task computing node.

Step 407, returning the state data result to the website background, so that the website background forwards the state data result to the client.

In addition, in a reference embodiment of the present invention, the detailed implementation of the method for querying status data results is already described in detail in the above-mentioned method for querying status data results, and therefore the repeated content is not described herein.

FIG. 5 is a diagram illustrating the main flow of a method for querying status data results according to another embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 5, the method for querying a status data result is applied to a first task computing node, and may include:

step 501, receiving a first query request sent by a job management node, where the first query request carries a query main key and a corresponding relationship between each hash index interval and address information of each task computing node.

The job management node sends a first query request to the first task computing node, where the first query request carries a query primary key and a corresponding relationship between each hash index interval and address information of each task computing node, and in some embodiments, the first query request also carries a job identifier and a status data name, so that the first task computing node matches a task computing node (i.e., a second task computing node) that actually stores the query primary key.

Step 502, calculating a hash index corresponding to the query key according to the query key, thereby determining second task computing node address information corresponding to the query key according to a corresponding relationship between each hash index interval and each task computing node address information.

The first task computing node computes the hash index of the query main key according to the query main key (key), the job identification and the state data name, and then queries the task computing node (namely, the second task computing node) which really stores the query main key based on the corresponding relation between each hash index interval and the address information of each task computing node.

Optionally, step 502 may include: converting the query primary key into a query primary key object instance; wherein the query primary key is in a json form; calculating the query key object instance by adopting a hash algorithm to obtain a query key hash index, thereby determining a hash index interval to which the query key hash index belongs; and matching second task computing node address information corresponding to the query main key based on the corresponding relation between each hash index interval and each task computing node address information. In the embodiment of the invention, a first task computing node firstly converts a json-form query main key (key) into a corresponding java object instance, then calculates the java object instance of the key by adopting a hash algorithm to obtain a key hash index, thereby determining a hash index interval to which the key hash index belongs, and finally queries a task computing node (namely a second task computing node) which really stores the key based on the corresponding relation between each hash index interval and the address information of each task computing node.

Optionally, converting the query primary key into a query primary key instance object includes: acquiring a primary key sequencer to generate primary key type information; and converting the query primary key and the primary key type information into a query primary key instance object by adopting a json tool library. As shown in fig. 5, after receiving the first query request, the first task computing node extracts a key serializer based on the JobID and the state name and by using a Flink local function (KvStateInfo # getkeyseralizer), thereby generating key type information (data types in the Flink, including numbers, character strings, user self-defined classes, and the like), then converts the json form key and the key type information into java object instances of the key by using a json tool library, and finally calculates the key index by using a hash algorithm. After the key index corresponding to the key is calculated, the address of the TaskManager where the key index is located can be inquired according to the corresponding relation between each hash index interval and each task computing node address information, which are transmitted by the RPC request, and the address is the server port inquired locally by the ip and the state of the TaskManager in the TaskManager.

Optionally, after matching address information of a second task computing node corresponding to the query primary key based on a corresponding relationship between each hash index interval and each task computing node address information, the method further includes: acquiring a primary key sequencer; and generating a query primary key in a byte array form by calling a local function according to the query primary key object instance and the primary key sequencer. 6-7, in order for the second task computing node to quickly query out the state result, the first task computing node also needs to convert the json form of keys into byte array form of keys. Specifically, the first task computing node acquires the key serializer from the local, then generates the key in the form of a byte array according to the java object instance of the key and the key serializer, and by calling a flight local function (kvstatesizer # seriarizeKeyAndNamespace).

Step 503, sending a second query request to the second task computing node, where the second query request carries the query primary key and the hash index corresponding to the query primary key.

After a task computing node (namely a second task computing node) which really stores the query main key is queried, the first task computing node sends a second query request to the second task computing node, and the second query request carries the key and the key index corresponding to the key. The key carried by the second query request may be in a byte array form, so as to facilitate the second task computing node to query the state result.

According to the various embodiments described above, it can be seen that the technical means of calculating the hash index corresponding to the query primary key according to the query primary key, determining the address information of the second task computing node corresponding to the query primary key according to the corresponding relationship between each hash index interval and each task computing node address information, and then sending the second query request to the second task computing node in the embodiments of the present invention solves the technical problem that it is difficult to obtain the handwritten code and the query parameter in the prior art. The embodiment of the invention integrates and encapsulates the work of parameter acquisition, construction, transmission, state query forwarding, result retrieval, serialization and the like in the state query into the kernel code, thereby greatly reducing the use threshold of a user and improving the usability and convenience. Therefore, the user can inquire the runtime state without writing codes, the inquiry parameters are simplified, the user does not need to be concerned about parameter acquisition, and the local inquiry of the online cluster state can be realized, so that the use threshold of the user is obviously reduced.

FIG. 8 is a diagram illustrating a main flow of a method for querying status data results according to one exemplary embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 7, the method for querying status data results is applied to a first task computing node, and may include:

step 801, receiving a first query request sent by a job management node, where the first query request carries a query main key and a corresponding relationship between each hash index interval and address information of each task computing node.

Step 802, calculating a hash index corresponding to the query key according to the query key, and determining second task computing node address information corresponding to the query key according to a corresponding relationship between each hash index interval and each task computing node address information.

Step 803, a second query request is sent to the second task computing node, where the second query request carries the query key and the hash index corresponding to the query key.

As shown in FIG. 2, a first task computing node utilizes a gateway (gateway) to forward an RPC request to a second task computing node that actually stores the state of a key. Alternatively, the gateway may call a forwarding component in the proxy (proxy) to forward the RPC request, that is, in the process of querying the state, the gateway calls the proxy to forward the state query request, and does not forward the query request directly.

After receiving the second query request, the second task computing node parses the second query request to obtain a key and a key index corresponding to the key, and then calls a Flink local function (kvstateseverhandler # getserialedvaluativalue) to obtain a state result in a byte array form, as shown in fig. 7, and the state result is encapsulated and then transmitted back to the first task computing node.

Step 804, receiving the state data query result in the form of byte array returned by the second task computing node.

Step 805, converting the state data query result into a state data query result object instance, and converting the state data query result object instance into a json-form state data result.

After receiving the state query result in the byte array form returned by the second task computing node, the first task computing node converts the state query result into a state result in a json form, so that a user can conveniently view the state result on a browser page.

Optionally, step 805 may include: acquiring a state data sequencer; generating a state data query result instance object by calling a local function according to the state data query result and the state data sequencer; and converting the state data query result instance object into a json-form state data result by adopting a json tool library. As shown in fig. 7, the first task computing node receives the state query result in the form of byte array returned by the second task computing node, then generates a java object instance of the state query result by calling a Flink local function (kvstatebuilder # createStateObject), and finally converts the java object instance of the state query result into the state result in the form of character string by adopting a json tool library.

Step 806, forward the state data result to the job management node.

As shown in fig. 2, the first task computing node transmits the state result in the form of a character string back to the job management node, and then the job management node transmits the state result back to the web background, and the web background forwards the state result to the client, so that the user can see the state result on the browser page.

In addition, in another embodiment of the present invention, the detailed implementation of the method for querying status data results is described in detail above, so that the repeated description is not repeated here.

FIG. 9 is a schematic diagram of the main blocks of an apparatus for querying status data results, according to one embodiment of the present invention. As shown in fig. 9, the apparatus 900 for querying status data result is disposed in a job management node, and includes a first receiving module 901, a selecting module 902, and a first sending module 903; the first receiving module 901 is configured to receive a status data query request, where the status data query request carries a job identifier, a status data name, and a query main key; the selecting module 902 is configured to match at least one task computing node according to the job identifier and the state data name, and randomly select a first task computing node from the at least one task computing node; the first sending module 903 is configured to match a corresponding relationship between each hash index interval and each task computing node information according to the name of the state data, and send a first query request to the first task computing node, where the first query request carries the query main key and a corresponding relationship between each hash index interval and each task computing node address information; the first receiving module 901 is further configured to receive a status data result returned by the first task computing node; the first sending module 903 is further configured to return the status data result.

Optionally, the first receiving module 901 is further configured to:

the first sending module 903 is further configured to:

It should be noted that, in the implementation of the apparatus for querying status data results according to the present invention, the above method for querying status data results has been described in detail, and therefore, the repeated content is not described herein.

Fig. 10 is a schematic diagram of main blocks of an apparatus for querying a status data result according to another embodiment of the present invention. As shown in fig. 10, the apparatus 1000 for querying status data result is disposed in a first task computing node, and includes a second receiving module 1001, a matching module 1002, and a second sending module 1003; the second receiving module 1003 is configured to receive a first query request sent by a job management node, where the first query request carries a query main key and a corresponding relationship between each hash index interval and address information of each task computing node; the matching module 1002 is configured to calculate a hash index corresponding to the query key according to the query key, so as to determine, according to a correspondence between each hash index interval and each task computing node address information, second task computing node address information corresponding to the query key; the second sending module 1003 is configured to send a second query request to the second task computing node, where the second query request carries the query primary key and the hash index corresponding to the query primary key.

Optionally, the matching module 1002 is further configured to:

acquiring a primary key sequencer to generate primary key type information;

Optionally, the matching module 1002 is further configured to:

acquiring a primary key sequencer;

Optionally, the second receiving module 1001 is further configured to:

the device also comprises a conversion module, a state data query module and a state data query module, wherein the conversion module is used for converting the state data query result into a state data query result object instance and converting the state data query result object instance into a json-form state data result; and forwarding the state data result to the operation management node.

Optionally, the conversion module 1002 is further configured to:

acquiring a state data sequencer;

Optionally, the apparatus further includes a reporting module, configured to:

Fig. 11 illustrates an exemplary system architecture 1100 to which the method of querying status data results or the apparatus querying status data results of embodiments of the invention may be applied.

As shown in fig. 11, the system architecture 1100 may include

terminal devices

1101, 1102, 1103, a network 1104 and a server 1105. The network 1104 is a medium to provide communication links between the

terminal devices

1101, 1102, 1103 and the server 1105. Network 1104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may use

terminal devices

1101, 1102, 1103 to interact with a server 1105 over a network 1104 to receive or send messages or the like. Various messaging client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (examples only) may be installed on the

terminal devices

1101, 1102, 1103.

The

terminal devices

1101, 1102, 1103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 1105 may be a server that provides various services, such as a backend management server (for example only) that provides support for shopping-like websites browsed by users using the

terminal devices

1101, 1102, 1103. The background management server can analyze and process the received data such as the article information query request and feed back the processing result to the terminal equipment.

It should be noted that the method for querying status data results provided by the embodiment of the present invention is generally executed by the server 1105, and accordingly, the apparatus for querying status data results is generally disposed in the server 1105.

It should be understood that the number of terminal devices, networks, and servers in fig. 11 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 12, shown is a block diagram of a computer system 1200 suitable for use with a terminal device implementing embodiments of the present invention. The terminal device shown in fig. 12 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.

As shown in fig. 12, the computer system 1200 includes a Central Processing Unit (CPU)1201, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)1202 or a program loaded from a storage section 1208 into a Random Access Memory (RAM) 1203. In the RAM1203, various programs and data necessary for the operation of the system 1200 are also stored. The CPU 1201, ROM 1202, and RAM1203 are connected to each other by a bus 1204. An input/output (I/O) interface 1205 is also connected to bus 1204.

The following components are connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output portion 12012 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 1208 including a hard disk and the like; and a communication section 1209 including a network interface card such as a LAN card, a modem, or the like. The communication section 1209 performs communication processing via a network such as the internet. A driver 1210 is also connected to the I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is mounted on the drive 1210 as necessary, so that a computer program read out therefrom is mounted into the storage section 1208 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1209, and/or installed from the removable medium 1211. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 1201.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a first receiving module, a selecting module, and a first sending module, wherein the names of the modules do not in some cases constitute a limitation on the module itself.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a second receiving, matching module and a second sending module, where the names of the modules do not in some cases constitute a limitation on the module itself.

As another aspect, the present invention also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: receiving a state data query request, wherein the state data query request carries an operation identifier, a state data name and a query main key; matching at least one task computing node according to the operation identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node; matching the corresponding relation between each hash index interval and each task computing node information according to the state data name, and sending a first query request to the first task computing node, wherein the first query request carries the query key and the corresponding relation between each hash index interval and each task computing node address information; and receiving a state data result returned by the first task computing node, and returning the state data result.

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: receiving a first query request sent by a job management node, wherein the first query request carries a query main key and a corresponding relation between each hash index interval and each task computing node address information; calculating a hash index corresponding to the query main key according to the query main key, and determining second task calculation node address information corresponding to the query main key according to the corresponding relation between each hash index interval and each task calculation node address information; and sending a second query request to the second task computing node, wherein the second query request carries the query primary key and the hash index corresponding to the query primary key.

As another aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any of the above embodiments.

According to the technical scheme of the embodiment of the invention, at least one task computing node is matched according to the job identification and the state data name, so that a first task computing node is randomly selected, the corresponding relation between each hash index interval and each task computing node information is matched according to the state data name, a first query request carrying a query main key and the corresponding relation is sent to the first task computing node, and a state data result returned by the first task computing node is received. The embodiment of the invention integrates and encapsulates the work of parameter acquisition, construction, transmission, state query forwarding, result retrieval, serialization and the like in the state query into the kernel code, thereby greatly reducing the use threshold of a user and improving the usability and convenience. Therefore, the user can inquire the runtime state without writing codes, the inquiry parameters are simplified, the user does not need to be concerned about parameter acquisition, and the local inquiry of the online cluster state can be realized, so that the use threshold of the user is obviously reduced.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for querying status data results, applied to a job management node, comprises:

matching the corresponding relation between each hash index interval and each task computing node information according to the state data name, and sending a first query request to the first task computing node, wherein the first query request carries the query main key and the corresponding relation between each hash index interval and each task computing node address information;

2. The method of claim 1, wherein prior to receiving the status data query request, further comprising:

3. The method of claim 1, wherein receiving a status data query request comprises:

returning the state data results, including:

4. A method for querying status data results, applied to a first task computing node, comprising:

calculating a hash index corresponding to the query main key according to the query main key, and determining second task calculation node address information corresponding to the query main key according to the corresponding relation between each hash index interval and each task calculation node address information;

and sending a second query request to the second task computing node, wherein the second query request carries the query key and the hash index corresponding to the query key.

5. The method according to claim 4, wherein calculating a hash index corresponding to the query key according to the query key, and thereby determining second task computing node address information corresponding to the query key according to a correspondence between each hash index interval and each task computing node address information, comprises:

calculating the query key object instance by adopting a hash algorithm to obtain a query key hash index, thereby determining a hash index interval to which the query key hash index belongs;

6. The method of claim 5, wherein converting the query primary key into a query primary key instance object comprises:

acquiring a primary key sequencer to generate primary key type information;

7. The method according to claim 5, wherein after matching out the address information of the second task computing node corresponding to the query primary key based on the corresponding relationship between each hash index interval and each task computing node address information, further comprising:

acquiring a primary key sequencer;

8. The method of claim 4, wherein after sending the second query request to the second task compute node, further comprising:

and forwarding the state data result to the operation management node.

9. The method of claim 8, wherein converting the state data query result into a state data query result object instance and converting the state data query result object instance into a json form state data result comprises:

acquiring a state data sequencer;

10. The method of claim 4, wherein before receiving the first query request sent by the job management node, further comprising:

11. An apparatus for querying status data results, disposed on a job management node, includes:

the selecting module is used for matching out at least one task computing node according to the operation identification and the state data name, and randomly selecting a first task computing node from the at least one task computing node;

12. An apparatus for querying status data results, disposed in a first task computing node, comprising:

and the second sending module is used for sending a second query request to the second task computing node, wherein the second query request carries the query key and the hash index corresponding to the query key.

13. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

the one or more programs, when executed by the one or more processors, implement the method of any of claims 1-10.

14. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-10.

15. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-10.