CN115361435A

CN115361435A - Data processing method

Info

Publication number: CN115361435A
Application number: CN202210821136.3A
Authority: CN
Inventors: 王梦杰; 黄敏飞
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-07-13
Filing date: 2022-07-13
Publication date: 2022-11-18

Abstract

An embodiment of the present specification provides a data processing method, which is applied to a data processing system, where the system includes a client management module, an initial client service module, and an update client service module, and the method includes: the client management module responds to an update request, acquires the update client service module and sends a received data processing request to the update client service module, wherein the data processing request carries data to be processed; and the update client service module acquires unit information of the target data processing unit from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, and calls the target data processing unit to process the data to be processed based on the unit information, so that the data processing efficiency and the user experience are improved in the process of upgrading the client.

Description

Data processing method

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to a data processing method.

Background

With the development of computer technology, many internet enterprises may publish clients of a specific product to users so that the users use the corresponding product through the clients, for example, clients of a distributed system may be published to the users so that the users can use services provided by the distributed system based on the clients.

However, in the process of upgrading the client, since a long time is required for data synchronization operation and restart operation between the new client and the old client, the client cannot process data in the upgrading process, so that the data processing efficiency is reduced, and the user experience is poor.

Disclosure of Invention

In view of this, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program, so as to solve the technical problems in the prior art.

According to a first aspect of embodiments of the present specification, there is provided a data processing method applied to a data processing system, where the system includes a client management module, an initial client service module, and an update client service module, and the method includes:

the client management module responds to an update request, acquires the update client service module and sends a received data processing request to the update client service module, wherein the data processing request carries data to be processed;

and the update client service module acquires unit information of the target data processing unit from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, and calls the target data processing unit to process the data to be processed based on the unit information.

According to a second aspect of embodiments herein, there is provided a data processing system comprising a client management module, an initial client service module, an updated client service module, wherein,

the client management module is configured to respond to an update request, acquire the update client service module and send a received data processing request to the update client service module, wherein the data processing request carries data to be processed;

the update client service module is configured to acquire unit information of the target data processing unit from the initial client service module and call the target data processing unit to process the data to be processed based on the unit information under the condition that it is determined that the target data processing unit for processing the data to be processed does not exist.

According to a third aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions and the processor is configured to execute the computer-executable instructions, which when executed by the processor implement the steps of the data processing method applied to the client management module, the initial client service module or the update client service module.

According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method described above as applied to a client management module, an initial client service module, or an update client service module.

According to a fifth aspect of embodiments herein, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method applied to a client management module, an initial client service module or an updated client service module.

The data processing method provided by the specification is applied to a data processing system, the system comprises a client management module, an initial client service module and an updated client service module, and the method comprises the following steps: the client management module responds to an update request, acquires the update client service module and sends a received data processing request to the update client service module, wherein the data processing request carries data to be processed; and the update client service module acquires unit information of the target data processing unit from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, and calls the target data processing unit to process the data to be processed based on the unit information.

Specifically, the method includes the steps that firstly, an updating client service module corresponding to an initial client service module is obtained through a client management module, so that the initial client service module is updated based on the updating client service module, then a received data processing request is sent to the updating client service module to be processed, and the updating client service module calls a target data processing unit to process data to be processed through unit information of the target data processing unit obtained from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, so that the data to be processed can be processed in the client updating process, the data processing efficiency is improved, and the use experience of a user is improved.

Drawings

Fig. 1 is a schematic view of an application scenario of a data processing method provided in an embodiment of the present specification;

fig. 2 is a processing schematic diagram of a data processing method applied to an online hot upgrade scenario performed on a client according to an embodiment of the present specification;

FIG. 3 is a flow chart of a data processing method provided by an embodiment of the present specification;

FIG. 4 is a flow chart illustrating a process of a method for data processing according to an embodiment of the present disclosure;

FIG. 5 is a block diagram of a data processing system, according to an embodiment of the present disclosure;

fig. 6 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if," as used herein, may be interpreted as "at \8230; \8230when" or "when 8230; \823030when" or "in response to a determination," depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

Hot upgrading: and upgrading without stopping service.

File flow: refers to a file stream class (FileStream class), a class used for implementing operations such as reading data in a file and writing data in a file.

Dynamic library: generally referred to as a dynamically linked library.

Symbol version: symbol versioning mechanism, a technique for versioning dynamic library symbols.

As computer technology has developed, many internet enterprises may publish clients of a particular product to users to enable the users to use the corresponding product through the clients, for example, clients of a distributed system may be published to the users to enable the users to use the distributed system based on the clients.

However, the client of the distributed system is in the process of the user, once the client is released to the user, the developer cannot control the version of the client, and can only push the user to upgrade the version of the client in a manual mode, which consumes a large amount of human resources and brings burden to the user.

In view of this problem, the present specification provides two client upgrade schemes: one scheme for upgrading the client is to output the client in a dynamic library form; when the client has a new version, the new client dynamic library is downloaded to the machine where the client is located, so that when the user process is restarted, the new client version can be used. However, the problem with this solution is that the user is still required to restart the process, and for some users, the process will normally run for a long time, and if the user needs to restart, data/task migration is required, which will bring extra burden to the process and may affect the service provided by the client to the user.

Another solution for performing online hot upgrade on a client is to treat the client as a separate service process, separate from the user process. When the client is upgraded in a hot mode, the client process is started by using the new version (the old client process is still in service), and then the file stream in the old client is reopened in the new client and the service is resumed. However, a disadvantage of this solution is that in some scenarios it is not allowed to reopen the file stream for reasons of hot upgrades or to allow the client to exist as a separate process. In addition, since data needs to be transferred between processes from a user process to a client process, additional overhead is caused, and delay and throughput are affected.

In view of this, in the present specification, a data processing method is provided, and the present specification simultaneously relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

Fig. 1 is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of the present specification, where the data processing method is applied to a data processing system, the system includes a client management module 102, an initial client service module 106, and an update client service module 104, and the method includes:

the client management module 102, in response to an update request, obtains the update client service module 104, and sends a received data processing request to the update client service module 104, where the data processing request carries data to be processed;

the update client service module 104, when determining that there is no target data processing unit for processing the data to be processed, acquires unit information of the target data processing unit from the initial client service module 106, and calls the target data processing unit to process the data to be processed based on the unit information.

The client service module can be understood as a client, and is a client program running on the user terminal or a client instance running on the user terminal; in order to avoid excessive redundancy, the client service module is referred to as a client in the following embodiments; correspondingly, the update client service module 104 may be referred to as an update client, and the initial client service module 106 may be referred to as an initial client.

The data processing system may be understood as a system that processes data, such as a distributed storage system, a distributed file system, or the like. In practical applications, the modules included in the data processing system, such as the client management module 102, the initial client service module 106, and the update client service module 104, may be run in a user terminal. The client management module 102 may be understood as a module within the data processing system that manages client service modules. In practical application, in the data processing method provided in this specification, a client management component is newly added in a distributed file system, and the client management component may be understood as the client management module 102, and may be responsible for forwarding a request to a client and managing a client hot upgrade; it should be noted that the client management component (i.e. the client management module 102) may be a client management instance, a client management program, etc. for managing a client, and this specification is not limited in this respect. In addition, the logic of the client management component is simple, and generally upgrading is not needed. In addition, the client management component is in the process of the user, and the client management component can load the client dynamic library by itself without the concern of the user.

The update client service module 104 may be understood as a new client in the distributed system that performs a hot upgrade on an old version client; it should be noted that the client is externally published in the form of a dynamic library.

The initial client service module 106 may be understood as an old version of the client in the distributed system. The update request may be understood as a request that instructs the client management model to upgrade the initial client service module 106. The data processing request may be understood as a request that requires processing by a client, such as a data read request, a data write request, and the like. The data to be processed can be understood as data which needs to be processed by the client; for example, data for which a client needs to perform a write operation, data for which a client needs to perform a read operation, and the like, the data processing unit may be understood as a class that performs various types of operations on data, for example, a file stream class. The target data processing unit may be understood as a class that performs operations on the data to be processed. The unit information may be understood as file flow information of the data processing unit. The filestream information may be metadata, e.g., metadata of a filestream class.

Specifically, the data processing method provided in this specification provides an online hot-upgrade scheme for distributed file system clients, where a client management module 102 in the data processing system can receive an update request for updating an initial client service module 106 in the data processing system, and the client management module 102 can obtain an update client service module 104 in response to the update request, so as to upgrade the initial client service module 106 based on the update client service module 104. Meanwhile, in the process of hot upgrade, in order to ensure that the client service module can provide service for the user and process the received data processing request in time, after the client management model sends the received data processing request carrying the data to be processed to the update client service module 104, the update client service module 104 first determines whether a target data processing unit for processing the data to be processed exists.

If not, that is, under the condition that there is no target data processing unit for processing the data to be processed, the update client service module 104 can obtain the unit information of the target data processing unit from the initial client service module 106, and call the target data processing unit to process the data to be processed based on the unit information.

If so, that is, under the condition that there is a target data processing unit for processing the data to be processed, the update client service module 104 directly calls the target data processing unit to process the data to be processed.

The data processing method provided in this specification is described below by taking an application of the data processing method in a scenario of performing online hot upgrade on a distributed file system client as an example. Referring to fig. 2, fig. 2 is a processing schematic diagram of a scenario in which a data processing method provided in an embodiment of the present specification is applied to perform online hot upgrade on a client. The client management module is a client management component, the initial client service module is a client dynamic library (version 1), the updated client service module is a client dynamic library (version 2), and the unit information is file stream information. It should be noted that the client management component forwards the data write request sent by the user to the currently running client.

Based on this, when a new version of the client is generated, the central management node generates an update instruction for the client dynamic library (version 1) to the client management component, where the update instruction carries identification information of the client dynamic library (version 2); after receiving the update instruction sent by the central management node, the client management component running in the user process downloads a new version of the client dynamic library (version 2) from the central management node to the local based on the identification information, and the local can be understood as a local disk.

It should be noted that the client management component running in the user process maintains two client instances, one as a primary instance, i.e. the instance currently being served, and the other as a backup instance. The client instance is not in a running user process.

Based on this, after the client management component downloads the client dynamic library (version 2) of the new version, the client management component performs hot upgrade on the old version client (the client dynamic library (version 1)) based on the new version client (namely the client dynamic library (version 2)), and in the process of performing hot upgrade, the client management component firstly analyzes the local standby example (if the standby example exists), then loads the new client dynamic library, re-instantiates the new dynamic library (namely the new version client), obtains the standby client example generated by the new version client, and initializes the standby client example.

Then, after the initialization of the standby client instance (i.e., the new version client instance) corresponding to the new version client is completed, the new version client instance performs information exchange and synchronization with the old client instance, and acquires file stream information from the old version client instance, that is, file stream information migration between the client dynamic library (version 1) and the client dynamic library (version 2) in fig. 2; the file stream information may be metadata of a file stream class. It should be noted that, the client dynamic library (version 2) may synchronize the file stream information from the client dynamic library (version 1) by calling an information synchronization function B provided by the client dynamic library (version 1), where the information synchronization function B is a function for acquiring file stream information that has not been synchronized to a new client in an old client, and calling the information synchronization function B. After the file stream information synchronization of the client dynamic library (version 1) and the client dynamic library (version 2) is completed, the hot upgrade of the client dynamic library (version 1) can be determined to be completed.

Meanwhile, in the data processing method provided by the present specification, in order to ensure that the request sent by the user can be processed in the process of the client hot upgrade as well, and ensure that the user is unaware of the client upgrade, the client management component forwards the received data write request to the new version client instance, that is, the client dynamic library (version 2) in fig. 2.

After receiving the data write request, the new version client instance determines the data to be written carried in the data write request and a file identifier A1 to which the data to be written needs to be written.

Then, the new version client instance determines the file flow information corresponding to the file identifier A1 from the file flow information held by the new version client instance based on the file identifier A1, calls the file flow class A2 based on the file flow information, instantiates the file flow class A2, and obtains an instantiated file flow A3; and writing the data to be written into the file corresponding to the file identifier A1 based on the file stream A3. It should be noted that there is a corresponding relationship between the file stream class and the file, and one file stream class may correspond to one or more files. Data can be written into the corresponding file or read from the corresponding file through the file stream class.

However, in the new version client instance, under the condition that there is no filestream information corresponding to the file identifier A1, it is indicated that the filestream information has not been synchronized from the old version client instance, and therefore, in order to avoid the problem that the new version client cannot timely process the data write request due to the fact that the filestream information is not timely synchronized, the new version client instance can also call an information obtaining function a provided by the old version client instance, where the information obtaining function a is a function for obtaining specified filestream information from the old client instance, and by calling the information obtaining function a, the new version client instance can obtain the filestream information corresponding to the file identifier A1 from the old version client instance.

Then, the new version client instance can call the file stream class A2 based on the file stream information, instantiate the file stream class A2 and obtain an instantiated file stream A3; and writing the data to be written into the file corresponding to the file identifier A1 based on the file stream A3. Therefore, the method and the device can process the request of the user in the process of carrying out hot upgrade on the client, and ensure the imperceptibility of the user.

In the data processing method provided by the specification, a mode of performing online hot upgrade on a distributed file system client is provided, so that a maintainer (namely a client management component) of a back-end distributed file system can perform hot upgrade by himself without user process perception, and meanwhile, the upgrade has no influence on the service of a user, and the user is ensured to be unaware.

Fig. 3 is a flowchart illustrating a data processing method applied to a data processing system including a client management module, an initial client service module, and an update client service module according to an embodiment of the present specification, where the method specifically includes the following steps.

Step 302: and the client management module responds to the updating request, acquires the updating client service module and sends the received data processing request to the updating client service module.

Wherein, the data processing request carries the data to be processed.

In an embodiment provided in this specification, when determining that there is a new version client, the central management node instructs the client management module to actively perform hot upgrade on the client, so as to ensure that the developer can control the client version, and avoid the consumption of human cost caused by pushing a user to perform upgrade in a manual manner, where specifically, the client management module, in response to an update request, obtains the update client service module, and includes:

the client management module receives an update request aiming at the initial client service module, which is sent by a client management node, wherein the update request carries a module identifier of the update client service module;

and acquiring the updated client service module from the client management node based on the module identifier.

The module identifier may be understood as an identifier that uniquely identifies a client service module, that is, an identifier that uniquely identifies a client, for example, a version number, a client name, and the like of the client, which is not specifically limited in this specification.

The client management node can be understood as a node which stores the latest version of the client and indicates the client management module to upgrade the client; the client management node may be a node in a distributed storage system; for example, the central management node in the above embodiments, that is, the data processing system includes a client management node.

Specifically, the client management node generates an update request for the initial client service module based on the module identifier of the update client service module when determining that the client has a new version; sending the updating request to a client management module;

after receiving the update request, the client management module can acquire the module identifier of the update client service module carried in the update request, generate a client download request based on the module identifier of the update client service module, and send the client download request to the client management node.

And when receiving the client downloading request, the client management node searches the corresponding updated client service module based on the module identifier of the updated client service module carried in the client downloading request, and sends the updated client service module to the client management module. Based on this, the client management module obtains the update client.

In an embodiment provided in this specification, the client management node is not only responsible for managing the client hot upgrade, but also can forward the request to the client for processing, so that the client service module can process the user request, and specifically, the sending the received data processing request to the update client service module includes:

and the client management module receives a data processing request sent by a user and sends the data processing request to the update client service module for processing.

Specifically, when the user needs to process data, for example, data is read, written, and the like, a data processing request is sent to the client management module running in the user process, and the client management module, upon receiving the data processing request, forwards the data processing request to the currently running update client service module.

It should be noted that, after obtaining the update client service module, the client management module determines the update client service module as the currently running client service module, and sends the received data processing request to the update client service module.

In an embodiment provided in this specification, the client management module may maintain two client instances, a primary client, and a corresponding backup client (i.e., a backup client instance), so as to ensure data security, and when an error occurs in the client, the client may replace the primary client to work, so as to ensure that the client stably provides a service to the user, based on which, in the process of obtaining an update client service module, the backup client service module corresponding to the initial client service module may be deleted, and then the update client service module is loaded, and specifically, before sending the received data processing request to the update client service module, the method further includes:

the client management module determines a backup client service module corresponding to the initial client service module;

and deleting the backup client service module, and loading the updated client service module through a version control mechanism.

In practical applications, the client management module will maintain two client instances, one as a primary client instance, i.e. the instance currently being served, and the other as a backup client instance. Therefore, when the hot upgrade is performed, the client management module firstly destructs the local standby client instance (that is, deletes the standby client instance), and then loads the new client dynamic library to generate the new version client instance. Based on this, the initial client service module can be understood as a main client instance of the old version client maintained by the client management module; the backup client service module may be understood as a backup client instance of an old version client maintained by the client management module.

The version control mechanism may be any mechanism that avoids Symbol collision between two versions of client dynamic libraries, which is not specifically limited in this specification, for example, the version control mechanism may be Symbol version. In practical application, in order to avoid the problem of Symbol collision between two versions of client dynamic libraries, symbol version control can be performed on symbols of the dynamic libraries in a Symbol version control mode, and symbols of clients of different versions should be different.

Specifically, after obtaining the updated client service module, the client management module determines an initial client service module maintained by the client management module and a backup client service module corresponding to the initial client service module; and then deleting the backup client service module, and loading the updated client service module through a version control mechanism, so that the initial client service module can be conveniently subjected to hot upgrade based on the updated client service module.

Step 304: and the update client service module acquires unit information of the target data processing unit from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, and calls the target data processing unit to process the data to be processed based on the unit information.

Specifically, in a case where it is determined that there is no target data processing unit for processing data to be processed, the update client service module can obtain unit information of the target data processing unit from the initial client service module, and call the target data processing unit based on the unit information, it should be noted that the data processing unit may be stored in a specific storage medium (for example, the data processing unit may be stored in a storage medium such as a local disk, a cache, or a memory of the user terminal), and the update client service module can call the corresponding target data processing unit by using the unit information. Then, the update client service module processes the data to be processed based on the target data processing unit.

In the above example, the unit information may be file stream information, and the target data processing unit is a file stream class for processing information to be processed; based on this, the new version client instance can obtain the file stream information corresponding to the file stream class from the old version client instance under the condition that the new version client instance is determined not to have the file stream class for processing the information to be processed, call the file stream class based on the file stream information, instantiate the file stream type, obtain the instantiated file stream, and read the information to be processed based on the instantiated file stream.

Further, in an embodiment provided in this specification, the determining that there is no target data processing unit that processes the data to be processed includes:

the update client service module determines unit information of a data processing unit and attribute information of the data to be processed, wherein the unit information is information acquired by the update client service module from the initial client service module;

determining that there is no target data processing unit that processes the data to be processed, in a case where the unit information does not match the attribute information.

The attribute information of the to-be-processed data may be understood as a data identifier corresponding to the to-be-processed data, and the data identifier may be understood as information uniquely identifying one data, such as a data name, a data number, and the like. In practical applications, the data to be processed may be a file to be processed, and based on this, the attribute information of the data to be processed may be understood as a file identifier corresponding to the file to be processed, and the file identifier may be understood as information that uniquely identifies a file, such as a file name, a file number, and the like.

Specifically, after determining the data to be processed, the update client service module can determine attribute information of the data to be processed, and determine attribute information of each data processing unit from the own holding unit information, where the attribute information of the data processing unit can be understood as the attribute information of the data to be processed corresponding to the data processing unit.

And matching the attribute information of the data processing unit with the attribute information of the data to be processed, and under the condition that the matching result is inconsistent, updating the client service module to determine that a target data processing unit for processing the data to be processed does not exist.

In one embodiment provided by the present specification, in order to enable information exchange and synchronization between the new client and the old client, a series of functions may be defined in the client, and a function pointer of the old client may be passed to the new client for calling. And when the generation and initialization of the new version client side instance are finished and the function pointer of the old client side is transferred to the new client side, the main and standby instances can be exchanged, and the new client side instance provides service for the user and receives subsequent requests. When the new version client receives the user request, but the new client does not locally have the file flow information corresponding to the data to be processed carried in the request, a function a which is provided by the old client and can acquire the specified file flow information can be called to acquire the specified file flow information, so that the new version client can timely process the request. Specifically, the obtaining the target unit information from the initial client service module includes:

and the update client service module acquires the target unit information from the initial client service module by calling an information acquisition function provided by the initial client service module.

The information obtaining function may be understood as a function capable of obtaining the specified unit information from the initial client service module.

Specifically, the update client service module can obtain the target unit information from the initial client service module by calling an information obtaining function provided by the initial client service module. Further, in an embodiment provided in this specification, the updating client service module, by calling an information obtaining function provided by the initial client service module, obtains target unit information from the initial client service module, and includes:

the update client service module determines an information acquisition function provided by the initial client service module, and sends the attribute information of the data to be processed to the initial client service module by calling the information acquisition function;

the initial client service module determines target unit information corresponding to the attribute information from unit information held by the initial client service module and sends the target unit information to the update client service module;

and the update client service module receives the target unit information sent by the initial client service module.

Specifically, when it is determined that there is no target data processing unit for processing the data to be processed locally, the update client service module determines a function pointer corresponding to the information obtaining function, where the function pointer is sent to the update client service module by the initial client service module.

And then, calling the information acquisition function based on the function pointer, and sending the attribute information of the data to be processed to the initial client service module.

After receiving the attribute information, the initial client service module can determine unit information held by itself, and determine target unit information corresponding to the attribute information from the unit information, for example, determine a file identifier of a file (i.e., the to-be-processed data) corresponding to each file stream from file stream metadata, and match the file identifier with the received file identifier. And sending the matched and consistent file stream information to the update client service module.

And finally, the update client service module receives the target unit information sent by the initial client service module.

Further, in the embodiments provided in this specification, in the process of unit information transmission between the update client service module and the initial client service module, in order to solve the problem of compatibility between the new and old clients, the old client may serialize the file stream information into a binary stream and send the binary stream to the new client, and the new client performs deserialization on the binary stream, thereby obtaining the file stream information and avoiding the problem of compatibility between the new and old clients. Specifically, the sending the target unit information to the update client service module includes:

the initial client service module carries out serialization processing on target unit information held by the initial client service module to obtain a binary stream corresponding to the target unit information, and sends the binary stream corresponding to the target unit information to the updating client service module;

correspondingly, the updating client service module receives the target unit information sent by the initial client service module, and includes:

and the update client service module performs deserialization processing on the binary stream corresponding to the received target unit information to obtain the target unit information.

Specifically, after determining the target unit information, the initial client service module performs serialization processing on the target unit information held by the initial client service module, and obtains a binary stream corresponding to the target unit information. In the process of serializing the target unit information, any one of the manners of serializing the unit information may be adopted, and this is not particularly limited in this specification.

And then, the initial client service module sends the binary stream corresponding to the target unit information to the update client service module.

And after receiving the binary stream, the update client service module performs deserialization processing on the binary stream corresponding to the target unit information, so as to obtain the target unit information.

In practical applications, in order for a new client to obtain metadata information (i.e., file flow information) about a file flow in an old client so as to continue a service, two functions need to be defined in the client to synchronize the file flow information, wherein the first function is to obtain specified file flow information, i.e., an information obtaining function; and the second function is a function of acquiring a file stream in the old client that is not yet synchronized to the new client, i.e., an information synchronization function. The new client may synchronize the file stream information from the old client by calling the information synchronization function, and specifically, the update client service module further includes, before acquiring the unit information of the target data processing unit from the initial client service module, in a case that it is determined that the target data processing unit for processing the data to be processed does not exist, the update client service module:

and the update client service module acquires the unit information of the data processing unit from the initial client service module by calling an information synchronization function provided by the initial client service module.

Specifically, the updating client service module obtains the unit information of the data processing unit from the initial client service module by calling an information synchronization function provided by the initial client service module, and includes:

the update client service module determines an information synchronization function provided by the initial client service module and sends an information synchronization request to the initial client service module by calling the information synchronization function;

and the initial client service module responds to the information synchronization request and synchronizes the unit information of the data processing unit to the update client service module.

Specifically, before the update client service module processes the data to be processed, a function pointer corresponding to the information synchronization function is determined, where the function pointer is sent to the update client service module by the initial client service module.

And then, calling the information synchronization function based on the function pointer, and sending an information synchronization request to the initial client service module through the information synchronization function, wherein the information synchronization request is a request for requesting the initial client service module to send local unit information to the updated client service module.

And after receiving the information synchronization request, the initial client service module can send the unit information held by the initial client service module to the updating client service module in response to the information synchronization request.

And finally, the update client service module can acquire the unit information held by the initial client service module. Therefore, the hot upgrade aiming at the initial client is realized, the data to be processed can be processed in the process of upgrading the client, and the data processing efficiency is improved.

Further, in the embodiments provided in this specification, in order to handle the compatibility problem between the new and old clients, the old client may serialize the file stream information into a binary stream and send the binary stream to the new client, and the new client performs deserialization on the binary stream, so as to obtain the file stream information and avoid the compatibility problem between the new and old clients. Specifically, the synchronizing unit information of the data processing unit to the update client service module includes:

the initial client service module carries out serialization processing on the unit information of the data processing unit to obtain a binary stream corresponding to the unit information, and sends the binary stream corresponding to the unit information to the update client service module;

and the update client service module performs deserialization processing on the binary stream corresponding to the received unit information to obtain the unit information of the data processing unit.

Specifically, the initial client service module performs serialization processing on unit information of a data processing unit held by the initial client service module to obtain a binary stream corresponding to the unit information, and sends the binary stream corresponding to the unit information to the update client service module;

after receiving the binary stream corresponding to the unit information, the update client service module performs deserialization processing on the binary stream to obtain the unit information of the data processing unit.

It should be noted that, in order to ensure the newest of the file stream information, the old client needs to wait for all ongoing requests of a certain file stream to be completed when synchronizing information of the file stream, and the new client needs to call the information synchronization function continuously in the background to synchronize other file stream information until all file stream information is synchronized.

In an embodiment provided in this specification, when the old client destructs, it needs to wait for all file streams to be migrated, and the new client does not call the function of the old client any more. For this purpose, an information synchronization completion function is defined in the client to notify that the file stream migration is completed, and the client can destruct. And when all the file streams returned by the information synchronization function are synchronously completed and no ongoing information acquisition function exists, the new client can call the information synchronization completion function of the old client. At this time, the hot upgrade is completely finished, and the client management component may perform destruct on the old client, specifically, the data processing method further includes:

the update client service module determines the current transmission states of the unit information and the target unit information, calls an information synchronization completion function provided by the initial client service module and sends an information synchronization completion instruction to the initial client service module under the condition that the current transmission state meets a transmission completion condition;

the initial client service module determines that the information synchronization is completed based on the information synchronization completion instruction and sends a client deletion notification to the client management module;

and the client management module responds to the client deletion notification to delete the initial client service module.

The current transmission state of the unit information may be understood as information indicating whether the unit information is still being transmitted synchronously, and the current transmission state of the target unit information may be understood as information indicating whether the target unit information is being transmitted. The current transmission state satisfying the transmission completion condition may be understood as having completed the information transmission of the unit information as well as the target unit information.

Wherein the information synchronization completion instruction may be understood as a function indicating that the initial client data has been completed by a synchronized transfer.

In the above example, when the new version client instance determines that the file stream information is synchronized and no file stream information needs to be acquired from the old client instance, the information synchronization completion function provided by the old version client is called, and an information synchronization completion instruction is sent to the old version client.

And after receiving the information synchronization completion instruction, the old version client instance determines that the information synchronization is completed, and sends an instance destructor instruction to the client management component, and the client management component destructs the old version client instance based on the instance destructor instruction.

It should be noted that, after the destructuring of the old version client instance is completed, the client management component determines the new version client instance as the main instance, and creates a corresponding standby instance for the new version client instance, thereby ensuring the security of the client instance.

The data processing method provided by the specification includes the steps that firstly, an updating client service module corresponding to an initial client service module is obtained through a client management module, so that the initial client service module is updated based on the updating client service module, then a received data processing request is sent to the updating client service module to be processed, and the updating client service module calls a target data processing unit to process data to be processed through unit information of the target data processing unit obtained from the initial client service module under the condition that the target data processing unit for processing the data to be processed does not exist, so that the data to be processed can be processed in the client updating process, the data processing efficiency is improved, and the use experience of a user is improved.

The following description further describes the data processing method by taking an application of the data processing method provided in this specification in an online hot upgrade scenario for a client as an example, with reference to fig. 4. Fig. 4 shows a flowchart of a processing procedure of a data processing method according to an embodiment of the present specification, which specifically includes the following steps.

Step 402: and the client management component downloads the client dynamic library of the new version from the central management node.

Step 404: and the client management component deletes the standby client of the old version client and loads the new client instance.

Specifically, a client management component running in a user process may maintain two client instances, one as a primary instance, i.e., the instance currently being serviced, and the other as a backup instance.

Based on this, after the client management component downloads the client dynamic library (version 2) of the new version, the local standby instance is firstly destructed, then the new client dynamic library is loaded, the new dynamic library (namely the new version client) is used for re-instantiation, the standby client instance generated by the new version client is obtained, and the standby client instance is initialized.

Step 406: the old client instance sends its own function pointer to the new client instance.

The function pointer comprises a function A, a function B and a function C.

The function A is a function for acquiring appointed file stream information from an old client instance; the function B is a function for acquiring the file stream information which is not synchronized to the new client in the old client; function C is a function indicating that the old client instance filestream information has been completed synchronously.

Step 408: the new client instance calls function B to obtain the file flow information from the old client instance.

Specifically, the new client instance calls the function B through a function pointer corresponding to the function B, and sends the information synchronization request to the old client instance;

the old client instance can synchronously transmit the file stream information held by the old client instance to the new client instance after receiving the information synchronization request.

It should be noted that, when synchronizing information of a certain file stream, the old client needs to wait for all ongoing requests of the file stream to end. Moreover, in order to solve the compatibility problem between the new client and the old client, the old client can serialize the file stream information into a binary stream, and the new client performs deserialization.

Step 410: and the client management component sends a data write-in request which is sent by the user and carries the data to be written to the new client instance.

Step 412: the new client instance determines whether the new client instance itself has file stream information for processing the file stream of the data to be written, if so, step 414 is executed, and if not, step 406 is executed.

Specifically, the new version client instance determines the data to be written carried in the data writing request and a file identifier A1 to which the data to be written needs to be written.

Then, the new version client instance determines whether file stream information corresponding to the file identifier A1 exists from file stream information held by the new version client instance based on the file identifier A1, if so, step 414 is executed, and if not, step 416 is executed.

Step 414: and the new client instance calls the file stream class to process the data to be processed.

Specifically, the new client calls a file stream class corresponding to the file stream information, instantiates the file stream class, and processes the data to be processed based on the file stream obtained after instantiation.

Step 416: the new client instance obtains file stream data from the old client instance based on function a.

Specifically, when a user request comes, if the new client does not have the information of the file stream locally, the function a of the old client is called to synchronize, meanwhile, the request of the file stream is suspended, and after the information stream is synchronized, the suspended requests are sequentially sent.

The method for synchronizing the file stream information by calling the function A of the old client comprises the following steps:

the new client instance calls the function A based on the function pointer corresponding to the function A, and sends the file identification corresponding to the data to be written to the old client instance;

and the old client instance determines the file flow information corresponding to the file identification from the old client instance and sends the file flow information to the new client instance.

Step 418: and the new client instance sends file stream information based on the old client and calls the file stream class to process the data to be processed.

Step 420: and after the new client instance determines that the information synchronization is completed, calling a function C to notify the old client instance that the information synchronization is completed.

Specifically, the new client instance calls the function C based on the function pointer corresponding to the function C, and sends an information synchronization completion notification to the old client instance.

Step 422: and the old client instance determines that the information synchronization is completed based on the information synchronization completion notification, and sends an instance destruct instruction to the client management component.

Step 424: and the client management component destructs the old client instance based on the instance destruct instruction.

The data processing method provided by the specification realizes online non-sensible heat upgrade of the distributed file system client by simultaneously loading the client dynamic libraries of two versions and synchronizing file stream information of the new client and the old client in the memory. And the client dynamic libraries of the two versions are loaded simultaneously by using Symbol version, so that the hot upgrade without service interruption can be realized, and the performance loss caused by a double-process mode is avoided. And the client dynamic libraries of the two versions synchronize file stream information in the memory, so that the new client and the old client can be seamlessly connected, and the influence of hot upgrade on the service delay of the user is reduced to the minimum.

Corresponding to the above method embodiment, this specification further provides a data processing system embodiment, and fig. 5 shows a schematic structural diagram of data processing provided by an embodiment of this specification. As shown in fig. 5, the system includes a client administration module 502, an initial client service module 506, and an update client service module 504, wherein,

the client management module 502 is configured to respond to an update request, obtain the update client service module 504, and send a received data processing request to the update client service module 504, where the data processing request carries data to be processed;

the update client service module 504 is configured to, in a case that it is determined that there is no target data processing unit that processes the to-be-processed data, obtain unit information of the target data processing unit from the initial client service module 506, and call the target data processing unit to process the to-be-processed data based on the unit information.

Optionally, the client management module 502 is further configured to:

receiving an update request for the initial client service module 506 sent by a client management node, where the update request carries a module identifier of the update client service module 504;

the update client service module 504 is obtained from the client management node based on the module identification.

Optionally, the client management module 502 is further configured to:

and receiving a data processing request sent by a user, and sending the data processing request to the update client service module 504 for processing.

Optionally, the client management module 502 is further configured to:

determining a backup client service module corresponding to the initial client service module 506;

the backup client service module is deleted and the update client service module 504 is loaded via a version control mechanism.

Optionally, the update client service module 504 is further configured to:

determining unit information of a data processing unit and attribute information of the data to be processed, wherein the unit information is information acquired by the update client service module 504 from the initial client service module 506;

Optionally, the update client service module 504 is further configured to:

the unit information of the data processing unit is acquired from the initial client service module 506 by calling an information synchronization function provided by the initial client service module 506.

Optionally, the update client service module 504 is further configured to:

determining an information synchronization function provided by the initial client service module 506, and sending an information synchronization request to the initial client service module 506 by calling the information synchronization function;

the initial client service module 506, in response to the information synchronization request, synchronizes the unit information of the data processing unit to the update client service module 504.

Optionally, the initial client service module 506 is further configured to:

serializing unit information of a data processing unit to obtain a binary stream corresponding to the unit information, and sending the binary stream corresponding to the unit information to the update client service module 504;

the update client service module 504 performs deserialization processing on the binary stream corresponding to the received unit information to obtain the unit information of the data processing unit.

Optionally, the update client service module 504 is further configured to:

the target unit information is obtained from the initial client service module 506 by calling an information obtaining function provided by the initial client service module 506.

Optionally, the update client service module 504 is further configured to:

determining an information obtaining function provided by the initial client service module 506, and sending the attribute information of the data to be processed to the initial client service module 506 by calling the information obtaining function;

the initial client service module 506 determines target unit information corresponding to the attribute information from unit information held by the initial client service module, and sends the target unit information to the update client service module 504;

the update client service module 504 receives the target unit information sent by the initial client service module 506.

Optionally, the initial client service module 506 is further configured to:

serializing target unit information held by the client side to obtain a binary stream corresponding to the target unit information, and sending the binary stream corresponding to the target unit information to the update client side service module 504;

accordingly, the update client service module 504 is further configured to:

the update client service module 504 performs deserialization processing on the binary stream corresponding to the received target unit information to obtain the target unit information.

Optionally, the update client service module 504 is further configured to:

determining the current transmission states of the unit information and the target unit information, calling the initial client service module 506 to provide an information synchronization completion function and sending an information synchronization completion instruction to the initial client service module 506 under the condition that the current transmission states meet transmission completion conditions;

the initial client service module 506, further configured to:

determining that the information synchronization is completed based on the information synchronization completion instruction, and sending a client deletion notification to the client management module 502;

the client management module 502, further configured to:

in response to the client delete notification, the initial client service module 506 is deleted.

In the data processing system provided by the present specification, first, an update client service module corresponding to an initial client service module is obtained through a client management module, so that the initial client service module is upgraded based on the update client service module, then, a received data processing request is sent to the update client service module for processing, and the update client service module calls a target data processing unit to process data to be processed through unit information of the target data processing unit obtained from the initial client service module under the condition that it is determined that there is no target data processing unit to process the data to be processed, so that in the process of upgrading the client, the data to be processed can be processed as well, the data processing efficiency is improved, and the user experience is improved.

The above is a schematic scheme of a data processing system of the present embodiment. It should be noted that the technical solution of the data processing system and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the data processing system can be referred to the description of the technical solution of the data processing method.

FIG. 6 illustrates a block diagram of a computing device 600 provided in accordance with one embodiment of the present description. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is coupled to the memory 610 via a bus 630 and a database 650 is used to store data.

Computing device 600 also includes access device 640, access device 640 enabling computing device 600 to communicate via one or more networks 660. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 640 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 600, as well as other components not shown in FIG. 6, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device structure shown in FIG. 6 is for illustration purposes only and is not intended to limit the scope of the present description. Other components may be added or replaced as desired by those skilled in the art.

Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 600 may also be a mobile or stationary server.

Wherein the processor 620 is configured to execute computer-executable instructions that, when executed by the processor 620, implement the steps of the data processing method applied to the client management module, the initial client service module or the update client service module described above.

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which when executed by a processor, implement the steps of the data processing method applied to the client management module, the initial client service module or the update client service module.

The above is an illustrative scheme of a computer-readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the data processing method applied to the client management module, the initial client service module or the updated client service module.

The above is an illustrative scheme of a computer program of the present embodiment. It should be noted that the technical solution of the computer program and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the data processing method.

The foregoing description of specific embodiments has been presented for purposes of illustration and description. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer-readable medium may contain suitable additions or subtractions depending on the requirements of legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer-readable media may not include electrical carrier signals or telecommunication signals in accordance with legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method is applied to a data processing system, the system comprises a client management module, an initial client service module and an updated client service module, and the method comprises the following steps:

2. The data processing method of claim 1, wherein the client management module, in response to the update request, obtains the update client service module, and comprises:

3. The data processing method of claim 2, wherein sending the received data processing request to the update client service module comprises:

4. The data processing method of claim 2, before sending the received data processing request to the update client service module, further comprising:

and deleting the backup client service module, and loading the update client service module through a version control mechanism.

5. The data processing method of claim 1, the determining that there is no target data processing unit to process the data to be processed, comprising:

and determining that there is no target data processing unit for processing the data to be processed in the case that the unit information does not match the attribute information.

6. The data processing method of claim 5, wherein the updating client service module, in the case that it is determined that there is no target data processing unit for processing the data to be processed, before acquiring unit information of the target data processing unit from the initial client service module, further comprises:

7. The data processing method of claim 6, wherein the updating client service module, by calling an information synchronization function provided by the initial client service module, obtains unit information of the data processing unit from the initial client service module, and includes:

the updating client service module determines an information synchronization function provided by the initial client service module and sends an information synchronization request to the initial client service module by calling the information synchronization function;

8. The data processing method of claim 7, wherein synchronizing unit information of data processing units to the update client service module comprises:

9. The data processing method of claim 1, wherein the obtaining the target unit information from an initial client service module comprises:

10. The data processing method of claim 9, wherein the updating client service module obtains the target unit information from the initial client service module by calling an information obtaining function provided by the initial client service module, and the method comprises:

the updating client service module determines an information acquisition function provided by the initial client service module, and sends the attribute information of the data to be processed to the initial client service module by calling the information acquisition function;

11. The data processing method of claim 10, the sending the target unit information to the update client service module, comprising:

the initial client service module carries out serialization processing on target unit information held by the initial client service module to obtain a binary stream corresponding to the target unit information, and sends the binary stream corresponding to the target unit information to the update client service module;

12. The data processing method of claim 1, further comprising:

13. A computing device, comprising:

a memory and a processor;

the memory is adapted to store computer-executable instructions for execution by the processor, which computer-executable instructions, when executed by the processor, implement the steps of the data processing method of any one of claims 1 to 12 applied to a client management module, an initial client service module or an update client service module.

14. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the data processing method of any one of claims 1 to 12 as applied to a client management module, an initial client service module or an update client service module.