CN115988089A - Data processing method, scheduler and client device - Google Patents

Abstract

The embodiment of the invention provides a data processing method, a scheduler and client equipment, wherein the method comprises the following steps: the scheduler responds to a data processing request sent by a target client, and obtains domain names of a plurality of target storage points meeting preset constraint conditions according to position information of the target client in the data processing request, and the scheduler sends data processing information including all the domain names to the target client corresponding to equipment identification according to the equipment identification of the target client in the data processing request, so that the target client executes data processing operation corresponding to the data processing request according to the data processing information. The invention improves the data processing efficiency.

Description

Data processing method, scheduler and client device

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method, a scheduler, and a client device.

Background

Data storage and processing are mainly realized by a server configured with a distributed data storage and processing scheme.

However, with the increase in the amount of data, it is necessary to increase the computing power of the server in order to maintain the efficiency of data storage and processing. The existing method for improving the computing power of the server is mainly to add an edge server or perform fragmented storage on data. However, since operations such as fragmentation and signature verification of existing data are still performed in the server, the computational load of the server is too high when a large amount of data processing traffic is encountered. Therefore, how to improve the data processing efficiency based on the computing power of the existing server has become an urgent problem to be solved.

Disclosure of Invention

Embodiments of the present invention provide a data processing method, a scheduler, and a client device, so as to achieve the purpose of improving data processing efficiency based on the computing power of an existing server. The specific technical scheme is as follows:

a method of data processing, the method comprising:

the method comprises the steps that a dispatcher responds to a data processing request sent by a target client, and domain names of a plurality of target storage points meeting preset constraint conditions are obtained according to position information of the target client in the data processing request;

and the scheduler sends the data processing information comprising the domain names to the target client corresponding to the equipment identification according to the equipment identification of the target client in the data processing request, so that the target client executes the data processing operation corresponding to the data processing request according to the data processing information.

Optionally, the obtaining, according to the location information of the target client in the data processing request, domain names of a plurality of target storage points that satisfy a preset constraint condition includes:

the scheduler responds to a data processing request sent by the target client, calls a preset storage point information table, and performs the following processing on each storage point in the preset storage point information table:

determining a storage point in the preset storage point information table, where a spatial distance between the storage point and the target client is not greater than a preset distance threshold, as an alternative storage point, where the spatial distance is a distance between a first spatial coordinate of the storage point and a second spatial coordinate carried in the location information, and the first spatial coordinate is a parameter in the preset storage point information table;

and processing each alternative storage point as follows:

determining the alternative storage point meeting at least one of a first condition or a second condition as the target storage point meeting the preset condition, and acquiring the domain name of the target storage point, wherein the first condition is that the bandwidth of the alternative storage point is not less than a preset bandwidth threshold, and the second condition is that the occupancy rate of a central processing unit of the alternative storage point is not more than a preset central processing unit occupancy threshold.

Optionally, the data processing request is a fragmented file storage request, and the scheduler sends, according to the device identifier of the target client in the data processing request, data processing information including each domain name to the target client corresponding to the device identifier, so that the target client executes, according to the data processing information, a data processing operation corresponding to the data processing request, including:

and the scheduler calls an interface which has a corresponding relation with the equipment identifier, and sends the data processing information to the target client, so that the target client stores the generated file fragment data groups to target storage points corresponding to the domain names respectively, and the storage operation of the fragment files is finished.

Optionally, the data processing request is a fragmented file loading request, and the scheduler sends, according to the device identifier of the target client in the data processing request, data processing information including each domain name to the target client corresponding to the device identifier, so that the target client executes, according to the data processing information, a data processing operation corresponding to the data processing request, including:

the scheduler searches target fragment index data corresponding to the loading file identification according to the loading file identification in the data processing request;

the scheduler calls an interface which has a corresponding relation with the equipment identifier, sends the data processing information including the domain names and the target fragment index data to the target client, so that the target client analyzes the target fragment index data to obtain fragment identifiers of a target file, and loads fragments of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to a plurality of domain names based on each download rate and each fragment identifier, wherein the download rate is the rate at which the target client loads data from the target storage point.

Optionally, the target client is an authenticated client that stores an authentication identifier indicating that the authentication passes, and the authentication process includes:

the scheduler responds to an authentication request sent by a client to be authenticated and judges whether user information carried in the authentication request is consistent with user verification information or not and whether equipment information carried in the authentication request is consistent with equipment verification information or not;

and when the user information is consistent with the user verification information and the equipment information is consistent with the equipment verification information, determining the client to be authenticated as the target client, and sending the authentication identifier to the target client.

A scheduler, the scheduler comprising:

the domain name acquisition module is used for responding to a data processing request sent by a target client and acquiring domain names of a plurality of target storage points meeting preset constraint conditions according to the position information of the target client in the data processing request;

and the domain name feedback module is used for sending the data processing information comprising each domain name to the target client corresponding to the equipment identifier according to the equipment identifier of the target client in the data processing request so as to enable the target client to execute the data processing operation corresponding to the data processing request according to the data processing information.

Optionally, the domain name obtaining module is configured to:

the scheduler responds to a data processing request sent by the target client, calls a preset storage point information table, and performs the following processing on each storage point in the preset storage point information table:

determining a storage point in the preset storage point information table, where a spatial distance between the storage point and the target client is not greater than a preset distance threshold, as an alternative storage point, where the spatial distance is a distance between a first spatial coordinate of the storage point and a second spatial coordinate carried in the location information, and the first spatial coordinate is a parameter in the preset storage point information table;

and processing each alternative storage point as follows:

determining the alternative storage point meeting at least one of a first condition or a second condition as the target storage point meeting the preset condition, and acquiring the domain name of the target storage point, wherein the first condition is that the bandwidth of the alternative storage point is not less than a preset bandwidth threshold, and the second condition is that the occupancy rate of a central processing unit of the alternative storage point is not more than a preset central processing unit occupancy threshold.

Optionally, the domain name feedback module is configured to:

when the data processing request is a fragmented file storage request, calling an interface corresponding to the equipment identifier, and sending the data processing information to the target client, so that the target client stores the generated file fragment data groups to target storage points corresponding to the domain names respectively, and the fragmented file storage operation is completed.

Optionally, the domain name feedback module is configured to:

when the data processing request is a fragment file loading request, searching target fragment index data corresponding to a loading file identifier according to the loading file identifier in the data processing request;

the scheduler calls an interface which has a corresponding relation with the equipment identifier, sends the data processing information including the domain names and the target fragment index data to the target client, so that the target client analyzes the target fragment index data to obtain fragment identifiers of a target file, and loads fragments of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to a plurality of domain names based on each downloading rate and each fragment identifier, wherein the downloading rate is the rate of the target client loading data from the target storage point.

Optionally, the scheduler further includes: an authentication module configured to:

responding to an authentication request sent by a client to be authenticated, and judging whether user information carried in the authentication request is consistent with user verification information or not and whether equipment information carried in the authentication request is consistent with equipment verification information or not;

and when the user information is consistent with the user verification information and the equipment information is consistent with the equipment verification information, determining the client to be authenticated as the target client, and sending the authentication identifier to the target client.

A method of data processing, the method comprising:

the method comprises the steps that a target client responds to a data processing instruction and sends a data processing request comprising position information of the target client to a scheduler, so that the scheduler responds to the data processing request and obtains domain names of a plurality of target storage points meeting preset constraint conditions according to the position information;

and the target client acquires the data processing information which is sent by the scheduler and comprises the domain names, accesses the target storage point corresponding to at least one domain name in the data processing information, and executes the data processing operation corresponding to the data processing request.

Optionally, the data processing method further includes:

when the data processing request is a fragmented file storage request, the target client reads and fragments a target file to obtain file information and a plurality of file fragments; generating target fragment index data according to each file fragment; and constructing a file fragment data group comprising the file information, each file fragment and the target fragment index data.

Optionally, the data processing method further includes:

when the data processing request is a fragment file loading request, the target client analyzes target fragment index data in the data processing information to obtain each fragment identifier of a target file, and loads each fragment of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to a plurality of domain names based on each downloading rate and each fragment identifier;

and respectively carrying out signature verification on each fragment, and merging each fragment based on the target fragment index data under the condition that each fragment passes the signature verification to obtain a target file.

A client device, the client device comprising:

a request sending module, configured to send, in response to a data processing instruction, a data processing request including location information of the target client to a scheduler, so that the scheduler obtains, in response to the data processing request, domain names of a plurality of target storage points that satisfy a preset constraint condition according to the location information;

and the data processing module is used for acquiring the data processing information which is sent by the scheduler and comprises the domain names, accessing the target storage point corresponding to at least one domain name in the data processing information, and executing the data processing operation corresponding to the data processing request.

According to the data processing method, the scheduler and the client device provided by the embodiment of the invention, the operations such as client authentication, data scheduling, storage point state monitoring and the like can be changed from the execution of the data processing server to the execution of the scheduler by adding the scheduler, so that the storage space and the computing resources of the data processing server are saved. Meanwhile, the scheduler is arranged to send the data processing information to the target client so that the target client executes corresponding data processing operation according to the data processing information, and compared with the prior art, the method and the system do not need the flow of frequent interaction and data verification between the data processing server and the client, and reduce the occupation of non-data processing steps on the memory and the computing resources of the data processing server. Therefore, the invention improves the data processing efficiency.

Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;

fig. 2 is a signaling diagram of a data processing method according to an alternative embodiment of the present invention;

fig. 3 is a signaling diagram of a data processing method according to another alternative embodiment of the present invention;

FIG. 4 is a block diagram of a scheduler provided in accordance with another alternative embodiment of the present invention;

FIG. 5 is a flow chart of a method of data processing provided by another alternative embodiment of the present invention;

FIG. 6 is a block diagram of a client device according to another alternative embodiment of the present invention

Fig. 7 is a block diagram of a data processing apparatus according to another alternative embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

An embodiment of the present invention provides a data processing method, as shown in fig. 1, the data processing method includes:

s101, the dispatcher responds to a data processing request sent by a target client, and domain names of a plurality of target storage points meeting preset constraint conditions are obtained according to position information of the target client in the data processing request.

Optionally, in an optional embodiment of the present invention, the scheduler may be a server configured to perform client authentication, data scheduling, and status monitoring on the storage point. In the existing data processing mode, the operations of client authentication, data scheduling, storage point state monitoring and the like are executed by the data processing server, which occupies the storage space and the computing resources of the data processing server. Therefore, the invention changes the operations of client authentication, data scheduling, storage point state monitoring and the like from the execution of the data processing server to the execution of the scheduler by adding the scheduler, thereby saving the storage space and the computing resources of the data processing server and improving the data processing efficiency.

It should be noted that, in an actual application scenario, the data processing request may be of various types, including but not limited to storing, downloading, and the like of data.

Optionally, in another optional embodiment of the present invention, the location information may represent information of a spatial location of the target client. For example: the IP address of the target client.

It should be noted that, in an actual application scenario, there may be a variety of implementation manners for obtaining domain names of a plurality of target storage points that satisfy preset constraint conditions according to the location information of the target client in the data processing request. For example: the method can be realized by an operation and research optimizer deployed in a scheduler, and the storage point screening is performed by setting the constraint condition of the operation and research optimizer.

It should be noted that, in an actual application scenario, the preset constraint condition may be set according to parameters that affect data processing and transmission efficiency in the historical operation and maintenance data. For example: the space distance between the storage point and the target client, the network transmission efficiency, the storage point memory and the like.

S102, the dispatcher sends the data processing information including the domain names to the target client corresponding to the equipment identification according to the equipment identification of the target client in the data processing request, so that the target client executes the data processing operation corresponding to the data processing request according to the data processing information.

Optionally, in an optional embodiment of the present invention, the device identifier may be a unique identifier that characterizes an identity of the target client. Because the same scheduler can respond to data processing requests of a plurality of different target clients at the same time. Therefore, the scheduler is set to perform data transmission and scheduling operation according to the device identification so as to avoid the risk of data error transmission or error scheduling.

Optionally, in another optional embodiment of the present invention, different data processing requests correspond to different types of data processing operations. Therefore, in addition to the domain name, other data necessary for the data processing operation may be included in the output processing information. For example: for data processing operation with the type of data downloading, the scheduler needs to feed back verification information representing data content, such as data fingerprints or data signatures, to the target client, so that the target client verifies the verification information.

According to the data processing method and the data processing system, the scheduler is arranged to send the data processing information to the target client, so that the target client executes corresponding data processing operation according to the data processing information, compared with the prior art, the data processing method and the data processing system do not need a process of frequent interaction and data verification between the data processing server and the client, occupation of memory and computing resources of the data processing server by non-data processing steps is reduced, and data processing efficiency is improved.

The invention changes the operations of client authentication, data scheduling, memory point state monitoring and the like from the execution of the data processing server to the execution of the scheduler by adding the scheduler, thereby saving the memory space and the computing resources of the data processing server. Meanwhile, the scheduler is arranged to send the data processing information to the target client so that the target client executes corresponding data processing operation according to the data processing information, and compared with the prior art, the method and the system do not need the flow of frequent interaction and data verification between the data processing server and the client, and reduce the occupation of non-data processing steps on the memory and the computing resources of the data processing server. Therefore, the invention improves the data processing efficiency.

Optionally, obtaining domain names of a plurality of target storage points that satisfy preset constraint conditions according to the location information of the target client in the data processing request includes:

the dispatcher responds to a data processing request sent by a target client, calls a preset storage point information table, and performs the following processing on each storage point in the preset storage point information table:

determining a storage point, which is in a preset storage point information table and has a spatial distance with a target client not greater than a preset distance threshold, as an alternative storage point, wherein the spatial distance is a distance between a first spatial coordinate of the storage point and a second spatial coordinate carried in the position information, and the first spatial coordinate is a parameter in the preset storage point information table;

and processing each alternative storage point as follows:

determining the alternative storage points meeting at least one of the first condition or the second condition as target storage points meeting preset conditions, and acquiring domain names of the target storage points, wherein the first condition is that the bandwidth of the alternative storage points is not less than a preset bandwidth threshold, and the second condition is that the occupancy rate of a central processing unit of the alternative storage points is not more than a preset central processing unit occupancy threshold.

Optionally, in an optional embodiment of the present invention, the preset storage point information table may be a data table constructed and maintained by the scheduler, and used for storing the storage point domain name and the storage point information required by the data processing. There are many kinds of storage point information, for example: memory occupancy rate of storage points, bandwidth of storage point outlets, health state parameters of storage points, and the like.

Those skilled in the art will understand that, in an actual application scenario, there may be various embodiments for comparing the spatial distance with the preset distance threshold and determining whether the candidate storage point satisfies at least one of the first condition or the second condition. For example: the system comprises an operation research optimizer configured to obtain an optimal solution of space distance, an operation research optimizer configured to obtain an optimal solution of bandwidth, an operation research optimizer configured to obtain an optimal solution of central processing unit occupancy rate, and the like. The invention does not limit and describe the specific construction mode of the operation and research optimizer.

According to the invention, the storage points in the preset storage point information table, the space distance between which and the target client is not more than the preset distance threshold value, are determined as the alternative storage points, so that a plurality of alternative storage points with the transmission distance meeting the screening condition can be screened out, and the purposes of shortening the data transmission distance and improving the transmission efficiency are achieved. Meanwhile, the bandwidth and the occupancy rate of the central processing unit are used as screening conditions, so that the screening of the alternative storage points with high transmission efficiency and high idle rate of operation resources can be realized, and the data transmission efficiency is also improved.

Optionally, the data processing request is a fragmented file storage request, and the scheduler sends the data processing information including each domain name to the target client corresponding to the device identifier according to the device identifier of the target client in the data processing request, so that the target client executes, according to the data processing information, the data processing operation corresponding to the data processing request, including:

and the scheduler calls an interface which has a corresponding relation with the equipment identifier, and sends the data processing information to the target client, so that the target client stores the generated file fragment data groups to target storage points corresponding to the domain names respectively, and the storage operation of the fragment files is finished.

It should be noted that, in an actual application scenario, the reliability of data storage of a single storage point is affected by the operational reliability of the storage point itself. If the storage point has failures such as a burst, a network break, etc., the risk of data loss or transmission failure may result. Therefore, the invention stores the same data by setting a plurality of target storage points, increases the data backup redundancy and improves the reliability of data transmission and storage.

Optionally, in an optional embodiment of the present invention, since in the process that the target client stores the file fragment data group to each target storage point, the target storage point has a risk of failure. Therefore, in order to improve the data storage reliability, the target client may be configured to send a request for reallocating the storage points to the scheduler after detecting that the data storage fails, so that the scheduler reallocates one storage point to the target client until the data backup quantity meets the quantity preset by the target client. Meanwhile, a timing task can be set for the scheduler to monitor the state of each storage point, the storage points with faults are removed, and the risk of data transmission failure caused by failure of the storage points is reduced.

Optionally, in another optional embodiment of the present invention, if the target storage point fails to transmit data due to failure in the data transmission process, the successfully transmitted part of the data may occupy the storage resource of the target storage point after the target storage point replies, and therefore, in order to save the storage resource, a timing task may be set for the target storage point, so that the storage point periodically cleans incomplete data.

Optionally, the data processing request is a fragment file loading request, and the scheduler sends the data processing information including each domain name to the target client corresponding to the device identifier according to the device identifier of the target client in the data processing request, so that the target client executes a data processing operation corresponding to the data processing request according to the data processing information, including:

the dispatcher searches target fragment index data corresponding to the loading file identifier according to the loading file identifier in the data processing request;

the scheduler calls an interface which has a corresponding relation with the equipment identifier, and sends data processing information comprising each domain name and target fragment index data to a target client so that the target client analyzes the target fragment index data to obtain each fragment identifier of a target file, and loads each fragment of the target file from a target storage point corresponding to one domain name or target storage points corresponding to a plurality of domain names based on each download rate and each fragment identifier, wherein the download rate is the rate at which the target client loads data from the target storage point.

Optionally, in an optional embodiment of the present invention, the target fragment index data is data generated by performing fingerprint calculation on fragmented data by a target client, and after splicing the fragment fingerprints, generating a signature, and represents an arrangement order of the fragment data in a target file. The target fragment index data is sent to the scheduler for storage along with the fragment file storage request when the target client sends the fragment file storage request to the scheduler.

Optionally, in another optional embodiment of the present invention, a specific implementation manner in which the target client parses the target fragment index data to obtain each fragment identifier of the target file, and loads each fragment of the target file from a target storage point corresponding to one domain name or from target storage points corresponding to multiple domain names based on each download rate and each fragment identifier may be that:

and the target client analyzes the target fragment index data by using a preset signature analysis algorithm to obtain each fragment identifier of the target file. The preset signature analysis algorithm may be a Hash (Hash) algorithm.

And the target client selects one or more target storage points according to the download rate of each target storage point.

And if one target storage point is selected, downloading each data fragment from the target storage point in series or in parallel according to each fragment identifier.

And if a plurality of target storage points are selected, grouping the fragment identifiers according to the download rate proportion of each target storage point, and downloading the data fragments in the corresponding fragment identifier group from each target storage point. For example, three storage points, i.e., a storage point A, a storage point B and a storage point C, are set and selected. The download rate ratio of the three storage points is 3:3:4. a total of ten slice designations 1 through 10 are set. Dividing the ten fragment identifications into three fragment identification groups according to the download rate proportion, wherein the first fragment identification group comprises three fragment identifications from 1 to 3. The second group of fragment identities comprises three fragment identities from 4 to 6. The third group of segment identifiers includes four segment identifiers from 7 to 10. And establishing a corresponding relation between the first storage point and the first fragment identification group, establishing a corresponding relation between the second storage point and the second fragment identification group, and establishing a corresponding relation between the third storage point and the third fragment identification group.

And downloading the data fragments in the corresponding fragment identification group from each target storage point.

Optionally, the target client is an authenticated client that stores an authentication identifier representing that the authentication passes, and the authentication process includes:

the scheduler responds to an authentication request sent by a client to be authenticated, and judges whether user information carried in the authentication request is consistent with user verification information or not and whether equipment information carried in the authentication request is consistent with equipment verification information or not;

and when the user information is consistent with the user verification information and the equipment information is consistent with the equipment verification information, determining the client to be authenticated as a target client, and sending the authentication identifier to the target client.

It should be noted that, in an actual application scenario, the above authentication process may be implemented by using a JSON format-based open standard (JSON Web Token, JWT), and since a Token (Token) of the JWT is stateless, a scheduler does not need to allocate a memory for storing state information, which results in resource waste.

It should be noted that, in an actual application scenario, there may be a plurality of specific implementations of the data processing method shown in fig. 1, and two are provided here by way of example:

as shown in fig. 2, which is a signaling diagram of a data processing method, for convenience of description, a data processing request in the data processing method shown in fig. 2 is a fragmented file storage request. The data processing process is completed by a client, a scheduler and a plurality of storage points. The method comprises the following specific steps:

in step S201, the client sends an authentication request to the scheduler. And triggers step S202.

Step S202, the dispatcher responds to the authentication request, and judges whether the user information and the equipment information carried in the authentication request are consistent with the respective corresponding check information. If yes, step S203 is triggered. If not, step S204 is triggered.

Step S203, the scheduler sends the authentication identifier to the client. And triggers step S205.

Step S204, the dispatcher feeds back authentication failure information and ends the process.

In step S205, the client sends a fragmented file storage request to the scheduler. And triggers step S206.

Optionally, in an optional embodiment of the present invention, the fragmented file storage request includes the authentication identifier and location information of the client. After the dispatcher receives the storage request of the fragmented file, whether the request has the authentication identification or not is checked preferentially, and if the request has the authentication identification, the dispatcher responds to the request and executes the subsequent steps.

And S206, the dispatcher responds to the fragment file storage request, calls a first operation and raising optimizer, and screens a plurality of alternative storage points which are not more than a preset distance threshold value from the space distance of the client side from a preset storage point information table. And calling a second operation and research optimizer, and screening a plurality of target storage points meeting at least one of the first condition or the second condition from the alternative storage points. And extracting the domain name of each target storage point. And triggers step S207.

Step S207, the scheduler sends data processing information including the domain name of each target storage point to the client. And triggers step S208.

Step S208, the client sends the file fragment data group to each target storage point corresponding to the domain name of each target storage point.

And step S209, each storage point receives and stores the file fragment data set. And ends the flow.

It should be noted that step S205 and step S206 shown in fig. 2 are alternative embodiments of step S101 shown in fig. 1. Step S207 and step S208 shown in fig. 2 are alternative embodiments of step S102 shown in fig. 1.

Fig. 3 is a signaling diagram of another data processing method, and for convenience of description, the data processing request in the data processing method shown in fig. 3 is a fragmented file loading request. The data processing process is completed by a client, a scheduler and a plurality of storage points. And the client in the process is the client passing the authentication. The method comprises the following specific steps:

step S301, the client sends a fragment file loading request to the scheduler. And triggers step S302.

Step S302, the dispatcher responds to the fragment file storage request and searches the target fragment index data corresponding to the loading file identification in the request. And triggers step S303.

Optionally, in an optional embodiment of the present invention, the fragmented file storage request includes location information of the client and a loaded file identifier.

Step S303, the scheduler calls a first operation and raising optimizer and screens a plurality of alternative storage points which are not more than a preset distance threshold value from the space distance of the client from a preset storage point information table. And calling a second operation and research optimizer, and screening a plurality of target storage points meeting at least one of the first condition or the second condition from the alternative storage points. And extracting the domain name of each target storage point and the corresponding download rate thereof. And triggers step S304.

Optionally, in another optional embodiment of the present invention, the scheduler may obtain other storage point information, such as an occupancy rate of a central processing unit of the storage point, a status identifier of the storage point, and the like, in addition to the domain name and the download rate of each target storage point.

Step S304, the dispatcher sends data processing information including the domain name of each target storage point, the download rate of each target storage point and the target fragment index data to the client. And triggers step S305.

Step S305, the client analyzes the target fragment index data to obtain each fragment identifier of the target file, and determines the corresponding relation between each fragment identifier and each target storage point based on the download rate of each target storage point by using a preset dynamic scheduling algorithm. And triggers step S306.

Step S306, the client sends the corresponding fragment identification to each target storage point corresponding to the domain name of each target storage point. And triggers step S307.

And step S307, calling the fragments corresponding to the fragment identifications by each target storage point according to the received fragment identifications. And triggers step S308.

Step S308, each target storage point sends the called fragment to the client. And triggers step S309.

And step S309, merging the fragments by the client to obtain the original file. And ends the flow.

Correspondingly to the above method embodiment, the present invention further provides a scheduler, as shown in fig. 4, the scheduler includes:

the domain name obtaining module 401 is configured to, in response to a data processing request sent by a target client, obtain domain names of a plurality of target storage points that satisfy a preset constraint condition according to location information of the target client in the data processing request.

And a domain name feedback module 402, configured to send the data processing information including each domain name to the target client corresponding to the device identifier according to the device identifier of the target client in the data processing request, so that the target client performs a data processing operation corresponding to the data processing request according to the data processing information.

Optionally, the domain name obtaining module 401 is configured to:

the dispatcher responds to a data processing request sent by a target client, calls a preset storage point information table, and performs the following processing on each storage point in the preset storage point information table:

determining a storage point, which is in a preset storage point information table and has a spatial distance with a target client not greater than a preset distance threshold, as an alternative storage point, wherein the spatial distance is a distance between a first spatial coordinate of the storage point and a second spatial coordinate carried in the position information, and the first spatial coordinate is a parameter in the preset storage point information table;

and processing each alternative storage point as follows:

and determining the alternative storage point meeting at least one of the first condition or the second condition as a target storage point meeting a preset condition, and acquiring the domain name of the target storage point, wherein the first condition is that the bandwidth of the alternative storage point is not less than a preset bandwidth threshold, and the second condition is that the occupancy rate of a central processing unit of the alternative storage point is not more than a preset central processing unit occupancy threshold.

Optionally, the domain name feedback module 402 is configured to:

when the data processing request is a fragmented file storage request, calling an interface which has a corresponding relation with the equipment identifier, and sending the data processing information to the target client, so that the target client stores the generated file fragmented data groups to target storage points which respectively correspond to the domain names, and the fragmented file storage operation is completed.

Optionally, the domain name feedback module 402 is configured to:

when the data processing request is a fragment file loading request, searching target fragment index data corresponding to a loading file identifier according to the loading file identifier in the data processing request;

calling an interface which has a corresponding relation with the equipment identifier, sending data processing information comprising each domain name and target fragment index data to a target client so as to enable the target client to analyze the target fragment index data to obtain each fragment identifier of a target file, and loading each fragment of the target file from a target storage point corresponding to one domain name or target storage points corresponding to a plurality of domain names based on each download rate and each fragment identifier, wherein the download rate is the rate of loading data from the target storage point by the target client.

Optionally, the scheduler shown in fig. 4 further includes: an authentication module configured to:

responding to an authentication request sent by a client to be authenticated, and judging whether user information carried in the authentication request is consistent with user verification information or not and whether equipment information carried in the authentication request is consistent with equipment verification information or not;

and when the user information is consistent with the user verification information and the equipment information is consistent with the equipment verification information, determining the client to be authenticated as a target client, and sending the authentication identifier to the target client.

An embodiment of the present invention further provides a data processing method, as shown in fig. 5, the method includes:

s501, the target client responds to the data processing instruction and sends a data processing request including the position information of the target client to the scheduler, so that the scheduler responds to the data processing request and obtains domain names of a plurality of target storage points meeting preset constraint conditions according to the position information.

S502, the target client side obtains the data processing information which is sent by the scheduler and comprises the domain names, accesses a target storage point corresponding to at least one domain name in the data processing information, and executes data processing operation corresponding to the data processing request.

Optionally, in an optional embodiment of the present invention, the data processing instruction may be an operation instruction generated by a user through a human-computer interaction interface.

Optionally, the data processing method shown in fig. 5 further includes:

when the data processing request is a fragmented file storage request, a target client reads and fragments a target file to obtain file information and a plurality of file fragments; generating target fragment index data according to each file fragment; and constructing a file fragment data group comprising file information, each file fragment and target fragment index data.

Those skilled in the art can understand that, in an actual application scenario, there are many ways to implement the above fragmentation operation, for example, a hash fragmentation algorithm, a hash modulo fragmentation algorithm, and the like. The present invention does not give redundant details to the specific implementation process of the slicing operation.

Optionally, in an optional embodiment of the present invention, the file information may characterize basic information of the target file, for example: file type, file size, creation time, etc.

Optionally, the data processing method shown in fig. 5 further includes:

when the data processing request is a fragment file loading request, a target client analyzes target fragment index data in the data processing information to obtain each fragment identifier of the target file, and loads each fragment of the target file from a target storage point corresponding to one domain name or target storage points corresponding to a plurality of domain names based on each download rate and each fragment identifier;

and respectively carrying out signature verification on each fragment, and merging the fragments based on the index data of the target fragment under the condition that each fragment passes the signature verification to obtain the target file.

It should be noted that, in an actual application scenario, there are various real-time manners for loading each fragment of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to multiple domain names based on each download rate and each fragment identifier, and an example herein provides one of:

and calling a dynamic scheduling algorithm built in the target client, and adjusting the downloading thread according to the downloading rate and the occupancy rate of a central processing unit in the data processing information. Thereby determining a plurality of target storage points to download the slices.

Optionally, in an optional embodiment of the present invention, during the downloading of each fragment, the client may render part of the downloaded fragments according to an extremely fast preview operation instruction generated by a user through a human-computer interaction interface, and display the rendered part of the downloaded fragments through the human-computer interaction interface.

Corresponding to the method embodiment shown in fig. 5, an embodiment of the present invention further provides a client device, as shown in fig. 6, where the client device includes:

a request sending module 601, configured to send, in response to a data processing instruction, a data processing request including location information of a target client to a scheduler, so that the scheduler obtains, in response to the data processing request, domain names of a plurality of target storage points that satisfy a preset constraint condition according to the location information;

the data processing module 602 is configured to obtain data processing information including each domain name sent by the scheduler, access a target storage point corresponding to at least one domain name in the data processing information, and execute a data processing operation corresponding to the data processing request.

Optionally, the client device shown in fig. 6 further includes: the data generation module is used for reading and fragmenting the target file to obtain file information and a plurality of file fragments when the data processing request is a fragmented file storage request; generating target fragment index data according to each file fragment; and constructing a file fragment data group comprising file information, each file fragment and target fragment index data.

Optionally, the client device shown in fig. 6 further includes: the fragment merging module is used for analyzing target fragment index data in the data processing information to obtain each fragment identifier of the target file when the data processing request is a fragment file loading request, and loading each fragment of the target file from a target storage point corresponding to one domain name or target storage points corresponding to a plurality of domain names based on each downloading rate and each fragment identifier;

and respectively carrying out signature verification on each fragment, and merging the fragments based on the index data of the target fragment under the condition that each fragment passes the signature verification to obtain the target file.

An embodiment of the present invention further provides a data processing apparatus, as shown in fig. 7, where the data processing apparatus includes:

a processor 701;

a memory 702 for storing instructions executable by the processor 701.

Wherein the processor 701 is configured to execute instructions to implement the data processing method as any of the above.

Embodiments of the present invention also provide a computer-readable storage medium, where instructions, when executed by a processor of a data processing system, enable the data processing system to perform any of the data processing methods described above.

The memory may include volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), including at least one memory chip. The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "...," or "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the system embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (10)

1. A method of data processing, the method comprising:

the method comprises the steps that a scheduler responds to a data processing request sent by a target client and obtains domain names of a plurality of target storage points meeting preset constraint conditions according to position information of the target client in the data processing request;

and the scheduler sends the data processing information comprising the domain names to the target client corresponding to the equipment identification according to the equipment identification of the target client in the data processing request, so that the target client executes the data processing operation corresponding to the data processing request according to the data processing information.

2. The method according to claim 1, wherein the obtaining domain names of a plurality of target storage points that satisfy a preset constraint condition according to the location information of the target client in the data processing request comprises:

the scheduler responds to a data processing request sent by the target client, calls a preset storage point information table, and performs the following processing on each storage point in the preset storage point information table:

determining a storage point in the preset storage point information table, where a spatial distance between the storage point and the target client is not greater than a preset distance threshold, as an alternative storage point, where the spatial distance is a distance between a first spatial coordinate of the storage point and a second spatial coordinate carried in the location information, and the first spatial coordinate is a parameter in the preset storage point information table;

and processing each alternative storage point as follows:

determining the alternative storage point meeting at least one of a first condition or a second condition as the target storage point meeting the preset condition, and acquiring the domain name of the target storage point, wherein the first condition is that the bandwidth of the alternative storage point is not less than a preset bandwidth threshold, and the second condition is that the occupancy rate of a central processing unit of the alternative storage point is not more than a preset central processing unit occupancy threshold.

3. The method according to claim 1, wherein the data processing request is a fragmented file storage request, and the scheduler sends data processing information including each domain name to the target client corresponding to the device identifier according to the device identifier of the target client in the data processing request, so that the target client performs a data processing operation corresponding to the data processing request according to the data processing information, including:

and the scheduler calls an interface which has a corresponding relation with the equipment identifier, and sends the data processing information to the target client, so that the target client stores the generated file fragment data groups to target storage points corresponding to the domain names respectively, and the fragment file storage operation is completed.

4. The method according to claim 1, wherein the data processing request is a fragmented file loading request, and the scheduler sends data processing information including each domain name to the target client corresponding to the device identifier according to the device identifier of the target client in the data processing request, so that the target client executes a data processing operation corresponding to the data processing request according to the data processing information, including:

the scheduler searches target fragment index data corresponding to the loading file identifier according to the loading file identifier in the data processing request;

the scheduler calls an interface which has a corresponding relation with the equipment identifier, sends the data processing information including the domain names and the target fragment index data to the target client, so that the target client analyzes the target fragment index data to obtain fragment identifiers of a target file, and loads fragments of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to a plurality of domain names based on each downloading rate and each fragment identifier, wherein the downloading rate is the rate of the target client loading data from the target storage point.

5. The method according to claim 1, wherein the target client is an authenticated client having an authentication identifier stored therein, and the authentication process comprises:

the scheduler responds to an authentication request sent by a client to be authenticated and judges whether user information carried in the authentication request is consistent with user check information or not and whether equipment information carried in the authentication request is consistent with equipment check information or not;

and when the user information is consistent with the user verification information and the equipment information is consistent with the equipment verification information, determining the client to be authenticated as the target client, and sending the authentication identifier to the target client.

6. A scheduler, characterized in that the scheduler comprises:

the domain name acquisition module is used for responding to a data processing request sent by a target client and acquiring domain names of a plurality of target storage points meeting preset constraint conditions according to the position information of the target client in the data processing request;

and the domain name feedback module is used for sending the data processing information comprising each domain name to the target client corresponding to the equipment identifier according to the equipment identifier of the target client in the data processing request, so that the target client executes the data processing operation corresponding to the data processing request according to the data processing information.

7. A method of data processing, the method comprising:

the method comprises the steps that a target client side responds to a data processing instruction and sends a data processing request comprising position information of the target client side to a dispatcher, so that the dispatcher responds to the data processing request and obtains domain names of a plurality of target storage points meeting preset constraint conditions according to the position information;

and the target client acquires the data processing information which is sent by the scheduler and comprises the domain names, accesses the target storage point corresponding to at least one domain name in the data processing information, and executes the data processing operation corresponding to the data processing request.

8. The method of claim 7, further comprising:

when the data processing request is a fragmented file storage request, the target client reads and fragments a target file to obtain file information and a plurality of file fragments; generating target fragment index data according to each file fragment; and constructing a file fragment data group comprising the file information, each file fragment and the target fragment index data.

9. The method of claim 7, further comprising:

when the data processing request is a fragment file loading request, the target client analyzes target fragment index data in the data processing information to obtain each fragment identifier of a target file, and loads each fragment of the target file from the target storage point corresponding to one domain name or the target storage points corresponding to a plurality of domain names based on each download rate and each fragment identifier;

and respectively carrying out signature verification on each fragment, and merging each fragment based on the target fragment index data under the condition that each fragment passes the signature verification to obtain a target file.

10. A client device, the client device comprising:

a request sending module, configured to send, in response to a data processing instruction, a data processing request including location information of the target client to a scheduler, so that the scheduler obtains, in response to the data processing request, domain names of a plurality of target storage points that satisfy a preset constraint condition according to the location information;

a data processing module for obtaining the data processing information including the domain names sent by the scheduler

Accessing the target storage point corresponding to at least one domain name in the data processing information,

and executing a data processing operation corresponding to the data processing request.

Images