IT202100029084A1 - Method and system for high-throughput distributed computing of computational jobs - Google Patents

Description

Title: "Method and system for high-throughput distributed computing of computational jobs"

DESCRIPTION

Field of application

The present invention fits into the sector of distributed computing for the processing of computational jobs.

Background of the invention

In the era of the Internet of Things (IoT) and Big Data, distributed file systems and cloud computing are fundamental elements for data management and processing. Owned global cloud servers of the most? big technology companies are the greatest resource? valuable that we can count on today to perform distributed calculations in outsourcing; edge processing, or edge computing, is has become indispensable since? grows more and more? the volume of data managed by companies for any purpose.

Numerous years of technological advances have paved the way for cloud computing towards Industry 4.0, making it possible to create a wide range of cloud solutions and irreversibly changing the way we look at things. As a result, in the last fifteen years numerous new companies operating in the cloud sector have been born, such as Snowflake, Cloudflare, Databricks and the well-known leaders of the tech sector Google, Microsoft, Amazon, IBM also owe their position as IT giants to introduction of the cloud.

Cloud computing enables the efficient and flexible processing of massive workloads via one or more outsourced servers. In particular, the field of distributed computing studies the coordination of units? networked computing devices placed in different positions, which can jointly perform tasks? of calculations, or tasks, disparate. Finally, the grid computing paradigm extends the concept of distributed computing by admitting heterogeneity? in the composition of networked computers and considering that a large original computational job can? be divided into individual tasks, distributed across the network.

There are some projects today that aim to distribute computational jobs on a grid of common devices, in particular desktop ones, in a grid computing framework. These are intended to exploit the considerable inherent widespread computational power found within common user devices. This power usually remains unused during inactivity. of human owners, for example at night.

Implementing distributed computations on existing, commonly active devices is not ? only an implementation of the grid computing paradigm, but also an ecological alternative to the construction of cloud computing infrastructures such as data centers, as it guarantees highly parallelized and low-intensity calculations. and achieves heat dispersion, without therefore requiring further cooling other than that provided by the environment.

Many well-known operational solutions belong to the category of voluntary computing, in which users make their devices available for hosting intensive external calculations on a voluntary basis. Examples of distributed computing projects are BOINC, Distributed.net, HTCondor, and Folding@Home.

BOINC ? a platform for high-intensity distributed computing, or High-Throughput computing, in which the worker nodes are desktop and laptop computers, tablets and smartphones voluntarily offered by their owners. Numerous projects are connected to BOINC and use its distributed infrastructure. For example: SETI@Home for digital signal processing of radio telescope data; Einstein@Home for searching for faint astrophysical signals from rotating neutron stars; IBM World Community Grid for scientific research on topics related to health, poverty and and sustainability; Climateprediction.net for climate model simulations.

HTCondor ? another open source software for distributed computing capable of increasing computing throughput, developed at the University? of Wisconsin-Madison. HTCondor offers a job queuing mechanism, scheduling policies, a priority scheme, resource monitoring and resource management and can integrate both dedicated resources (rack-mounted clusters) and non-dedicated desktop machines (thanks to Volunteer Computing) into a single computing environment.

Summary of the invention

An aim of the present invention? that of creating an innovative high throughput distributed computing environment, capable of receiving and solving various computational jobs.

Another purpose of the invention? to offer a sustainable alternative to the increased resource consumption of cloud computing.

Yet another purpose of the invention? that of spreading distributed computing across a large number of user devices, such as personal computers, smartphones, tablets, game consoles and smart TVs, maintaining a high level of efficiency.

This and other objectives are achieved thanks to a method and a distributed computing system for the processing of computational jobs in accordance with any of the attached claims.

The invention provides a customer platform where an entity customer can? load job specifications and therefore the related computational job, executable from a grid of user devices. An internal services software system queries user devices to evaluate a grid state. Then, based on the job specifications and grid state, select a job partitioning scheme.

The job data input stream is loaded and divided into pieces, or chunks, of data input, based on partitioning scheme parameters. Input chunks are included in executable activity files, or tasks, that are distributed to user devices based on distribution parameters of the splitting scheme. Since the state of the grid is taken into account when configuring the partitioning scheme, each device will receive an appropriate task for which? It is possible to foresee a pre-established execution and sending time.

Preferably, the subdivision and distribution phases already take place? while loading the input stream. Therefore, isn't it? It is necessary for complete job data to be stably saved in the databases that support the internal services system, but? It is possible to simply temporarily save the input chunks, and part of them can be deleted even before the upload is completed.

User devices receive and execute input chunks and obtain output chunks that are sent back to the internal services system. Advantageous realizations provide solutions aimed at compensating for missing or unusable responses from one or more? user devices, both by previously duplicating the tasks why? are performed by separate devices, or by subsequently assigning uncompleted tasks to other computing devices.

Finally, the output chunks are assembled to obtain a job result, accessible to the client from the platform.

Therefore, the invention makes distributed computing efficient and reliable, with minimal risks of late delivery of the job result. Job specifications for different jobs allow you to assign priorities to the calculation of certain jobs, with an appropriate distribution of tasks to user devices with higher performance? or less high.

Users who offer their capacity? calculations can be rewarded with credits or discounts, to motivate participation. Thanks to the fact that the calculation is not ? concentrated in the central servers, there is a notable increase in heat dissipation, since? each user device ? naturally cooled by the environment. Is this avoided? of having to provide cumbersome cooling systems, which allows for significant improvements to be achieved from a sustainability point of view? economic and environmental.

Brief description of the drawings

Will this invention come? now described in greater detail with reference to the attached drawings, in which some embodiments of the invention are illustrated.

FIG. 1 ? a schematic diagram of a distributed computing system for processing computational jobs in accordance with an embodiment of the invention, FIGS. 2 and 3 are schematic representations of different steps of the processes for dividing job data into input chunks, distributing tasks to user devices, receiving output chunks, and assembling them as a job result, of a distributed computing method for processing computational jobs in accordance with an embodiment of the invention,

FIGS. 4 and 5 are schematic representations of preselection and grouping phases for assigning tasks to user devices, according to the method of figures 2 and 3,

FIG. 6 ? a schematic representation of the assignment of cluster chunk sizes and cluster chunk numbers to clusters, as parameters for the subdivision of job data into input chunks, according to the method of figures 2 and 3, and FIG.7 ? a schematic representation of the generation of cluster task queues and individual device task queues, to assign tasks to user devices, according to the method of Figures 2 and 3.

Detailed description

A distributed computing method for processing computational jobs, and an electronic system 100 configured to execute the method, are described.

System 100 includes client-side software 110, provided to entities customer 200. The entities? customer 200 are subjects interested in having the system 100 process one or more? computational jobs 400, to obtain respective results of job 500.

The client-side software 110 includes a client computing platform 111 for uploading computational jobs 400 by entities. customer 200, and for other functions specified below. More? in detail, the client-side software 110 includes a plurality? of customer software services 112, 113, 114 configured to perform individual platform-related functions.

Each computational job 400 ? composed of job data with a certain job size. The job data may be input to the client platform 111 in the form of an input data stream. To this end, the client-side software 110 includes a job upload service 112 .

Commonly, the entities client 200 provide computational jobs 400 with relatively large job sizes, and consequently receiving individual input streams requires a non-negligible upload time interval.

For reasons that will become clear below, uploading the input stream is ? preceded by the entry of job specifications into the customer platform 111, again by the entities? client 200, the job specifications being uploaded by the job upload service 112. The job specification of a given computational job 400 includes parameters characterizing the respective computational job 400. Preferred examples of such parameters are the job size, the format of input and/or output of the job, the programming language, execution using library algorithms or custom algorithms, the object of the job (such as multimedia operations, tabular data processing, text operations or web testing), the level of priority? execution and geographic limitations in the execution location.

The system 100 also includes a data storage system 120, comprising one or more databases 121, 122, 123, 124, 125, in the same or different locations, some of them being cloud databases in some embodiments. The data storage system 120 ? managed by the owner of the electronic system, and not? intended to be accessible to entities? customer 200 or to entities? user 300, as described below. The job upload service 112 ? configured to upload the input stream and job specifications to the data storage system 120.

An internal services software system 130 runs in one or more? database 121, 122, 123, 124, 125 of the data storage system 120, to receive and manage job data received through the customer platform 111, and to perform other functions illustrated below. The internal services system 130 can? understand a plurality? of internal services 131, 132, 133, 134, 135, 136, 137 with algorithms that perform different functions, among the functions that will be described below.

System 100 includes user-side software 140, provided to entities user 300. The entities? user 300 are subjects who own respective user devices 310, suitable for installing and running the user-side software 140, and who are interested in offering computational power. Examples of user devices 310 include personal computers, smartphones, tablets, game consoles, and smart TVs. Preferably, the computational power is offered in exchange for money, services or other credits according to a pre-established fee schedule.

The user-side software 140 can? understand a device application 141 and more? user services 142134, 142, 143, with algorithms that perform different functions, among the functions that will be described below. The device application 141 ? configured to provide a graphical user interface, and to activate user services 142, 143, 144, where necessary.

The user devices 310 are remote from the storage system 120. When the user devices 310 run the user-side software 140, they are at least periodically in signal communication with the storage system 120. Thus, the user devices 310 collectively form a grid , or grid, in signal communication with the storage system 120. The grid is set up by progressively adding user devices 310.

The internal services system 130 ? configured to query and periodically receive responses from the user devices 310. In particular, these functions are performed by a grid management database 121 of the storage system 120, preferably a Firebase database, where the responses coming from the user devices 310 are then stored .

More? in detail, the user devices 310 are queried about a respective device state. The device status can? be expressed in terms of parameters such as capacity parameters? of calculation, characterizing the maximum potential performance of the user device 310, and availability parameters, characterizing the current and/or historical limits that prevent the use of the maximum potential performance.

Examples of capacity parameters? calculation are the values of RAM, CPU and memory installed. The availability parameters? include the power status of the user device 310 (on/off), and currently available RAM, CPU, storage and bandwidth values, why? not otherwise in use by the user device 310. Other availability parameters? include usage threshold values for RAM, CPU, storage, bandwidth, and time thresholds. Can usage threshold values be set by entities? user 300 so that the capacity? calculation used by the user-side software does not cover the entire capacity? of calculation available. Are the time thresholds one or more? availability time windows? settable by the entity? user 300, possibly with different use threshold values for availability time windows? different.

Other examples of availability parameters? are the remaining battery, the internal temperature, the activity? current of the user device 310, the position of the user device 310 and accelerometer data.

Other availability parameters? do they refer to the efficiency of communication, or on the other hand to limitations in the use of capacity? of calculation due to communication inefficiencies, such as type of network, speed? network, network latency and IP address.

Yet another parameter? related to historical data on the actual computing performance of user devices 310. Performance is evaluated based on the execution of tasks, i.e. packets of executable files and data chunks. As described below, user devices 310 are assigned tasks, generated from chunks of input data, in order to obtain parts of the results of job 500. Tasks distributed to different user devices 310 will generally be different. Thus, in one embodiment, the historical data includes the performance of the user device 310 on one or more of such recent tasks, for example represented by an execution time and an execution completion.

In other implementations, performance is evaluated based on the execution of one or more? recent pre-established test tasks. The test task can? be sent from the storage system 120 to the user device 310 with the status query, without any function of obtaining the result of job 500, but only for the purpose of testing the capacity? of calculation and the availability? of the user device 310, as well as? its calculation precision. The test task can? be the same for all user devices 310, so that their performance tests are comparable.

The user-side software 140 ? configured to receive status queries from the storage system 120, to evaluate at least some of the capacity parameters? of calculation and availability? of the user device 310, and to generate and send a status response that includes the capacity parameters? of calculation and availability? evaluated.

In the preferred embodiments, the state queries include first multiple state queries. frequent, and less frequent second status queries. The first status queries are also called heartbeat queries, which only ask the user device 310 for an available/unavailable response. Second status queries can include requests for part or all of the status. of the parameters described above, i.e. a quantity? of parameters greater than those required for the heartbeat query.

The assessment of capacity? calculation and availability? can? include the execution of the test task, to obtain a chunk of test output data. There? allows the internal system to verify a test execution and delivery time of the test task, and an integrity? o correctness of the test output chunk, i.e. the correspondence between the test output chunk and a pre-established result chunk expected for the test task.

The storage system 120 should theoretically receive status responses from all queried user devices 310 . However, in real applications, status responses are received only from some of the queried user devices 310. In fact, status responses may be missing from turned off user devices 310, from offline user devices 310, and from user devices 310 that for whatever reason do not are able to receive the status query, evaluate the device status, or send the status response.

The internal services system 130 includes a grid management service 131, which is configured to define a grid state based at least on the status responses received, stored in the grid management database 121, preferably based on the status responses received, the response times of the status responses received, any missing status responses, and integrity? of any test output chunk. It is therefore worth noting that some of the capacity parameters? of calculation and availability? may not be determined by the user-side software 140, but by the internal services system 130, based on received/missing status responses, response times and integrity. of test output chunks.

Preferably, defining the grid state includes working out at least some of the capacity parameters? of calculation and availability? of each device state to obtain for each user device 310 one or more? device scores 150. Therefore, defining grid state includes sorting user devices into one or more? grid indices 160 based on respective device scores 150. Distinct device scores 150 may be based on distinct sets of capacity parameters. of calculation and availability.

A first favorite device score 150 ? a capability score? of calculation. The ability score? can? be an increasing function of the availability values? of RAM, CPU and memory and a decreasing function with instability? connection, low battery, high internal temperature and activity? intensity of the device.

A second favorite device score 150? a failure rate score. The failure rate score can be a function of the failure history data of user devices 310. Failure history data may include past events of failure to respond to a task, delays in response to a task, and response with corrupted or incorrect results for a task.

A third favorite device score 150 ? a balancing score, which provides statistically more scores? high to 310 historically least used user devices. The balance score can? be obtained for example by multiplying a randomly generated number with the time interval since the last assignment of a task to the user device.

A fourth favorite score 150? an overall score, which is? a combination of two or more, preferably all of them? of the other 150 device scores, including the three described above, and any other additional 150 device scores that may be expected by one skilled in the art. Preferably, the overall score ? an increasing function of ability scores? and balancing, and a decreasing function with the failure rate score.

As user devices 310 are periodically queried about their status, device scores 150 and grid indices 160 are periodically updated and may vary from query to query. Frequent updates are useful to avoid problems such as an offline state of a user device 310 while ? registered in the grid state as online, and to prevent this? assigning tasks to unavailable user devices 310.

According to one aspect of the invention, the internal services system is configured to select for each computational job 400 a respective job partitioning scheme, preferably different for different computational jobs 400. The job partitioning scheme includes parameters at least for the subdivision of the job data and their distribution to the user devices 310.

The job partitioning scheme, together with the related parameters, are calculated according to the grid state and the job specification. Therefore, different grid states and different job specifications will generally result in different partitioning schemes.

It is worth noting that the job specification ? sufficient for this purpose, even in the absence of complete job data available. So?, since? the job specification input ? prior to that of the job data, the partitioning scheme can? be selected by the internal services system 130 before the upload time interval begins or during the upload time interval, without waiting for the complete input stream of job data to complete.

Subsequently, the internal services system 130 ? configured to split the job data, included in each input stream, into input chunks 410 of data, according to the splitting parameters of the partitioning scheme selected for the computational job 400.

Preferably, the subdivision step is performed in real time, or on-the-fly. In other words, it starts during the upload time interval. Furthermore, each input chunk 410 ? preferably stored in the storage system 120 only temporarily. The input chunks 410 are stored on a queuing database 122 of the storage system 120 (i.e., a first queuing database), which is preferably a Redis database. The queuing database 123 ? configured for short-term storage to support frequent read and write operations.

Input chunks 410 can be used and deleted before the end of the upload time slot, i.e. before the complete job data has been uploaded through the 111 customer platform. Pi? in detail, each input chunk 410 remains stored from a respective saving instant to a respective deletion instant. The saving moment takes place during the upload time interval, since? the instant of saving the last input chunk 410 marks the end of the upload of the input stream. For at least one input chunk 410, preferably for most input chunks 410, depending on the job size, the deletion time also takes place during the upload time interval. Thus, the use of some 410 input chunks, as described below, can be completed even before complete job data has been received.

Advantageously, the queuing database memory 122 can never be occupied by the complete job data of a computational job 400, since? are partially deleted before full reception. There? saves storage space.

The internal services system 130 ? configured to generate one or more? task 420 for each input chunk 410. As described more? forward in more detail, the one or more? task 420 for each input chunk 410 are they a single task 420, or a plurality? of identical tasks 420. Each task 420 includes a respective input chunk 410, and an executable file that includes instructions for performing calculations on the input chunk 410.

The internal services system 130 ? therefore configured to assign and send each task 420 to one or more? respective user devices 310, for the execution of the task 420 by the user devices 310. It should be noted that some assignment sub-phases may involve the interaction of the internal services system 130 with the user devices 310.

The assignment of the tasks 420 involves selecting the user devices 310 based on the distribution parameters of the partitioning scheme of the respective computational job 400.

We will now describe preferred characteristics of the job partitioning scheme, to determine the partitioning and distribution parameters, which are related to each other. The following steps, unless otherwise indicated, are preferably performed by a partitioning service 132 of the internal services system 130.

In some embodiments, selecting the job splitting scheme includes discarding, for the computational job 400, a number of user devices 310, and at least pre-selecting for the computational job 400 the remaining user devices 310. From the following description it will appear it is evident that in the preferred embodiments not all the pre-selected devices 310 will be? necessarily assigned a task 420.

More? in detail, the distribution parameters of the partitioning scheme include one or more? target ranges for respective capacity parameters? of calculation and availability, or combinations thereof. For example, a target range can be applied to one or more? device scores 150 and/or grid indices 160, which are combinations of the capacity parameters? of computation and availability, so that the assignment is based on the device score 150 or a device position in the grid index 160.

Thus, in order to assign the tasks 400 to the user devices 310, the job partitioning scheme provides that the internal services system 130 rejects for the computational job 400 incompatible user devices 310 and at least pre-selects the remaining devices for the computational job 400 compatible user 310, depending on whether their device status has capacity parameters? of calculation and availability? (individually, or combined into device scores 150 and/or grid indices 160) that are incompatible or compatible with one or more? target intervals. Any user device 310 from which a corrupted test output chunk has been received can be automatically discarded as incompatible.

The target intervals are determined according to the job specifications, more? preferably depending on the priority level? of execution, that is? a priority level? desired selected by the customer for job 400. Priority levels can be selected during job specification input. different, which may be available at different prices. 400 computational jobs with priority levels? more? high will result in the selection of a relatively high target range for the overall score or the ability score? computational, while priority levels? more? low will result in the selection of a relatively low target range.

Further target intervals for discarding or pre-selecting user devices 310 can be determined as a function of geographic limitations in the execution location, which can be selected at the input stage of the job specification, for example for national security limitations that may be applicable for particular computational jobs.

In figure 4? shows a pre-selection example, where user devices are sorted by overall device score. A group of user devices 310, indicated overall with 311, is pre-selected based on a target range on the overall device score. Then, some devices in group 311, denoted 312, are discarded based on another criterion, such as a geographic restriction.

In preferred embodiments, selecting the job splitting scheme includes grouping user devices 310 into clusters 320. Clustering preferably follows pre-selection of user devices 310 based on target ranges, as described above. In other words, only the pre-selected user devices 310 are grouped into clusters 320. However, other embodiments may be envisaged in which all user devices 310 are suitable for grouping into clusters, and no pre-selection phase is performed.

For clustering, a cluster number of 320 is selected. This number can? be a fixed number or can? be determined based on job specifications and/or grid status.

Once the cluster number 320 is selected, the user devices 310 are assigned to the clusters 320, preferably based on their device scores 150 and/or based on the grid indices 160. Thus, the assignment of the user devices 310 to different clusters 320 ? ultimately based on the respective capacity parameters? of calculation and availability.

In more detail, each cluster 320 is assigned a range of cluster scores, and user devices 310 are assigned to cluster 320 if their capacity parameters of calculation and availability? are compatible with the range of cluster scores. Similar to target ranges used for pre-selection, cluster score ranges may be ranges of values applied to device scores 150 or device position in grid indices 160, to select which user devices 310 will be included or excluded from 320 different clusters.

In the preferred realization, this assignment ? based on ability score. In any case, since? the pre-selection is based on the overall score, the assignment to clusters 320 is also based at least indirectly on the overall score, and therefore? on any device score 150 used to determine the overall score.

After the clustering step, selecting the job partitioning scheme includes selecting cluster chunk sizes 411 and assigning them to the different clusters 320. The cluster chunk sizes 411 are splitting parameters that are used to split the job data into input chunk 410. Pi? in detail, each cluster chunk size 411 represents a size value of the data that is intended to be included in the input chunks 410 for a cluster 320.

When selecting cluster chunk sizes 411, larger cluster chunk sizes 411 are preferably attributed to clusters 320 having ranges of cluster scores compatible with better capacity parameters. of computation and availability, and smaller cluster 411 chunk sizes are attributed to clusters 320 having cluster score ranges compatible with capacity parameters. of calculation and availability? worse.

When selecting cluster chunk sizes 411, a minimum size is preferably first selected to be attributed to cluster 320 having the highest cluster score range? Bass. Subsequently, the cluster sizes 411 of the remaining clusters 320 are selected larger, preferably equal to multiples of the minimum size, according to respective expansion factors.

The minimum size can? be chosen by a technician in the field in order to balance the need to provide small dimensions, to accelerate the execution of tasks by the user devices 310, with the need? to provide dimensions that are not excessively small, so? to avoid multiplying the number of input chunks 410 and to prevent the onset of communication problems of the internal services system 130 with the grid.

The minimum size can? be fixed, but can? can also be customized, entered as input by the entities? 200 client under job specification.

In preferred embodiments, selecting the job subdivision scheme further includes assigning to the clusters 320 respective numbers of cluster chunks 412, for example in the form of crowding factors, representing the ratio between the number of cluster chunks 412 and the number of user devices 310 of each cluster 320. As described in more detail below, the number of cluster chunks 412 ? a distribution parameter that is used in one or more? phases during the process of assigning tasks 420 to user devices 310. More? in detail, the number of cluster chunks 412 represents a number of input chunks 410 that are intended to be assigned to each cluster 320.

The cluster chunk numbers 412 are selected based on the job specification, and in particular the job size, and preferably also the priority level. Higher numbers of cluster chunks 412 are assigned to all clusters 320 for computational jobs 400 with larger job size, and lower numbers of cluster chunks 412 are assigned to all clusters 320 for computational jobs 400 with smaller job size.

Furthermore, higher numbers of 412 cluster chunks are assigned to 320 clusters with better capacity parameters? of calculation and availability?, for 400 computational jobs with priority levels? more? higher, and lower numbers of 412 cluster chunks are assigned to 320 clusters with better capacity parameters? of calculation and availability?, for 400 computational jobs with priority levels? inferior. Conversely, cluster chunk numbers lower than 412 are assigned to clusters 320 having capacity parameters? of calculation and availability? worse, for computational jobs 400 with priority levels? more? high, and higher numbers of cluster chunks 412 are assigned to clusters 320 having capacity parameters? of calculation and availability? worse, for computational jobs 400 with priority levels? inferior.

Therefore, the job data is divided so as to generate, for each cluster 320, input chunks 410 in a number not greater than, preferably equal to, the respective number of cluster chunks 412, and with a size not greater than, preferably equal to , to the respective cluster chunk size 411, until the job size is reached. In some cases, the actual number and size may be less than the selected cluster chunk number 412 and the cluster chunk size 411, the actual job size being different from the sum of the chunk sizes of all expected chunks.

Preferably, the splitting of the job data includes a pre-splitting phase, in which the input stream is split into intermediate chunks, with a fixed size greater than the selected cluster chunk size 411 for all input chunks 410. Then, in a final splitting step, the intermediate chunks are split into input chunks 410 to obtain the sizes of cluster chunks 411 described above.

At the same time as the progressive generation of the input chunks 410 by subdividing the job data as described, the tasks 420 are generated from the input chunks 410. In some embodiments, only one task 420 is generated from each input chunk 410. Conversely, in the embodiment preferred, select the job partitioning scheme includes selecting one or more? replication parameters, in particular a replication factor representing a ratio between the number of tasks 420 and the number of input chunks 410, although? other similar or equivalent replication parameters may alternatively be selected. ? Only one replication parameter can be selected for all 320 clusters, or different replication parameters can be selected for respective 320 clusters.

The replication factor can be selected equal to one, to indicate that only one task 420 is created for each input chunk 410, or greater than one, to indicate that more are created? task 420 for at least some input chunks 410, i.e. an original task and one or more? task replications. For example, for a replication factor of two, a task replica is created? for each input chunk 410. For a replication factor between one and two, a certain fraction of the input chunks 410 gives rise to respective task replicas, and a complementary fraction of the input chunks 410 does not give rise to any task replicas.

With more replication factors? elevated will come? performed a greater number of identical tasks 410 from distinct user devices 310, as described below. Do they reduce like this? the risks of failure, incorrect or delayed delivery of calculation results. However, more replication factors? high also involve an increase in the total computational power used to execute the computational job 400, for which a technician in the branch will be able to choose a balanced replication factor based on the job specification and grid state.

To increase the replication factor to two, without exhausting the maximum number of concurrent tasks 420 sent to a cluster 320, the crowding factor of each cluster 320 ? preferably selected not exceeding 0.5.

It is worth noting that the distinction between an original task and a corresponding task replica can be purely lexical, as there is no hierarchical distinction inherent in these tasks, which generally have the same content but distinct identifiers.

Therefore, the preferred method for assigning tasks 420 to user devices 320 includes pre-assigning groups of cluster tasks to clusters 320, in the form of cluster queues 421 stored in the queuing database 122. It should be noted that the assignment of tasks 420 to cluster queues 421 is performed progressively, while ? the input flow is being uploaded at the same time as the 410 input chunks are being formed.

Each cluster task group includes all tasks 420 (original and replicas) originating from input chunks 410 having the cluster chunk size 411 attributed to cluster 320.

Preferably, pre-assignment of cluster task groups to clusters 310 ? the last function performed by the partitioning service 132.

A subsequent step of assigning tasks 420 to user devices 310 consists of feeding tasks 420 of each cluster queue 421 into individual device queues 422 of all devices in the cluster. There? is preferably performed by a power service 133 of the internal services system 130. Individual device queues 422 can be stored in a queuing database, which can be identical to the queuing database 122 which stores the cluster queues 421, that is? the first queuing database 122, or a second separate queuing database 123, as illustrated in Figure 1.

In the preferred embodiment, feeding the tasks 420 into the device queues 422 includes shuffling, or shuffling, the tasks 420 of each cluster queue 421 with different orders for the different user devices 310 of the cluster 320. The different orders of the individual device queues 422 can be selected as random orders or with specific shuffling patterns which will make the orders different from each other. Thus, tasks 420 are fed into individual device queues 422 in their respective different orders.

The device queues 422 are sent from the internal services system 130 to the respective user devices 310, which include all the tasks 420 of the queue 422, or simply a list of pointers or indicators referring to the tasks 420. The user-side software 140 includes a task download 142 to download tasks 420 and/or the respective device queue 422.

Individual device queues 422 include task labels, which indicate assignment states of each task. Assignment states include a state that is not yet assigned ("free" in Figure 7) and a state that has already been assigned. assigned (“busy” in Figure 7). In some embodiments, it can? also be expected an assignment state defining a state already? assigned and completed.

Initially, the labels indicate the unassigned status for all tasks. Thus, assigning tasks 420 includes selecting, by an available user device 310 of the cluster 320 running the user-side software 140, a specific task 420, from the relevant individual device queue 422, which has not so far been assigned to another user device 310, and is therefore labeled as not yet assigned. Preferably, the selected task 420 ? the first task 420 in the relevant device queue 422 that has not yet been selected by another user device 310. This will be? different for many user devices 310 thanks to the shuffling phase.

When a user device 310 selects a task 420, the user-side software 140 ? configured to send an assignment signal to the internal services system 130. Therefore, the internal services system 130 is configured to confirm the assignment of the selected specific task 420 to the user device 310. In particular, the internal services system 130 updates the label on an assignment state indicating a state already assigned for that specific task 420, in all individual device queues 422 of cluster 320. As a result, task 420 is not assigned. more? deemed available, and it will not be? selected by other user devices 310 of the cluster 320.

The user-side software 140 ? configured to receive any task 420 assigned to the user device 310, and to execute such tasks 420, executing the executable file of the task 420 on the input chunk 410 of the task 420. The execution of the task 420 generates in the user device 310 an output chunk 430 of the data. The user-side software 140 ? configured to send the output chunk 430 back to the storage system 120. The user-side software 140 includes an output chunk upload service 143 to upload the output chunks 430 to the storage system 120.

Accordingly, the storage system 120 receives different output chunks 430 from different user devices 310 for each computational job 400. The output chunk storage function 430 is preferably performed on a database of buckets 124 of the storage system. storage 120, what? preferably a cloud database. In one embodiment, the bucket database 124 ? configured to temporarily create a virtual storage bucket of each output chunk 430. Thus, the bucket database 124 can? function as a web folder with access restrictions.

It should be noted that the generation of some tasks 420, the delivery of these tasks 420 to the user devices 310, and the reception of some output chunks 430 may take place during the upload time interval, and may continue progressively in a similar manner to as described to split the input flow. There? ? schematically illustrated in Figure 3, where the check mark represents the successful reception of output chunks 430 from some user devices 310, and the hourglass sign represents the pending reception of output chunks 430 from other user devices 310.

In the absence of replicas (or a replication factor of 1), the output chunks 430 will generally be different from each other, since? the tasks 420 originating from a computational job 400 are generally different. Conversely, with more replication factors? high, ? expected that the output chunks 430 originating from each task replica are equal to the output chunks 430 originating from the corresponding original tasks, unless the execution of a task 420 involves a stochastic process.

There? can? however, not be applicable to the case of corrupt output chunks 430, in which data corruption has occurred during transmission to the user device 310 of task 420, during execution of task 420, or during transmission to the storage system 120 of output chunk 430.

In the case of deterministic tasks 420 for which at least one task replica has been generated, and therefore for which more? output chunk 430, the internal services system 130 ? preferably configured to verify integrity? of the output chunk 430 by comparing the output chunks 430 originating from the same input chunk 410.

This function is preferably performed by a validation service 134 of the internal services system 130. In case a mismatch is found, the internal services system 130 ? preferably configured to generate an additional task replica for the input chunk 410 from which corrupt results were received. This further reply can? be sent to, and executed by, a further user device 310, in order to produce a further output chunk 430 for the same input chunk 410, to perform a further comparison on the output chunks 430, and identify the intact output chunks and corrupt 430.

Similarly, further replication can be performed directly by the 134 validation service.

Even in the absence of task replicas, the internal service system 130, in particular the validation service 134, is preferably configured to verify integrity? of the output chunks 430 by comparing chunk parameters, such as output chunk sizes and formats, with corresponding expected parameters. In the event that chunk corruption emerges from this check, corrective actions similar to those described above can be adopted for the case of output chunk mismatch.

Furthermore, the internal service system 130, in particular the validation service 134, is configured to ensure the timely arrival of the 430 output chunks, i.e. arrival before a pre-established maximum time.

Preferably, for activities? replicate 420, ensuring the timely arrival of the output chunks 430 includes ensuring that at least one output chunk 430 has arrived for each input chunk 410. In other words, even if only one output chunk 430 has been received on time although? were originated more? task 420 from the input chunk 410, the reception of any other output chunks 430 for any identical task 420 is not waited beyond the pre-established maximum time, and the overall punctual arrival for that input chunk 410 is confirmed.

There? strongly reduces delivery delays of the complete job result 500 all the more? how high? the replication factor, since? each failure or delayed delivery of an output chunk 430 can? be compensated by the timely arrival of another output chunk 430 originating from the same input chunk 410. ? therefore the risk of simultaneous missed or delayed deliveries of chunks for more is low? user devices 310. However, failure to replicate output chunks 430 can reduce the possibility? of integrity checks.

Punctual arrival can not occur in the event of a delayed arrival, but also in the event of an error during the transmission to the user device 310 of the task 420, during the execution of the task 420, or during the transmission to the storage system 120 of the output chunk 430.

Preferably, for at least some types of failure during the execution of a task 420, the user-side software 140 ? configured to send an error signal to the internal services system 130. Conversely, for other types of failure, for example in the case in which the user device 310 is turned off, the internal services system 130 will not receive? simply no response from the user device 310 that received the task 420.

More? in detail, the verification of the timely arrival of an output chunk 430 from a specific user device 310 can? behave more? phases and more? cases. In a first case, a failure to arrive on time is determined when an error signal is received from the user device 310. In another case, a failure to arrive on time is determined when no status response is received from the user device 310 in response to a status heartbeat query following the delivery of the input chunk 410 to the user device 310. In this case, preferably the failure to arrive on time is confirmed only if the reply from the user device 310 is not received even after a certain time by a confirmation query of the system internal to the user device 310. In yet another case, failure to arrive on time is determined when no output chunk 430 is received until the pre-established maximum time has expired.

The internal service system 130, in particular the validation service 134, is configured to take mitigating actions in the event that no output chunk 430 has been received for a specific input chunk 410 within the pre-established maximum time (including all cases described above). The mitigation actions are similar to those described for the case of corrupt results. In particular, these include reassigning the task 420 relating to the specific input chunk 410, or a copy thereof, to another user device 310 or for execution by the same validation service 134. Preferably, the input chunk delayed or incorrect 410 is reassigned to the validation service 134, so that no further risk is incurred and the overall performance of the computational job 400 is not compromised, but the decision is ? always subject to capacity? current of the storage system 130.

Upon receiving one or more? output chunk 430 for each single input chunk 410 originating from a computational job 400, the internal services system 130 ? configured to assemble these output chunks 430 thereby obtaining job result 500. Part of an assembled job result 500 ? illustrated in Figure 3. This function is preferably performed by an assembly service 135 of the internal services system 130.

Finally, the client-side software 110 ? configured to transmit output to the entities? customer 200 the results of assembled jobs 500, preferably allowing downloading of job results 500 through the customer platform 111. Pi? in detail, the client-side software 110 includes a results download service 113 to download the result of jobs 500 and/or one or more? output chunks 430.

Although? the main components and services that process the computational job 400 have been described so far, further components and services will preferably be provided, as described below.

Preferably, the internal services system 130 includes a reward service 136 configured to apply the pre-established tariff to generate a charge to the entities? customer 200 for processing computational job 400, and to generate a credit to the entities? of user 300 for the execution of the assigned tasks 420.

More? in detail, the rewarding service 136 ? configured to record the debit and credit values in a consistency database 125 of the storage system 120, which is preferably a Structured Query Language (SQL) database configured to support atomic computer transactions.

Furthermore, the customer-side software 110 includes a charging service 114 configured to download the charging values from the storage system 120, and output them through the customer platform 111.

Furthermore, the user-side software 140 includes a crediting service 144 configured to download and output to the user device 320 the crediting values taken from the storage system 120.

Preferably, the internal services system 130 includes a transaction service 137, configured to perform atomic transactions on the consistency database 125. The performed atomic transactions include variable modifications that must be performed simultaneously for all the services of the internal services system 120. In particular , preferred examples of such variable changes include changes in the state of grids, changes in the state of task assignment 420, changes in the state of computational jobs 400, and changes in the state of client platforms 111.

Claims (11)

1. Method for distributed computation of computational jobs (400) composed of job data, comprising: - prepare a data storage system (120) comprising one or more? databases (121, 122, 123, 124, 125), - set a grid of user devices (310) coming from entities? user devices (300), the user devices (310) being remote from the storage system (120) and at least periodically in signal communication with the storage system (120), each user device (310) having a capacity pre-established calculation, - periodically query each user device (310) on a respective device state, in the form of capacity parameters? of calculation and availability, - receive in the storage system (120) status responses from at least some user devices (310), which include their respective device states, and define a grid state based on at least the status responses received , - provide to entities? customers (200) a customer IT platform (111), - receive job specifications, loaded into the storage system (120) through the customer platform (111) from the entities? customer (200), each job specification including parameters characterizing a respective computational job (400), - select for each computational job (400) a respective job partitioning scheme based on the grid status and job specification, - receiving data input streams, each of which includes the job data of a respective computational job (400), and being uploaded through the customer platform (111) by an entity customer (200), - divide the job data included in each input flow into input chunks (410) of data, according to the subdivision parameters of the partitioning scheme selected for the computational job (400), - generate one or more? task (420) for each input chunk (410), each task (420) including the respective input chunk (410) and executable instructions for performing computations on the respective input chunk (410), - assign and send each task (420) to one or more? respective user devices (310), selected based on distribution parameters of the partitioning scheme selected for the respective computational job (400), - receiving in the storage system (120) output chunks (430) of data from user devices (310), each output chunk (430) being obtained from a user device (310) by executing a task (420), - ensure the timely arrival of the output chunks (430), - provide as output to the entities? customer (200) job results (500) through the customer platform (111), each job result (500) including data obtained by assembling the output chunks (430) related to the corresponding computational job (400). 2. Method according to claim 1, wherein: - for each computational job (400), receiving the respective input stream requires an upload time interval, - the steps of splitting job data, generating tasks (420) and sending tasks (420) to user devices (310) start during the upload time interval, - each input chunk (410) remains temporarily stored in the storage system (120), from a respective saving instant which takes place during the upload time interval, to a respective deletion instant, - for at least one input chunk (410), the deletion instant takes place during the upload time interval of the respective computational job (400). 3. Method according to claim 1 or 2, wherein:: - the distribution parameters of the partitioning scheme include one or more? target ranges for respective capacity parameters? of calculation and availability? or combinations thereof, the target intervals being determined according to the job specifications, - the partitioning scheme provides, in order to assign tasks (420) of a computational job (400) to user devices (310), to discard incompatible user devices (310), having device status with capacity parameters? of calculation and availability? incompatible with one or more? target intervals, and to at least pre-select the remaining compatible user devices (310), having device status with capacity parameters? of calculation and availability? compatible with one or more? target intervals. 4. Method according to any of claims 1 to 3, wherein:: - select the job partitioning scheme includes grouping the user devices (310) into clusters (320), each cluster (320) including user devices (310) selected based on values of the respective capacity parameters? of calculation and availability, - assigning tasks (420) to user devices (310) includes: - pre-assign groups of cluster tasks to distinct clusters (320), in the form of cluster queues (421), - feed the tasks (420) of each cluster queue (421) to individual device queues (422) of all user devices (310) of the cluster (320), - select, by an available user device (310) of the cluster (320), a specific task (420) from the respective individual device queue (422), which has not yet been assigned to another user device (310), and confirm the assignment of the selected specific task (420) to said available user device (310). 5. Method according to claim 4, wherein: - feeding the tasks (420) of the cluster queues (421) to the device queues (422) includes a shuffling of the tasks (420) of each cluster queue (421) with different orders, preferably random, for the different user devices ( 310) of the cluster (320), and feed the tasks (420) to the individual device queues (422) in their different orders, - selecting the specific task (420) by the available user device (310) includes selecting the first task (420) in the relevant device queue (422) that has not yet been selected by another user device (310). 6. Method according to claim 4 or 5, wherein: - selecting the job partitioning scheme includes assigning to the clusters (320), as a subdivision parameter, respective different cluster chunk sizes (411), and as a distribution parameter, respective cluster chunk numbers (412), based on the job specifications, preferably where larger cluster chunk sizes (411) are assigned to clusters (320) having more values? high capacity parameters? of computation and availability, and smaller cluster chunk sizes (411) are assigned to clusters (320) having lower values of the capacity parameters? of calculation and availability, - subdividing the job data generates for each cluster (320), and feeds to the relevant cluster queue (421), input chunks (410) in a number no greater than, preferably equal to the respective number of cluster chunks (412), and with a size no greater than, preferably equal to, the respective cluster chunk size (411). 7. Method according to any of claims 1 to 6, wherein:: - the job partitioning scheme selected for each computational job (400) includes one or more? replication parameters, - generate one or more? task (420) for each input chunk (410) of a computational job (400) includes generating more? replicated tasks (420) for at least some input chunks (410), so that the tasks (420) are more? numerous of the input chunks (410) according to the one or more? replication parameters, e - ensuring the timely arrival of the output chunks (430) includes ensuring that at least one output chunk (430) has arrived for each input chunk (410). 8. Method according to any of claims 1 to 7, wherein, after ascertaining the timely arrival of the output chunks (430), in the event that no output chunk (430) has been received for a specific chunk of input (410) within a predetermined maximum time, the method comprises reassigning the task (420) related to the specific input chunk (410) to another user device (310) or to a validation software service (134) running on the storage system (120). 9. Method according to any of claims 1 to 8, wherein:: - defining the grid state includes processing at least the capacity parameters? of calculation and availability? of each device state to obtain for each user device (310) one or more? device scores (150), - defining the grid state includes sorting the user devices (310) into one or more? grid indices (160) based on respective device scores (150), e - said job partitioning scheme distribution parameters are set to assign tasks (420) to user devices (310) based on the positions of the devices in said one or more? grid indices (160). 10. Method according to any of claims 1 to 9, wherein the capacitance parameters? of calculation and availability? of each device state include one or more? Between: - on/off, availability time interval, activity? user device current (310), remaining battery, internal temperature, user device location and/or accelerometer data, - installation values, availability? and/or usage threshold for RAM, CPU, storage and/or bandwidth, - network type, speed, latency and/or IP address, - performance on a recent test task or on a recent (420) task generated from an input chunk (410). 11. Electronic system (100) for the distributed calculation of computational jobs (400) with the method according to any one of claims 1 to 10, comprising: - said data storage system (120), - a client-side software (110) comprising said client platform (111), with algorithms for the input of said job specifications, the input of said data input streams and the output of said job results (500) , - an internal services software system (130), for execution in the data storage system (120), with algorithms for querying the user devices (310) on the device state, receiving said state responses and defining said state of grid, select said job partitioning schemes, divide said job data, generate said tasks (420), assign and send tasks (420) to user devices (310), receive said output chunks (430) from user devices ( 310), ensure the timely arrival of the output chunks (430), and assemble the output chunks (430) into said job results (500), - a user-side software (140), for execution in user devices (310), with algorithms to evaluate the capacity parameters? of calculation and availability? of a user device (310), generate and send said status responses, receive the tasks (420) assigned to the user device (310), execute the tasks (420) to generate said output chunks (430), and send the chunks output (430) to the storage system (120).