RU2829151C2 - Method and system for generating digital task label by machine learning algorithm - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method and a system for generating a digital task label using a machine learning algorithm (MLA). Method is performed by a server connected to the crowdsourced digital platform and includes: at the training stage: receiving by the server a digital training task for execution on the crowdsourced digital platform; receiving, by a server, a plurality of digital training task labels corresponding to the digital training task from a plurality of client devices associated with workers, wherein said plurality of client devices is associated with a crowdsourced digital platform, in response to the digital training task sent to the plurality of client devices using the crowdsourced digital platform; receiving, by the server, a digital mark history containing the digital task marks previously received from each client device; training by the server of the MLA algorithm, which includes: input by the server of the digital training task into the MLA algorithm capable of generating a vector representation of the task corresponding to the vector representation of the digital training task; input by the server of digital labels histories into the MLA algorithm to form a vector representation of the employee with which the given client device is connected from the plurality of client devices; wherein the formation of a vector representation of an employee with which said client device is connected includes: determining for a given worker, with which a given client device is connected, a latent parameter indicating the degree of displacement of the worker in the direction of one or more latent features from the digital training task and determined by analysing the disparity matrix corresponding to the worker, with which said client device is associated; formation of a triplet of training objects containing a vector representation of a task, a vector representation of an employee, with which said client device is associated, and a digital training task label associated with a vector representation of an employee with which said client device is associated; using the training objects triplet to train the MLA algorithm to predict the digital task label for the vector representation of the digital task and the vector representation of the worker with which the client device is connected; at the stage of use: receiving a digital task by a server; determining, by the server, a vector representation of the task for the digital task; predicting, using the MLA algorithm, a plurality of digital task labels for the digital task based on a set of vector representations of workers with which the corresponding client devices are associated, and vector representation of task for digital task; server determines a digital task mark corresponding to at least one digital task mark from a plurality of digital task marks for the digital task.

EFFECT: formation of a more accurate marking of a digital task by the MLA algorithm due to preliminary training of the MLA algorithm using the generated triplet of training objects.

16 cl, 5 dwg

Description

Field of technology to which the invention relates

[001] The present technology relates to methods and systems for generating a digital task label, in particular, to methods and systems for generating a digital task label for a digital task in a crowdsourcing environment.

State of the art

[002] Training machine learning algorithms (MLA) requires large amounts of labeled data. Crowdsourcing platforms such as Amazon Mechanical Turk™ make it possible to obtain large labeled data sets more quickly and at a lower cost than with a limited number of experts.

[003] It is known that the evaluators usually available on crowdsourcing platforms are usually not professionals and have different levels of knowledge and experience, so the resulting markup turns out to be noticeably noisier than the markup provided by experts.

[004] Patent application US 20200327582 A1 "Method and System for Determining Result for Task Executed in Crowd-Sourced Environment" (Yandex Europe AG, published October 15, 2020) describes a system for determining a result for a task executed in a crowdsourcing environment. The method includes: receiving a plurality of task results submitted by a plurality of human evaluators; obtaining a quality score for each human evaluator from the plurality of human evaluators; generating a plurality of vector representations comprising a vector representation for each result; mapping the plurality of vector representations to a vector space; combining the plurality of vector representations into at least a first cluster and a second cluster; executing a machine learning algorithm capable of generating a first confidence parameter and a second confidence parameter; generating a combined vector representation if the first confidence parameter or the second confidence parameter meets a predetermined condition; and selecting the combined vector representation as the task result.

[005] Patent application US 20210133606 A1 "Method and System for Selecting Label from Plurality of Labels for Task in Crowd-Sourced Environment" (Yandex Europe AG, published May 6, 2021) describes a method and system for selecting a label for a task. The method includes: receiving a plurality of labels, each of which indicates a subjective preference of an evaluator for a first object over a second object; analyzing a comparison task to determine a set of latent biasing features; executing an MLA algorithm capable of generating a latent evaluation parameter for the first object and the second object, wherein the latent evaluation parameter indicates a probable bias of the evaluator's subjective preference relative to an unbiased preference parameter of the first object over the second object; generating a predicted degree of bias parameter for the evaluator; generating an unbiased preference parameter; using the unbiased preference parameter as a label in the comparison task for the evaluator by the server.

Disclosure of invention

[006] The aim of the present invention is to provide an improved method and system for forming a mark in a digital problem.

[007] Without wishing to be bound by any particular theory, embodiments of the present technology are developed based on the assumption that a properly trained machine learning algorithm can correctly imitate the label selection process performed by expert evaluators or evaluators with high quality scores in selecting the correct labels.

[008] According to a first aspect of the present technology, a computer method for forming a digital task label by an MLA algorithm is implemented, executed by a server associated with a crowdsourcing digital platform. The method includes: at the training stage - receiving by the server a digital training task for execution on the crowdsourcing digital platform; receiving by the server a plurality of digital training task labels corresponding to the digital training task and sent by a plurality of workers of the crowdsourcing digital platform, wherein the digital training label is sent by the worker in response to the digital training task using the crowdsourcing digital platform; receiving by the server a history of worker actions associated with each worker from the plurality of workers and containing digital task labels previously sent by each worker; training by the server of the MLA algorithm, including: inputting by the server the digital training task into the MLA algorithm, capable of forming a vector representation of the task corresponding to the vector representation of the training digital task; input by the server of the histories of worker actions into the MLA algorithm capable of forming a vector representation of the worker corresponding to the vector representation of the history of worker actions for the worker from a plurality of workers; formation of a triplet of training objects containing a vector representation of the task, a vector representation of the worker and a label of the digital training task associated with the vector representation of the worker; use of the triplet of training objects for training the MLA algorithm to predict the label of the digital task for the vector representation of the task determined for the digital task and the vector representation of the worker; at the stage of use - receiving by the server the digital task; determining by the server the vector representation of the task for the digital task; predicting, using the MLA algorithm, a plurality of digital task labels for the digital task based on the set of vector representations of the workers and the vector representation of the task for the digital task; determining by the server a digital task label corresponding to at least one digital task label from the plurality of digital task labels for the digital task.

[009] In some non-limiting embodiments of the method, determining a digital task label includes performing a majorization based on a plurality of digital task labels for the digital task.

[010] In some non-limiting embodiments of the method, the method further includes determining for each worker of the plurality of workers a quality score corresponding to a previous success rate in providing correct digital task labels, determined based on the history of the worker's actions.

[011] In some non-limiting embodiments of the method, the set of vector representations of workers is a subset of the set of workers that meet a predetermined condition.

[012] In some non-limiting embodiments of the method, the predetermined condition corresponds to a subset of the plurality of workers that includes one or more workers with a previous success rate that exceeds a predetermined threshold.

[013] In some non-limiting embodiments of the method, the digital task is a digital task of the first type, and the predetermined condition corresponds to a subset of the plurality of workers containing one or more workers with a previous success rate exceeding a predetermined threshold for the digital task of the first type.

[014] In some non-limiting embodiments of the method, generating a vector representation of a worker for a worker includes: determining for the worker a latent parameter indicating the degree of bias of the worker in the direction of one or more latent features from a digital training task and determined by analyzing a discrepancy matrix associated with the worker; generating a representation of the worker based on the latent parameter.

[015] In some non-limiting embodiments of the method, generating a vector representation of a task for a training digital task includes: determining for the training digital task one or more latent features that influence the selection of a training label by the worker; generating a vector representation of the task based on one or more latent features.

[016] In some non-limiting embodiments of the method, one or more latent features include at least one of the following: a font size associated with the content of the educational digital task; an image size associated with the content of the educational digital task; a number of selectable labels associated with the educational digital task; a location of selectable labels in the content of the educational digital task.

[017] According to a second aspect of the present technology, a system is implemented for generating a digital task label using an MLA algorithm with a server associated with a crowdsourcing digital platform and containing a processor capable of: at the training stage - receiving a digital learning task to be performed on the crowdsourcing digital platform; receiving a plurality of digital learning task labels corresponding to the digital learning task and sent by a plurality of workers of the crowdsourcing digital platform, wherein the digital learning label is sent by the worker in response to the digital learning task using the crowdsourcing digital platform; receiving a history of worker actions associated with each worker from the plurality of workers and containing digital task labels previously sent by each worker; training the MLA algorithm, for which the processor is capable of: inputting the digital learning task into the MLA algorithm capable of generating a vector representation of the task corresponding to the vector representation of the digital learning task; input the worker action histories into an MLA algorithm capable of generating a vector representation of a worker corresponding to a vector representation of a worker action history for a worker from a plurality of workers; generate a triplet of training objects containing a vector representation of a task, a vector representation of a worker, and a label of a digital training task associated with the vector representation of the worker; use the triplet of training objects to train the MLA algorithm to predict a digital task label for the vector representation of a task determined for the digital task and the vector representation of the worker; at the use stage, obtain a digital task; determine a vector representation of a task for a digital task; predict, by executing the MLA algorithm, a plurality of digital task labels for the digital task based on the set of vector representations of workers and the vector representation of a task for the digital task; determine a digital task label corresponding to at least one digital task label from the plurality of digital task labels for the digital task.

[018] In some non-limiting embodiments of the system, the processor is configured to perform majorization based on a plurality of digital task labels for the digital task for the purpose of determining a digital task label.

[019] In some non-limiting embodiments of the system, the processor is further configured to determine, for each worker of the plurality of workers, a corresponding quality score corresponding to a previous success rate in providing correct digital task labels, determined based on the history of the worker's actions.

[020] In some non-limiting embodiments of the system, the set of vector representations of the workers is a subset of the set of workers that meet a predetermined condition.

[021] In some non-limiting embodiments of the system, the predetermined condition corresponds to a subset of the plurality of workers that includes one or more workers with a previous success rate that exceeds a predetermined threshold.

[022] In some non-limiting embodiments of the system, the digital task is a digital task of the first type, and the predetermined condition corresponds to a subset of the plurality of workers that contains one or more workers with a previous success rate that exceeds a predetermined threshold for the digital task of the first type.

[023] In some non-limiting embodiments of the system, for the purpose of generating a vector representation of a worker for a worker, the processor is configured to: determine for the worker a latent parameter indicating the degree of bias of the worker in the direction of one or more latent features from a digital training task and determined by analyzing a discrepancy matrix associated with the worker; generate a representation of the worker based on the latent parameter.

[024] In some non-limiting embodiments of the system, for the purpose of generating a vector representation of a task for a training digital task, the processor is capable of: determining for the training digital task one or more latent features that influence the selection of a training label by the worker; generating a vector representation of the task based on the one or more latent features.

[025] In some non-limiting embodiments of the system, one or more latent features include at least one of the following: a font size associated with the content of the educational digital task; an image size associated with the content of the educational digital task; a number of selectable labels associated with the educational digital task; a location of selectable labels in the content of the educational digital task.

[026] In the context of the present description, the term "server" means a computer program executed by appropriate hardware and capable of receiving requests (e.g., from client devices) via a network and executing or causing these requests to be executed. The hardware may be one physical computer or one computer system, which is not essential for the present technology. In this context, the expression "server" does not mean that every task (e.g., a received command or request) or some specific task is received, executed or started by the same server (i.e., the same software and/or hardware). This expression means that any number of software or hardware can receive, send, execute or cause to be executed any task or request or the results of any tasks or requests. All of these software and hardware may represent one server or several servers, and both of these cases are implied in the expression "at least one server".

[027] As used herein, the term "client device" refers to any computer hardware capable of executing programs suitable for performing a given task. Thus, some (non-limiting) examples of client devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network equipment such as routers, switches and gateways. It should be noted that in this context, a device that functions as a client device may also function as a server for other client devices. The use of the term "client device" does not exclude the use of multiple client devices to receive, send, perform or initiate the execution of any task or request, or the results of any tasks or requests, or the steps of any method described herein.

[028] As used herein, the term "database" means any structured collection of data, regardless of its specific structure, database management software, or computer hardware for storing, using, or otherwise enabling the use of that data. A database may be located on the same hardware as the process that stores or uses the information stored in the database, or the database may be located on separate hardware, such as a dedicated server or multiple servers.

[029] In the context of the present description, the expression "information" includes information of any kind or type capable of being stored in a database. Thus, information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, word lists, etc.

[030] In the context of the present description, the expression "component" includes the designation of software (suitable for certain hardware) necessary and sufficient to perform a certain function or several functions.

[031] In the context of this description, the expression "computer-usable storage medium" means media of any kind and type, including random access memory (RAM), read-only memory (ROM), disks (CD-ROM, DVD, floppy disks, hard disks, etc.), USB drives, solid state drives, magnetic tape drives, etc.

[032] As used herein, the numerals "first," "second," "third," etc., are used merely to distinguish between the nouns to which they refer, and not to describe any particular relationships between those nouns. For example, it should be understood that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy, or classification of servers in this case, nor does their use (in and of itself) imply the presence of a "second server" in every situation. Furthermore, as found in other contexts herein, reference to a "first" element and a "second" element does not preclude the fact that those two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, and in other cases different software and/or hardware.

[033] Each embodiment of the present technology relates to at least one of the above-mentioned objectives and/or to one of the above-mentioned aspects, but not necessarily to all of them. It should be understood that some aspects of the present technology associated with an attempt to achieve the above-mentioned objective may not correspond to this objective and/or may correspond to other objectives not explicitly mentioned herein.

[034] Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the attached drawings and in the claims.

Brief description of the drawings

[035] The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the attached drawings.

[036] Fig. 1 shows a diagram of a system implemented according to non-limiting embodiments of the present technology.

[037] Fig. 2 shows a screenshot of a crowdsourcing interface implemented according to non-limiting embodiments of the present technology and displayed on the screen of an electronic device of the system shown in Fig. 1.

[038] Fig. 3 shows an example of the training process of the MLA algorithm implemented according to non-limiting embodiments of the present technology.

[039] Fig. 4 shows a diagram of the process of determining a label for a digital task in a crowdsourcing environment.

[040] Fig. 5 shows a block diagram of a method for determining a label for a digital task in a crowdsourcing environment.

Implementation of the invention

[041] Fig. 1 shows a diagram of a system 100 suitable for implementing non-limiting embodiments of the present technology. System 100 is an example of a computer crowdsourcing environment. It is obvious that system 100 is given only to illustrate an embodiment of the present technology. Thus, the following description of the system is a description of examples illustrating the present technology. This description is not intended to define the scope or boundaries of the present technology. In some cases, useful examples of modifications to system 100 are given. They facilitate understanding, but also do not define the scope or boundaries of the present technology. These modifications do not constitute an exhaustive list. As will be understood by a person skilled in the art, other modifications are probably possible. Moreover, if in some cases modifications are not described (i.e., examples of modifications are missing), this does not mean that they are impossible and/or that this description contains the only possible embodiment of a particular element of the present technology. A person skilled in the art will understand that this may be otherwise. It should also be understood that the system 100 may in some cases represent a simplified implementation of the present technology and that such embodiments are presented to facilitate a better understanding thereof. Those skilled in the art will appreciate that other embodiments of the present technology may be significantly more complex.

[042] The examples and conventional language provided herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically provided examples and conditions. It is obvious that those skilled in the art are able to devise various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its spirit and scope. To facilitate a better understanding, the following description may contain simplified embodiments of the present technology. One skilled in the art will understand that other embodiments of the present technology may be significantly more complex.

[043] The description of the principles, aspects and embodiments of the present technology, as well as specific examples thereof, are intended to cover their structural and functional equivalents, regardless of whether they are currently known or will be developed in the future. For example, those skilled in the art should understand that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams implementing the principles of the present technology. It should also be understood that any block diagrams, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that can be represented on a machine-readable physical storage medium and can be executed by a computer or processor, regardless of whether such a computer or processor is explicitly shown or not.

[044] The functions of the various elements shown in the drawings, including any functional unit designated as a "processor", may be implemented using dedicated hardware as well as hardware capable of executing associated software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, and/or a plurality of separate processors, some of which may be shared. In some non-limiting embodiments of the present technology, the processor may be a general-purpose processor, such as a central processing unit (CPU), or a dedicated processor, such as a graphics processing unit (GPU). Furthermore, the explicit use of the term "processor" or "controller" should not be construed as referring solely to hardware capable of executing software, and may include, but are not limited to, hardware of a digital signal processor (DSP), a network processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a ROM for storing software, RAM, and/or a non-volatile memory. Other hardware, general purpose and/or custom, may also be included.

[045] Next, taking into account the above principles, some non-limiting examples are considered, illustrating various options for implementing aspects of the present technology.

[046] System 100 comprises server 102 and database 104 accessible to server 102.

[047] As shown schematically in Fig. 1, the database 104 contains indications of identifiers of a plurality of human evaluators 106 who have indicated their willingness to perform a crowdsourced digital task of at least one type and/or who have performed at least one crowdsourced digital task in the past and/or who are registered to perform a crowdsourced digital task of at least one type.

[048] In some non-limiting embodiments of the present technology, the database 104 is managed and/or administered by a crowdsourcing service provider, such as Yandex LLC (16 Leo Tolstoy St., Moscow, 119021, Russia). In other non-limiting embodiments of the present technology, the database 104 may be managed by another organization.

[049] There are no particular limitations on the implementation of the database 104. The database 104 may be implemented using any suitable known technology that provides the functions described herein. According to non-limiting embodiments of the present technology, the database 104 comprises a communication interface (not shown) for two-way communication with (or has access to) the communication network 110.

[050] In some non-limiting embodiments of the present technology, the Internet may be used as the communication network 110. In other non-limiting embodiments of the present technology, the communication network 110 may be implemented differently, such as any global communication network, local communication network, private communication network, etc.

[051] It is contemplated that the database 104 may be at least partially stored on the server 102 and/or at least partially controlled by the server 102. According to non-limiting embodiments of the present technology, the database 104 contains information associated with identifiers of at least some of the human evaluators 106 of the plurality of human evaluators 106 and sufficient for an element accessing the database 104, such as the server 102, to assign and send one or more digital tasks for execution by one or more of the human evaluators 106.

[052] At any given time, the plurality of human evaluators 106 may contain a varying number of human evaluators 106, such as 50 human evaluators, available to perform digital tasks. The plurality of human evaluators 106 may contain a greater or lesser number of human evaluators 106.

[053] The server 102 may be implemented as a conventional computer server. In a non-limiting example of implementing the present technology, the server 102 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. It is obvious that the server 102 may be implemented using any other suitable hardware and/or application software and/or firmware or a combination thereof. In the presented non-limiting embodiment of the present technology, the server 102 is implemented as a single server. In other non-limiting embodiments of the present technology, the functions of the server 102 may be distributed among multiple servers.

[054] Server 102 comprises a communication interface (not shown) for providing two-way communication with communication network 110 via communication line 108.

[055] There are no particular limitations on the implementation of the communication line 108, it depends on the implementation of the server 102. For example, the communication line 108 can be implemented as a wireless communication line (such as a 3G communication network channel, a 4G communication network channel, Wireless Fidelity or abbreviated WiFi ^® , Bluetooth ^® , etc.) or a wired communication line (such as an Ethernet-based connection).

[056] It should be understood that the embodiments of the server 102, the communication line 108 and the communication network 110 are given for illustrative purposes only. Other specific details of the implementation of the server 102, the communication line 108 and the communication network 110 should be clear to a person skilled in the art. The examples presented above do not limit the scope of the present technology in any way.

[057] The server 102 comprises a server memory 114 comprising one or more storage media and generally storing computer program instructions executed by the server processor 116. For example, the server memory 114 may be implemented as a computer-readable physical storage medium, including ROM and/or RAM. The server memory 114 may also include one or more persistent storage devices, such as hard disk drives (HDDs), solid state drives (SSDs), and flash memory cards.

[058] In some non-limiting embodiments of the present technology, the server 102 may be operated by an organization that manages the database 104. In other non-limiting embodiments of the present technology, the server 102 may be operated by an organization other than the one that manages the database 104.

[059] In some non-limiting embodiments of the present technology, the server 102 is capable of executing the crowdsourcing application 118. For example, the crowdsourcing application 118 may be implemented as a crowdsourcing platform such as Toloka™ or another proprietary or commercially available crowdsourcing platform.

[060] For this purpose, the server 102 is connected to a digital task database 121. In other non-limiting embodiments of the invention, the digital task database 121 may be connected to the server 102 via the communication network 110. Although the digital task database 121 is schematically shown here as a single element, it is contemplated that the digital task database 121 may be distributed.

[061] The database 121 of digital tasks stores an indication of a plurality of digital tasks (not separately designated), each of which corresponds to a task for human intelligence (HIT, Human Intelligence Task), hereinafter also referred to simply as a task (not separately designated).

[062] There are no restrictions on how the digital task database 121 can be populated with a plurality of digital tasks. In general, one or more digital task requesters (not shown) can submit one or more digital tasks to be stored in the digital task database 121. In some non-limiting embodiments of the present technology, one or more digital task requesters can specify the type of evaluators for which the digital task is intended and/or the budget allocated to each human evaluator 106 who provides a result.

[063] In some non-limiting embodiments of the present technology, the digital task database is further configured to store a set of task features (not shown) associated with each digital task stored in the digital task database 121. For example, the set of task-dependent features for a digital task may, among other things, comprise one or more task identifiers and one or more latent features associated with the digital task. As used herein, the term "latent feature" may correspond to any feature associated with a digital task that, due to the bias of the human evaluator 106, may influence in one direction or another the decision of the human evaluator 106 in performing a digital task, but is not associated with the quality of the selected options provided in this digital task (i.e., in the given digital assessment task). In other words, the latent features of the digital task do not (or should not) have a direct correlation with the label assigned to one of the two choice options in the digital task, but may nonetheless influence human raters 106 when they perform the digital task (as described in more detail below).

[064] There are no restrictions on the formation of the set of latent features. For example, the set of latent features may be formed by the operator of the crowdsourcing application 118 or automatically by using a machine learning algorithm in response to the submission of a digital task by the requestor to the crowdsourcing application 118.

[065] There are no restrictions on the implementation of the digital task. In some non-restrictive embodiments of the present technology, the digital task database 121 comprises digital annotation tasks (also known as digital classification tasks).

[066] The digital classification task corresponds to a digital task in which a human evaluator 106 is asked to select an option using a label from a first category and/or a second category. Fig. 2 shows a screenshot of a crowdsourcing interface 200 implemented according to a non-limiting embodiment of the present technology (an example of an interface 200 displayed on a screen of one of the electronic devices 120 is shown). The interface 200 illustrates a digital image classification task 208.

[067] The interface 200 comprises an instruction 202 for human raters 106 and an object (i.e., an image 204). For clarity, it should be noted that the text (and, in particular, each letter) in the interface 200 is represented by the symbols "X". However, the text actually consists of words in some language (e.g., English). In particular, the instruction 202 may comprise instructions to the human raters 106 to select the correct classification option for the animal represented in the image 204. The interface comprises a first label 210 associated with the word "cat" and a second label 212 associated with the word "dog". Obviously, other types of classification tasks are also possible, such as classification of documents, texts, etc. In addition, it should be understood that only two labels (i.e., the first label 210 and the second label 212) are available for selection in the digital task 208 solely for the convenience of explanation. It is assumed that more than two labels may be presented as options to select from.

[068] The set of latent features may, among other things, contain visual and/or textual features associated with the digital task. For example, in the digital task 208, the set of latent features may, among other things, contain at least one of the following:

- the font size associated with the instruction 202 or with other text content of the interface 200;

- a type of digital task related to digital task 208 (e.g. categorization, translation, relevance assessment, etc.);

- the complexity inherent in the digital task 208 as assessed by the operator associated with the crowdsourcing application 118;

- demographic data of the human evaluator 106;

- the determination of a human evaluator 106 who has read the instruction 202, based on the response time for completing the digital task 208;

- image size 204;

- the number of selectable marks in digital problem 208 and their corresponding locations.

[069] In some non-limiting embodiments of the present technology, the digital task database 121 also comprises digital tasks that are not digital classification tasks. For example, a non-classification digital task may comprise pairwise comparison digital tasks. A pairwise comparison corresponds to a digital task in which human raters 106 are asked to rank one or more objects (such as search engine result pages (SERPs), translations, etc.). In such a case, the set of latent features may, among other things, further comprise:

- morphological, syntactic and semantic relationships between lexemes contained in texts not related to the classification of a digital task, and/or

- coincidences of words or phrases contained in texts not related to the classification of the digital task, and/or

- the context of words and verbal expressions within phrases contained in a non-classification digital task.

[070] According to non-limiting embodiments of the present technology, the crowdsourcing application 118 is capable of assigning a digital task to at least a subset of the plurality of human evaluators 106 who have indicated their availability in the database 104 (see FIG. 1).

[071] The server 102 is capable of communicating with various elements via the communication network 110. Examples of various elements include the database 104, the corresponding electronic devices 120 of the human evaluators 106, and other devices that can be connected to the communication network 110. Accordingly, the crowdsourcing application 118 is capable of receiving a digital task from the database 121 of digital tasks and sending the digital task to the corresponding electronic device 120 used by the plurality of human evaluators 106 to perform the task, for example, via the communication network 110.

[072] It is contemplated that any suitable file transfer technology and/or medium may be used for this purpose. It is also contemplated that the digital task may be sent to the plurality of human evaluators 106 using any other suitable method, such as by providing remote access to the digital task for the plurality of human evaluators 106.

[073] According to non-limiting embodiments of the present technology, the server 102 is capable of receiving a set of results of a digital task (in the form of labels) performed by a plurality of human evaluators 106. According to non-limiting embodiments of the present technology, the set of results may be received by the server 102 in one or more data packets 122, such as via the communication network 110. It is contemplated that any other suitable means of data transmission may be used.

[074] In general, the crowdsourcing application 118 is capable of aggregating a set of results in order to determine a "true" label for the digital task at hand. For example, in response to the digital task 208, the crowdsourcing application 118 may obtain 200 raw (i.e., unaggregated) labels using the data packets 122, each of which was selected by a corresponding human evaluator 106. In general, the crowdsourcing application 118 is capable of performing one or more aggregation methods in order to obtain a true label from the conflicting selected labels. The most basic aggregation model known in the art is majorization, in which the most frequently occurring label for the digital task 208 is considered the true label.

[075] In some non-limiting embodiments of the present technology, the server 102 is also connected to the log database 124 using a communication line (not indicated), which may be a dedicated communication line, etc. In other non-limiting embodiments of the present technology, the log database 124 may be connected to the server 102 using a communication network 110 without going beyond the scope of the present technology. Although the log database 124 is schematically shown here as a single element, it is contemplated that the log database 124 may be distributed.

[076] The log database 124 is capable of collecting and storing information related to the human evaluators 106 and to the digital tasks performed by the human evaluators 106.

[077] For example, the database 124 of logs may store worker action histories containing parameters and characteristics of the interaction of human evaluators 106 with the crowdsourcing application 118. In particular, each worker action history is associated with a specific human evaluator 106. The worker action history may contain, among other things:

- the time of registration of the human evaluator 106 in the crowdsourcing application 118;

- the number of digital tasks completed by a human evaluator 106 since registration and the time of their completion;

- the number of tasks performed per hour;

- types of digital tasks performed by the human evaluator 106 since registration;

- identifiers of digital tasks performed by the human evaluator 106 since registration and the corresponding selected labels;

- the quality assessment or success rate of a human evaluator 106, corresponding to the reliability of the result of a digital task performed by a human evaluator 106, or, in other words, the error rate of the human evaluator.

[078] There are no restrictions on the method for determining the quality score of the human evaluator 106. For example, the quality score may be determined based on a first set of "trap tasks" performed by the human evaluator 106. Here, the term "trap task" means a digital task whose correct result is known before it is sent for execution to the human evaluator 106, which is tested or evaluated for the purpose of determining the associated quality score, while the correct result is not known to the human evaluator 106 being evaluated.

[079] In some non-limiting embodiments of the present technology, the quality score may also be determined (in addition to or without the "trap tasks") based on an analysis of labels previously selected by the human evaluator 106 and determined to be true labels for the corresponding digital task by the crowdsourcing application 118. It should be understood that the quality score may correspond to a general quality score that takes into account all previously completed digital tasks, and may also include a type-specific quality score that takes into account only previously completed digital tasks of a particular type.

[080] Although the system 100 is described with reference to various hardware elements (such as the database 104, the server 102, the log database 124, the digital task database 121, etc.) shown separately, it should be understood that this is done for the sake of clarity. It is contemplated that the various functions performed by these elements may be performed by a single element or may be distributed among different elements.

[081] In some non-limiting embodiments of the present technology, the crowdsourcing application 118 is capable of executing the MLA algorithm 126. In some non-limiting embodiments of the present technology, the MLA algorithm 126 is trained to perform a digital task (such as the digital task 208) without the involvement of human evaluators 106 (described in more detail below). For example, in the case of the digital task 208, the MLA algorithm 126 is capable of selecting a label (e.g., the first label 210 and/or the second label 212) in response to receiving the digital task 208.

Algorithm 126 MLA – training stage

[082] Fig. 3 shows a diagram of the training process of the 126 MLA algorithm.

[083] To better understand the underlying concepts of the present technology, it should be understood that the training of the MLA algorithm 126 can be divided into a first stage and a second stage. In the first stage, training input data (described below) is generated. In the second stage, the MLA algorithm 126 is trained using the training input data. Although the steps of training the MLA algorithm 126 are described as being performed by the processor 116, the scope of the invention is not limited thereto. It should be understood that the server 102 and/or another server associated with the server 102 can train and/or execute the MLA algorithm 126.

[084] The following is a description of the formation of training input data, which begins with the digital task database 121. As indicated above, the digital task database 121 contains a plurality of digital tasks. The training step is further described with reference to one or more training digital tasks stored in the digital task database 121. It should be understood that these training digital tasks may not differ from the digital tasks described above with reference to Fig. 2. In other words, the training digital tasks may be stored in the digital task database 121 in a similar manner as described above.

[085] For illustration purposes, it may be assumed that the digital task database 121 contains a training digital task 302 that the crowdsourcing application 118 sends to a set of human raters 106 comprising a first human rater 304, a second human rater 306, and a third human rater 308. It is obviously contemplated that the training digital task 302 may be sent to more or less than three human raters.

[086] In response to receiving the training digital task 302, each evaluator of the first human evaluator 304, the second human evaluator 306, and the third human evaluator 308 performs the training digital task 302 by selecting a label. For example, the MLA algorithm 126 is capable of receiving a set of responses 310 that includes a first training label 312 assigned by the first human evaluator 304, a second training label 314 assigned by the second human evaluator 306, and a third training label 316 assigned by the third human evaluator 308. It should be noted that the MLA algorithm 126 is capable of receiving a set of responses 310 that includes unprocessed (i.e., unmerged) first training label 312, second training label 314, and third training label 316.

[087] In some non-limiting embodiments of the present technology, the training digital task 302 is further input into the MLA algorithm 126 to generate a vector representation 318 of the training digital task 302. In some non-limiting embodiments of the present technology, the MLA algorithm 126 is configured to determine or obtain from the digital task database 121 one or more latent features associated with the training digital task 302 and generate a vector representation 318 of the task based on the one or more latent features associated with the training digital task 302. There are no restrictions on generating the vector representation 318 of the task based on the one or more latent features associated with the training digital task 302. This process may include, among other things, the use of an algorithm based on Long Short-Term Memory (LSTM), an algorithm based on Bidirectional Encoder Representations from Transformers (BERT), an algorithm based on a Convolutional Neural Network (CNN), etc.

[088] In some non-limiting embodiments of the present technology, the MLA algorithm 126 is further configured to obtain from the log database 124 a history of worker actions for each human rater in the set of human raters 106. The MLA algorithm 126 is then configured to generate a set 320 of worker vector representations comprising a first worker vector representation 322 associated with the first human rater 304, a second worker vector representation 324 associated with the second human rater 306, and a third worker vector representation 326 associated with the third human rater 308. No limitations are imposed on the generation of the set 320 of worker vector representations. This process may include, among other things, using an LSTM memory-based algorithm, a BERT representation-based algorithm, a CNN network-based algorithm, etc.

[089] In some non-limiting embodiments of the present technology, the MLA algorithm 126 is configured to determine a latent parameter for the human rater that is indicative of the degree of bias of the human rater in the direction of one or more latent features. There are no limitations on the determination of the latent parameter. Given that the worker's action history contains (a) an indication of all identifiers of previously completed tasks and (b) an indication of identifiers of tasks for which labels previously selected by the associated human rater 106 were determined to be true labels for the corresponding digital task, the MLA algorithm 126 is configured to determine the latent parameter by analyzing a confusion matrix of the human rater using, in some non-limiting embodiments of the present technology, a method based on common noise adaptation layers (CoNAL). In particular, the MLA algorithm 126 is capable of determining a latent parameter for a human rater by analyzing the influence of one or more sets of latent features on the response of the human rater 106. Obviously, other means of determining the latent parameter are also possible, such as using the David-Skene model, the Generative model of Labels, Abilities, and Difficulties (GLAD) and the Matrix-Mean-Subsequence-Reduced (M-MSR) model. In some non-limiting embodiments of the present technology, the MLA algorithm 126 is capable of forming a set 320 of vector representations of workers based on one or more latent features of the first human rater 304, the second human rater 306 and the third human rater 308.

[090] There are no restrictions on the formation of the set 320 of vector representations of workers. In some non-limiting embodiments of the present technology, a vector representation of a worker (such as a first vector representation 322 of the worker) is formed as a random vector (not shown) of random length. In a training step, the first random vector is concatenated with a previous vector representation of a task associated with a digital task previously completed by the worker, resulting in a "glued" vector. A backpropagation method is then used to improve the first random vector and the previous vector representation of the task, wherein the backpropagation is performed based on a label selected by the worker for a previously completed digital task.

[091] It should be appreciated that although only one training digital task (i.e., training digital task 302) and three human raters (i.e., first human rater 304, second human rater 306, and third human rater 308) are shown here for clarity, it is contemplated that more than one training digital task may be sent to more than three human raters.

[092] The set 310 responses, the task vector representation 318, and the set 320 worker vector representations collectively form the training data set 328 (described in more detail below).

[093] The training of the MLA algorithm 126 is described below using a training data set 328. The training data set 328 comprises triplets of training data, namely: a first triplet 330 of training objects, a second triplet 332 of training objects, and a third triplet 334 of training objects. In some non-limiting embodiments of the present technology, each triplet of training objects is associated with a human evaluator and with a training digital task. For example, the first triplet 330 of training objects is associated with a first human evaluator 304 and comprises a first training label 312, a vector representation 318 of a task, and a first vector representation 322 of a worker.

[09] A set of 328 training data, or in particular individual triplets of training objects, are input to an MLA algorithm 126. The MLA algorithm 126 comprises a learning logic for determining a set of features associated with each triplet of training objects (e.g., with a first training label 312, with a task vector representation 318, and with a first worker vector representation 322). Based on the set of features associated with each triplet of training objects, the MLA algorithm 126 is configured to learn to predict a label for a digital task based on a task vector representation defined for the digital task and a worker vector representation. In particular, at the use stage (described in detail below), the MLA algorithm 126 is configured to generate a probability distribution for selecting a digital task label by a human evaluator given a worker vector representation for the human evaluator and a task vector representation for the digital task (described in detail below).

[095] Although only one instance of the training process of the MLA algorithm 126 is presented here, this is done solely for the sake of clarity. It is obvious that the training of the MLA algorithm 126 is performed iteratively using many different triplets of training objects.

Algorithm 126 MLA – Use Stage

[096] Next, after describing the training of the MLA algorithm 126 prior to the use stage, a flow chart of the operation of the crowdsourcing application 118 capable of executing the MLA algorithm 126 (see Fig. 1) is described with reference to Fig. 4. The crowdsourcing application 118 executes the receiving procedure 402, the selecting procedure 404, and the combining procedure 406 (or otherwise accesses them).

[097] In the context of the present description, the term "procedure" means a subset of the computer program instructions of the crowdsourcing application 118 executed by the processor 116 for the purpose of implementing the functions described below associated with various procedures (with the reception procedure 402, the selection procedure 404 and the merging procedure 406). It should be clearly understood that the reception procedure 402, the selection procedure 404 and the merging procedure 406 are schematically shown as separate elements only for the convenience of explaining the processes performed by the crowdsourcing application 118. It is contemplated that some or all of the procedures from the reception procedure 402, the selection procedure 404 and the merging procedure 406 can be implemented in the form of one or more combined procedures. In addition, it is assumed that some procedures from among the receiving procedure 402, the selecting procedure 404 and the merging procedure 406 are performed by an application (not shown) associated with the crowdsourcing application 118, which is stored on the server 102 or in another element.

[098] For a better understanding of the present technology, the functions and the processed or stored data and/or information of the receiving procedure 402, the selecting procedure 404, and the combining procedure 406 are described below.

Reception procedure 402

[099] The receiving procedure 402 is capable of receiving a data packet 408 from the digital task database 121. The data packet contains a digital task to be performed by one or more human evaluators 106. For the sake of explanation, it may be assumed that the data packet 408 contains a digital task 208. It is obvious that only one data packet 408 is shown solely for better understanding. It should be understood that the receiving procedure 402 may receive multiple data packets, each of which contains a digital task.

[0100] In some non-limiting embodiments of the present technology, the receiving procedure 402 is capable of analyzing a set of task features for the digital task 208 by accessing the digital task database 121. In particular, the receiving procedure 402 is capable of determining the type and complexity of the digital task 208.

[0101] In some non-limiting embodiments of the present technology, the receiving procedure 402 is further configured to access the database 124 of logs and select one or more worker action histories that meet a predetermined condition. In some non-limiting embodiments of the present technology, the predetermined condition corresponds to selecting one or more worker action histories with a quality score that exceeds a predetermined threshold. In further non-limiting embodiments of the present technology, the predetermined condition corresponds to N worker action histories with a quality score that exceeds a predetermined threshold for tasks that are similar in type and complexity to the digital task 208. In particular, given that the digital task 208 is a digital classification task, the receiving procedure 402 is configured to select one or more worker action histories with a quality score for digital classification tasks of similar complexity (similar to the digital task 208) that exceeds a predetermined threshold. There are no restrictions on determining the predetermined threshold. For example, it can be determined by an administrator of the crowdsourcing application 118.

[0102] It may be assumed that the receiving procedure 402 has determined three worker action histories that meet the predetermined condition. Then, the receiving procedure 402 may generate a set 418 of vector representations of workers, each of which corresponds to a vector representation of a worker action history that meets the predetermined condition. Although only three worker action histories are represented here as meeting the predetermined condition, this is done solely for the sake of clarity. It should be understood that more or less than three worker action histories may meet the predetermined condition.

[0103] The receiving procedure 402 is further configured to send a data packet 414 to the selecting procedure 404. The data packet 414 comprises a digital task 208 and a set 418 of vector representations of workers.

404 Selection Procedure

[0104] In response to receiving the data packet 414, the selection procedure 404 is capable of performing the following functions.

[0105] First, the selection procedure 404 is configured to generate a vector representation 416 of the problem for the digital problem 208 using the MLA algorithm 126.

[0106] Having generated the task vector representation 416 and having the set 418 worker vector representations, the selection procedure 404 is capable of executing the MLA algorithm 126, previously trained to predict a label (i.e., the first label 210 or the second label 212) based on the task vector representation 416 and each worker vector representation from the set 418 worker vector representations. Given that the set 418 worker vector representations contains three worker vector representations, the MLA algorithm 126 can generate a first set 420 of selected labels (each individually selected result or label by the MLA algorithm 126 is shown as a triangle) and send the set 420 of selected labels to the merging procedure 406.

406 Merger Procedure

[0107] The first set 420 selected labels forms a total set 422 selected labels.

[0108] In some non-limiting embodiments of the present technology, the merging procedure 406 is configured to select a "true" or "correct" label for the digital task 208 from the total set 422 selected labels. There are no restrictions on determining such a label. In some non-limiting embodiments of the present technology, the correct label corresponds to the label with the majority vote as a result of majorization from the total set 422 selected labels. In some non-limiting embodiments of the present technology, it is contemplated that the "true" or "correct" label can be determined using a weighted majority system. An implementation of a weighted majority system is known in the art and can be based on the GLAD algorithm or the David-Skeen algorithm.

[0109] For example, it may be assumed that in the total set 422 of selected labels, the first label 210 is selected twice and the second label 212 is selected once, then the merging procedure 406 may select the first label 210 as the correct label for the digital task 208.

[0110] Although in the above description the total set 422 of selected labels comprises only the first set 420 of selected labels (obtained by the MLA algorithm 126), the scope of the invention is not limited thereto.

[0111] In some non-limiting embodiments of the present technology, it is contemplated that the total set 422 of selected labels also comprises a second set of selected labels (not shown) obtained by one or more human raters 106. In particular, in some non-limiting embodiments of the present technology, the receiving procedure 402 is further configured to send a data packet 408 to one or more human raters 106 that select one or more labels that form the second set of labels sent to the combining procedure 406. In this case, the total set 422 of selected labels comprises a first set 420 of selected labels (obtained by the MLA algorithm 126) and a second set of labels (obtained by the human raters 106). In some non-limiting embodiments of the present technology, the three worker action histories that meet the predetermined condition and are selected by the selection procedure 404 are not associated with one or more human raters 106 that received the data packet 408 and sent a label from the second set of selected labels. In some non-limiting embodiments of the present technology, the merging procedure 406 is further capable of calculating and sending a reward to the human raters 106 that selected the correct label, i.e., the first label 210.

[0112] In some non-limiting embodiments of the present technology, the merging procedure 406 is further configured to access the log database 124 (see FIG. 1) and update the quality ratings of the human raters 106 who submitted the second set of selected labels.

[0113] Various non-limiting embodiments of the present technology may permit selection of a label for a digital task using the MLA algorithm 126 with or without the use of labels obtained by one or more human evaluators 106.

[0114] The architecture and examples described above allow for a computer method of generating a digital task label by a machine learning algorithm on a crowdsourcing digital platform to be performed. Fig. 5 shows a flow chart of a method 500 performed according to non-limiting embodiments of the present technology. The method 500 may be performed by the server 102 or another element (such as a server) associated with the server 102.

[0115] Step 502: receiving by the server a digital learning task for execution on the crowdsourcing digital platform.

[0116] The method 500 begins with step 502, in which the training digital task 302 is sent by the crowdsourcing application 118 to a set of human evaluators 106.

[0117] Step 504: receiving by the server a plurality of digital learning task labels corresponding to the digital learning task and sent by a plurality of workers of the crowdsourcing digital platform, wherein the digital learning label is sent by the worker in response to the digital learning task using the crowdsourcing digital platform.

[0118] In step 504, in response to receiving the training digital task 302, each evaluator of the first human evaluator 304, the second human evaluator 306, and the third human evaluator 308 performs the training digital task 302 by selecting a label. For example, the MLA algorithm 126 is capable of receiving a set of responses 310 that contains a first training label 312 assigned by the first human evaluator 304, a second training label 314 assigned by the second human evaluator 306, and a third training label 316 assigned by the third human evaluator 308. It should be noted that the MLA algorithm 126 is capable of receiving a set of responses 310 that contains unprocessed (i.e., unmerged) first training label 312, second training label 314, and third training label 316.

[0119] Step 506: receiving by the server a history of worker actions associated with each worker from the plurality of workers and containing digital task labels previously submitted by each worker.

[0120] At step 506, the MLA algorithm 126 of the server 102 is configured to obtain from the log database 124 the history of worker actions for each human evaluator from the set of human evaluators 106.

[0121] Step 508: training, by a server, an MLA algorithm, including: inputting, by the server, a digital training task into an MLA algorithm capable of generating a vector representation of the task corresponding to the vector representation of the digital training task; inputting, by the server, histories of worker actions into an MLA algorithm capable of generating a vector representation of a worker corresponding to a vector representation of a worker action history for a worker from a plurality of workers; generating a triplet of training objects containing a vector representation of the task, a vector representation of the worker, and a label of the digital training task associated with the vector representation of the worker; using the triplet of training objects to train the MLA algorithm to predict a label of the digital task for the vector representation of the digital task determined for the digital task and the vector representation of the worker.

[0122] At step 508, the MLA algorithm 126 is configured to generate a set 320 of vector representations of workers comprising a first vector representation 322 of the worker associated with a first human evaluator 304, a second vector representation 324 of the worker associated with a second human evaluator 306, and a third vector representation 326 of the worker associated with a third human evaluator 308.

[0123] The MLA algorithm 126 is further configured to receive as input a training digital task and to generate a task vector representation 318 for the training digital task 302. In some non-limiting embodiments of the present technology, the MLA algorithm 126 is configured to determine or obtain from the digital task database 121 one or more latent features associated with the training digital task 302 and to generate the task vector representation 318 based on the one or more latent features associated with the training digital task 302.

[0124] The set 310 responses, the task vector representation 318, and the set 320 worker vector representations collectively form a training data set 328 (as described in more detail below).

[0125] The training of the MLA algorithm 126 is described below using the training data set 328. The training data set 328 comprises triplets of training data, namely: a first triplet 330 of training objects, a second triplet 332 of training objects, and a third triplet 334 of training objects. In some non-limiting embodiments of the present technology, each triplet of training objects is associated with a human evaluator and with a training digital task. For example, the first triplet 330 of training objects is associated with a first human evaluator 304 and comprises a first training label 312, a vector representation 318 of the task, and a first vector representation 322 of the worker.

[0126] A set of 328 training data, or in particular individual triplets of training objects, are input to an MLA algorithm 126. The MLA algorithm 126 comprises a learning logic for determining a set of features associated with each triplet of training objects (e.g., with a first training label 312, with a vector representation 318 of a task, and with a first vector representation 322 of a worker). Based on the set of features associated with each triplet of training objects, the MLA algorithm 126 is capable of learning to predict a label for a digital task based on a vector representation of the task and a vector representation of a worker.

[0127] Step 510 - at the use stage: receiving by the server a digital task; determining by the server a vector representation of the task for the digital task; predicting, using the MLA algorithm, a plurality of digital task labels for the digital task based on the set of vector representations of the workers and the vector representation of the task for the digital task; determining by the server a digital task label corresponding to at least one digital task label from the plurality of digital task labels for the digital task.

[0128] In step 510, the server 102 receives a data packet 408 containing a digital task 208 from the digital task database 121. The server 102 is capable of accessing the log database 124 and selecting one or more worker action histories that meet a predetermined condition. It may be assumed that the server 102 has determined three worker action histories that meet a predetermined condition. Then, the server 102 may generate a set 418 of vector representations of workers, each of which corresponds to a vector representation of a worker action history that meets a predetermined condition. The server 102 is further capable of generating a vector representation 416 of a task for the digital task 208 using the MLA algorithm 126.

[0129] Having generated the task vector representation 416 and having the set 418 worker vector representations, the server 102 is capable of executing the MLA algorithm 126 previously trained to predict a label (i.e., the first label 210 or the second label 212) based on the task vector representation 416 and each worker vector representation from the set 418 worker vector representations.

[0130] This completes Method 500.

[0131] It should be understood by those skilled in the art that at least some embodiments of the present technology are intended to expand the range of technical solutions to a specific technical problem characteristic of traditional crowdsourcing technology, namely, determining a result for a problem in a crowdsourcing environment.

[0132] It is obvious that not all technical effects mentioned in this description must be present in every embodiment of the present technology. For example, there may be embodiments of the present technology where the user does not receive some of these technical effects, or other embodiments where the user receives other technical effects or where the technical effect is absent.

[0133] Possible changes and improvements to the above-described embodiments of the present technology may be obvious to a person skilled in the art. The preceding description is provided for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of the present technology is determined solely by the scope of the appended claims.

[0134] Although the embodiments described above are described with reference to specific steps performed in a particular order, it should be understood that these steps may be combined, separated, or that their order may be changed without departing from the scope of the present technology. Accordingly, the order and grouping of the steps are not limiting to the present technology.

Claims (60)

1. A computer-implemented method for generating a digital task label using a machine learning algorithm (MLA), performed by a server associated with a crowdsourcing digital platform and comprising: at the training stage: - receiving a digital training task by the server for execution on a crowdsourcing digital platform; - receiving by the server a plurality of digital learning task labels corresponding to the digital learning task from a plurality of client devices associated with workers, wherein said plurality of client devices are associated with a crowdsourcing digital platform, in response to a digital learning task sent to the plurality of client devices using the crowdsourcing digital platform; - receiving by the server the history of digital tags containing digital task tags previously received from each client device; - server training of the MLA algorithm, including: - input of a digital learning task by the server into the MLA algorithm, which is capable of generating a vector representation of the task corresponding to the vector representation of the digital learning task; - input of digital tag histories by the server into the MLA algorithm to form a vector representation of the worker with whom a given client device from a set of client devices is associated; Moreover, the formation of a vector representation of the employee with whom this client device is associated includes: - determining for a given worker, with whom a given client device is associated, a latent parameter indicating the degree of the worker's bias in the direction of one or more latent features from a digital learning task and determined by analyzing the discrepancy matrix corresponding to the worker with whom a given client device is associated; - formation of a triplet of training objects containing a vector representation of the task, a vector representation of the worker with whom the given client device is associated, and a label of the digital training task associated with the vector representation of the worker with whom the given client device is associated; - using a triplet of training objects to train the MLA algorithm to predict the label of a digital task for a vector representation of a digital task and a vector representation of the worker with whom the client device is associated; at the stage of use: - receiving a digital task by the server; - determination by the server of the vector representation of the task for the digital task; - predicting, using the MLA algorithm, a set of digital task labels for a digital task based on a set of vector representations of workers associated with the corresponding client devices and a vector representation of the task for the digital task; - determining by the server a digital task label corresponding to at least one digital task label from a plurality of digital task labels for the digital task. 2. The method according to claim 1, characterized in that determining the digital task label includes performing majorization based on a plurality of digital task labels for the digital task. 3. The method according to claim 1, characterized in that it further includes determining, for each employee with whom the corresponding client device from the plurality of client devices is associated, a quality assessment corresponding to the previous success rate in providing correct digital task labels, determined on the basis of the history of digital labels. 4. The method according to paragraph 3, characterized in that the set of vector representations of workers with which the corresponding client devices are associated belongs to a subset of the set of workers corresponding to a predetermined condition. 5. The method according to claim 4, characterized in that the predetermined condition corresponds to a subset of the plurality of workers with which the corresponding client devices are associated, containing one or more workers with a previous success rate exceeding a predetermined threshold. 6. The method according to claim 3, characterized in that the digital task is a digital task of the first type, and the predetermined condition corresponds to a subset of the plurality of workers with which the corresponding client devices are associated, containing one or more workers with a previous success rate exceeding a predetermined threshold for the digital task of the first type. 7. The method according to paragraph 1, characterized in that the formation of a vector representation of the task for the training digital task includes: - determining for a training digital task one or more latent features that influence the selection of a training label received from a client device; - formation of a vector representation of the problem based on one or more latent features. 8. The method according to claim 7, characterized in that one or more latent features include at least one of the following: - font size related to the content of the educational digital task; - image size associated with the content of the educational digital task; - the number of selectable labels associated with the educational digital task; - the location of selectable labels in the content of the educational digital task. 9. A system for generating a digital task label using the MLA algorithm with a server connected to a crowdsourcing digital platform and containing a processor configured to: at the training stage: - receiving a digital learning task to complete on a crowdsourcing digital platform; - receiving a plurality of digital learning task labels corresponding to the digital learning task from a plurality of client devices associated with workers, wherein said plurality of client devices are associated with a crowdsourcing digital platform, in response to a digital learning task sent to the plurality of client devices using the crowdsourcing digital platform; - obtaining a history of digital tags containing digital task tags previously received from each client device; - training the MLA algorithm, for which the processor is designed with the ability to: - input of a digital learning task into the MLA algorithm, which is capable of generating a vector representation of the task corresponding to the vector representation of the digital learning task; - input of digital tag histories into the MLA algorithm to form a vector representation of the worker with whom a given client device from a set of client devices is associated; Moreover, in order to form a vector representation of the employee with whom the given client device is connected, the processor is designed with the ability to: - determining, for a given worker with whom a given client device is associated, a latent parameter indicating the degree of bias in the direction of one or more latent features from a digital learning task and determined by analyzing a discrepancy matrix corresponding to the worker with whom a given client device is associated; - forming a triplet of training objects containing a vector representation of the task, a vector representation of the worker with whom the given client device is associated, and a label of the digital training task associated with the vector representation of the worker with whom the given client device is associated; - using a triplet of training objects to train the MLA algorithm to predict the label of a digital task for a vector representation of a digital task and a vector representation of the worker with whom the client device is associated; at the stage of use: - obtaining a digital task; - definition of vector representation of the problem for a digital problem; - predicting, by executing the MLA algorithm, a set of digital task labels for a digital task based on a set of vector representations of workers associated with the corresponding client devices and a vector representation of the task for the digital task; - determining a digital task label corresponding to at least one digital task label from a plurality of digital task labels for a digital task. 10. The system according to claim 9, characterized in that the processor for determining the digital task label is configured to perform majorization based on a plurality of digital task labels for the digital task. 11. The system according to claim 9, characterized in that the processor is additionally configured to determine for each employee with whom the client device is associated, from a plurality of client devices, a quality assessment corresponding to the previous success rate in providing correct digital task labels, determined on the basis of the history of digital labels. 12. The system according to claim 11, characterized in that the set of vector representations of workers with which the corresponding client devices are associated relates to a subset of the set of workers with which the corresponding client devices are associated, corresponding to a predetermined condition. 13. The system of claim 12, wherein the predetermined condition corresponds to a subset of the plurality of workers with which the corresponding client devices are associated, containing one or more workers with which the corresponding client devices are associated, with a previous success rate exceeding a predetermined threshold. 14. The system of claim 11, wherein the digital task is a digital task of the first type, and the predetermined condition corresponds to a subset of the plurality of workers with which the corresponding client devices are associated, containing one or more workers with which the corresponding client devices are associated, with a previous success rate exceeding a predetermined threshold for the digital task of the first type. 15. The system according to item 9, characterized in that, in order to form a vector representation of the task for the training digital task, the processor is designed with the ability to: - determining for a training digital task one or more latent features that influence the selection of a training label received from a client device; - formation of a vector representation of the problem based on one or more latent features. 16. The system of claim 15, wherein one or more latent features include at least one of the following: - font size related to the content of the educational digital task; - image size associated with the content of the educational digital task; - the number of selectable labels associated with the educational digital task; - the location of selectable labels in the content of the educational digital task.