KR102155873B1 - Method and apparatus for performing crowdsourcing project for collecting various natural language conversation datasets - Google Patents

Abstract

Provided are a method and an apparatus for performing a crowdsourcing project for collecting various natural language conversation datasets. The method includes the steps of: setting a target value of a plurality of natural language conversation datasets; generating a plurality of channels corresponding to the plurality of natural language conversation datasets in accordance with the target value; randomly determining one of the plurality of channels and allowing a worker to enter the channel in accordance with a request of the worker for participation in the crowdsourcing project; requesting the worker to perform a task inputting one utterance through the channel, and receiving the utterance from the worker; and making the worker leave the channel when the task inputting the utterance is completed. Each of the natural language conversation datasets comprises at least two pieces of utterance data. Therefore, the present invention is capable of making the progress of a project fast by reducing a working time and a workload of each worker.

Description

A crowdsourcing project execution method and device for collecting various natural language conversation datasets {METHOD AND APPARATUS FOR PERFORMING CROWDSOURCING PROJECT FOR COLLECTING VARIOUS NATURAL LANGUAGE CONVERSATION DATASETS}

The present invention relates to a method and apparatus for performing a crowdsourcing project for collecting various natural language conversation data sets.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

And, in order to collect more reliable information, the company assigns the results of work completed by the worker to the inspector to perform the inspection work.

On the other hand, in order to collect various natural language dialogues, a project to build dialogue datasets must be carried out.

To this end, in the related art, one worker proceeds in a role-playing manner from the beginning to the end of the conversation. In this case, there is a problem that the working time is lengthened and the conversation content and style are gradually restricted. In the end, this has a problem that it does not meet the purpose of the project to build various natural language dialogue datasets.

In addition, in the above case, because one worker has to do many tasks at once, the working time for one source is lengthened, and workers who feel burdened do not enter the project or task itself, There was a problem that it took a lot of time to complete the work.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a crowdsourcing project execution method and apparatus for collecting various natural language conversation datasets that enable obtaining various work results rather than uniform results.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method of performing a crowdsourcing project according to an aspect of the present invention for solving the above problem includes the steps of setting target values of a plurality of natural language conversation datasets, and a plurality of channels corresponding to the plurality of natural language conversation datasets according to the target values. Generating, receiving a request to participate in the crowdsourcing project from a worker, randomly determining any one of the plurality of channels, and entering the worker into the channel, to the worker through the channel Requesting to perform a task for inputting one utterance, receiving the utterance from the worker, and leaving the worker from the channel when the task for inputting the utterance is completed, and each natural language conversation The dataset consists of two or more speech data.

In some embodiments, the method provides a condition, situation, context, or scenario for inputting the utterance to the worker through the channel.

In some embodiments, requesting the operator to perform a task of inputting one utterance through the channel, and receiving the utterance from the operator, the first utterance of the channel or the previous utterance of the channel through the channel Following with is a request to perform the task of entering the next utterance.

In some embodiments, the method comprises the steps of: switching the channel state to a non-entry state, requesting an inspector to inspect the utterance input from the operator of the channel, and the utterance input from the operator of the channel Upon completion of the inspection, the step of switching the state of the channel to an entry-enabled state.

In some embodiments, after leaving the worker from the channel, randomly determining any one of the plurality of channels, entering the worker into the channel, and giving the worker one A step of requesting to perform a task of inputting an utterance, receiving input of the utterance from the worker, and a step of leaving the worker from the channel when the task of inputting the utterance is completed are repeatedly performed.

In addition, after leaving the worker from the channel, randomly determining any one of the plurality of channels and entering the worker into the channel, the last utterance of the plurality of channels is performed by the worker. Excluding the input channel, any one channel to which the operator is to be admitted is randomly determined.

In some embodiments, the method allows only one worker to enter the channel and does not enter two or more workers at the same time.

In some embodiments, the method converts a state of a channel in which a target value of speech data of a corresponding natural language conversation data set among the plurality of channels has been reviewed, to a state in which entry is impossible.

A computer program according to another aspect of the present invention for solving the above problem is combined with a computer that is hardware, and is stored on a computer-readable recording medium to execute the crowdsourcing project execution method of any one of claims 1 to 8. Is saved.

A crowdsourcing project execution apparatus according to another aspect of the present invention for solving the above problem, transmits a work of a crowdsourcing project to a plurality of workers, receives work results from the plurality of workers, and receives from the plurality of workers. A communication module that transmits the input work results to a plurality of inspectors and receives the inspection results from the plurality of inspectors, a memory in which a computer program for collecting the natural language conversation data set is stored, and a computer program stored in the memory is executed. According to, target values of a plurality of natural language conversation datasets are set, a plurality of channels corresponding to the plurality of natural language conversation datasets are generated according to the target values, and according to a request of the worker to participate in the crowdsourcing project, the plurality of Randomly determine any one of the channels of, enter the operator into the channel, request the operator to perform a task of inputting one utterance through the channel, and receive the utterance from the operator, And a processor that exits the operator from the channel when the operation of inputting the utterance is completed, and each natural language conversation data set is composed of two or more utterance data.

Other specific details of the present invention are included in the detailed description and drawings.

According to the means for solving the above-described problems, there is an advantage in that it is possible to collect more diverse natural language conversation datasets than when one worker proceeds from the beginning to the end of conversation in a role-playing manner, and it is possible to collect natural language conversation datasets. There is an advantage that the overall project can be completed quickly because the work time required for this is reduced and the work burden of each worker is also reduced.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a process of a crowdsourcing project according to an embodiment of the present invention.
3 is a flowchart of a crowdsourcing project execution method for collecting various natural language conversation data sets according to an embodiment of the present invention.
4 is a diagram for explaining an example of a channel corresponding to a natural language conversation data set.
5 is an example of a case where the first utterance starts.
6 and 7 are examples of a case in which continuous speech continues to a previous conversation.
FIG. 8 is a diagram illustrating an apparatus for performing a crowdsourcing project for collecting various natural language conversation data sets according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

2 is a flowchart illustrating a process of a crowdsourcing project according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the workers 32 of the corresponding grade or higher according to the grade of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the inspection work on the work result to the inspector 34 (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and determines rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

Meanwhile, in FIGS. 1 and 2, the client 10, the service provider 20, the worker 32, and the inspector 34 are simply expressed, but these are smartphones, tablets, PDAs, and laptops operated by each participant. It means a computer device or a telecommunication device such as a desktop or a server.

3 is a flowchart of a crowdsourcing project execution method for collecting various natural language conversation data sets according to an embodiment of the present invention. 4 is a diagram for explaining an example of a channel corresponding to a natural language conversation data set.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the worker 32 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Referring to FIG. 3, the server sets target values of a plurality of natural language conversation data sets (S105).

One project contains multiple tasks. The plurality of jobs may be grouped into a plurality of tasks according to types, or each job may be a different type of job or the same job, but is not limited thereto.

In an embodiment of the present invention, the project may be a project for collecting a plurality of natural language conversation datasets, and in this case, the task may be a task of allowing a worker to directly input a speech.

Each natural language conversation dataset consists of two or more speech data. Speech is when an idea is expressed in words or voice. Two or more utterances make up the conversation.

The target value means a target value for the number of natural language conversation data sets and the number of speech data constituting each natural language conversation data set. For example, a goal is to collect 100 natural language conversation data sets, and each natural language conversation data set may consist of 10 speech data. In this case, as a whole, 1000 speech data are collected.

In the following description of the present invention, a project for collecting various natural language conversation data sets using channels will be described as an example with reference to FIG. 4.

Next, the server generates a plurality of channels corresponding to the plurality of natural language conversation datasets according to the target value (S110).

The server receives the crowdsourcing project participation request from the worker (S115), randomly determines any one of the plurality of channels, and enters the worker who transmitted the participation request to the determined channel (S120).

According to an embodiment of the present invention, a channel is created to correspond to a natural language conversation dataset, and the operator 32 can perform a task of inputting a speech only when entering the channel.

For example, referring to FIG. 4, 100 channels corresponding to 100 natural language conversation datasets are generated. And the operator randomly enters any one of the 100 channels.

In one embodiment, when the server receives a request to participate in a crowdsourcing project from the worker 32, the server enters the worker 32 into a specific channel in a random order among a plurality of channels generated for the project to be participated.

In the project in which a total of 100 natural language conversation datasets, which is the example of FIG. 4 described above, must be collected, the server enters a random worker into'Channel 2'in a random order among 100 channels, and is already input by another worker. Following the fifth utterance, the next utterance, the sixth utterance, can be entered.

At this time, an embodiment of the present invention aims to collect various natural language conversation datasets, except for a channel in which the last utterance is input by the same operator among a plurality of channels, one to enter the operator 32 The channel of can be randomly determined. That is, in any channel, a worker different from the worker who has already performed the work and input the last utterance of the channel enters the channel.

That is, when the input of the 5th utterance in'Channel 2'described above is completed, the operator A who inputs the 5th utterance is not allowed to enter'Channel 2', and only other workers other than worker A It is possible to enter 2', so you can enter the 6th utterance following the 5th utterance.

Likewise, if the 6th ignition is input by worker B, worker B cannot enter'Channel 2', and only other workers including worker A can enter'Channel 2', so that the 6th utterance This can input the 7th utterance.

Of course, worker A can enter any channel other than'Channel 2'.

On the other hand, when a certain worker enters a specific channel, other workers are restricted from entering the same channel, and when a worker in the corresponding channel leaves, it is possible to enter. That is, only one worker can enter the channel, and two or more workers cannot enter the channel at the same time.

Next, the server requests the operator to perform a task of inputting an utterance through the channel to which the operator 32 enters, and receives the utterance from the operator 32 (S125).

In one embodiment, the server requests to perform a task of inputting only one utterance through one channel, and the worker 32 may complete the task by inputting only one utterance.

In an embodiment, the server may request to perform a task of inputting the first utterance of the channel or the next utterance following the previous utterance of the channel through a channel, and may receive one utterance from the operator 32.

That is, the server may receive the first utterance of the channel from the operator 32 through the channel. In this way, when there is no previously performed utterance in the channel, the server may request the operator to input a utterance suitable for the subject requested by the client 10. The server may provide a condition, situation, context, or scenario for inputting the utterance to a worker through a channel. In addition, in an embodiment, the operator 32 may request to input a random speech.

In addition, the server may receive input from the operator 32 for the continuous utterance following the previous utterance of the channel through the channel. In this case, as described above, the worker 32 who has input the previous utterance cannot input the continuous utterance, and only different workers can input the continuous utterance. One or more previous utterances may have been entered.

In an embodiment, while an utterance is input from one worker through a channel, the server may switch the state of the corresponding channel to a state in which the entry of other workers is impossible.

For example, while worker A enters'Channel 2'and inputs the 5th utterance, if another worker B enters'Channel 2', the same 5th utterance may be input at the same time. , In the'Channel 2', the entrance of other workers may be restricted until the utterance input starts and is completed.

Next, when the operation of inputting the utterance is completed, the server leaves the worker 32 from the corresponding channel (S130).

In one embodiment, when the worker 32 inputs one utterance and clicks the input completion button or the next job button, the server may automatically exit the worker 32 from the channel.

After leaving the worker 32 from the channel, the server randomly determines any one of the plurality of channels again, enters the worker 32 into the determined channel, and the worker 32 enters the channel. The task of entering the utterance again is performed. In this case, the entry channel is a channel different from the previously entered channel, as described above. In addition, when the input of the utterance is completed, the server leaves the worker 32 from the channel, and continues to perform this repeatedly until the worker 32 no longer wants to participate in the project.

Meanwhile, among the plurality of channels, the state of the channel in which the target value of the utterance data of the corresponding natural language conversation dataset is filled (that is, all utterances have been input and completed until the inspection) is switched to a state in which all workers cannot enter. For example, if entry to a channel is randomly determined, and a specific channel receives all 100 speech data first, and the inspection, which will be described later, is completed, the channel becomes inaccessible. Subsequently, the channel is excluded in randomly determining admission to the channel. Workers can only enter the channel where the remaining work exists, and no worker can enter the channel where all work has been completed.

For example, when worker A who enters'Channel 2'among 100 channels completes the fifth utterance input, the server automatically exits worker A from'Channel 2'. In this case, operator A cannot enter'Channel 2'unless another operator completes the input of the 6th utterance of'Channel 2', and the server has remaining work (ignition input) except for'Channel 2'. Worker A is randomly entered for the channel to be used.

In addition, when one conversation set is composed of 10 speeches and it is necessary to input 10 speeches through one channel, the corresponding "channel 2" is completed as the 10th speech input is completed in'channel 2'and until inspection is completed. 'Is set to a state in which all workers cannot enter.

In one embodiment, the server converts the state of the channel into which the utterance has been input to a state that cannot be entered, requests an inspector to inspect the utterance input from the operator of the channel (S135), and receives the inspection result of passing from the inspector. Upon completion of the verification of the utterance, the state of the channel is switched to an entry-enabled state (S140).

While the inspection result is input from the inspector 34, the corresponding channel is set to a state in which no entry of any operator 32 is allowed. Thereafter, as the input of the inspection result is completed, the state of the channel is switched to the entry-enabled state.

On the other hand, if the inspection result transmitted from the inspector 34 is rejected, the server may request rework from the operator 32 who has entered the utterance. Alternatively, the rejected utterance of the worker 32 may be deleted, and the same or new worker may enter the corresponding channel to input a new utterance.

Hereinafter, a method of performing a crowdsourcing project according to an embodiment of the present invention will be described with reference to an example shown in FIG. 5 below.

5 is an example of a case where the first utterance starts.

In a project for the purpose of collecting various natural language conversation datasets, a channel is created for each natural language conversation dataset composed of a plurality of speech data, and one speech data can be input through one task. The worker 32 randomly enters the channel and exits the channel when input for one utterance is completed through the channel.

FIG. 5 shows a case in which a previously performed utterance does not exist, that is, when a conversation starts. In this case, an initial utterance may be input from the operator 32.

The worker 32 may input a utterance that fits the project subject set by the client 10 into the input window, or input a utterance that starts a conversation at random into the input window.

In FIG. 5, the operator 32 inputs an initial utterance of "What can I do for you?" through an input window.

6 and 7 are examples of a case in which utterance following a previous conversation continues.

6 shows a case where a previous conversation exists, and all of the utterances of the previous conversation may be in a state in which the inspection by the inspector 34 has been completed as described above.

And with the utterance corresponding to the previous conversation, "I want to buy a mobile phone", "Which model do you want?", "Galaxy Note 10", "What color and capacity do you want?", "The capacity is 256g. But, what are the colors?" has already been entered.

One of the plurality of workers (32) said, "The capacity is 256g, but what are the colors?" It is a case of randomly entering a channel for inputting a subsequent utterance, and when the subsequent utterance is input, the corresponding channel is exited.

On the other hand, channels for inputting utterances are randomly assigned to the worker 32, and when the worker 32 exits the channel and presses the next task execution button, the greetings "Hello" are divided as shown in FIG. 7. May be provided. Accordingly, the worker 32 enters one utterance sentence “Which part can I help you with?” and then exits the channel.

When the worker 32 enters one utterance and leaves the channel (this is referred to as one turn), the channel is switched to a state of entry impossible waiting for inspection, and the operator who inputs the utterance cannot enter. do. In addition, all workers are not allowed to enter the channel while waiting for inspection for the corresponding channel.

On the other hand, the channel is to be inspected by the inspector 34, and an inspection completion button, rejection button, and rejection reason window are generated on the inspection screen. In this case, the case where the inspector 34 rejects the work may be a case where the worker 32 performs a completely irrelevant task, for example, when the worker 32 inputs an utterance that is completely irrelevant to the previous utterance.

In addition, as the inspection is completed by the inspector 34, the corresponding channel is switched to an accessible state, and as the state of the channel is switched, any one of the other workers may be allowed to enter the corresponding channel.

At this time, the operator who has entered the previous utterance cannot enter the channel again unless there is an input of the next utterance following the previous utterance by another worker. You can enter

Also, depending on the situation, when the operator re-enters the same channel, the utterance may be input as a role other than the role (role) in which the utterance was previously input. As a result, a more diverse and rich natural language conversation dataset is constructed.

Meanwhile, in the above description, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later may be applied to the method of performing the crowdsourcing project of FIGS. 1 to 7.

Hereinafter, an apparatus 100 for allocating a task for collecting various natural language conversation datasets according to an embodiment of the present invention will be described with reference to FIG. 8.

FIG. 8 is a diagram illustrating a crowdsourcing project execution apparatus 100 for collecting various natural language conversation datasets according to an embodiment of the present invention.

Referring to FIG. 8, a task assignment device 100 for collecting various natural language conversation data sets includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 transmits the work of the crowdsourcing project to a plurality of workers 32 and receives work results from the plurality of workers 32. Then, the work results input from the plurality of workers 32 are transmitted to the plurality of inspectors 34, and the inspection results are received from the plurality of inspectors 34.

A computer program for collecting a natural language conversation data set is stored in the memory 120. This is a computer program that implements the crowdsourcing project execution method for collecting the natural language conversation data set.

The processor 130 executes a program stored in the memory 120. As the processor 130 executes the program stored in the memory 120, the above-described crowdsourcing project execution method for collecting the natural language conversation data set is executed.

The job allocation apparatus 100 described with reference to FIG. 8 may be provided as a component of the above-described server.

The crowdsourcing project execution method for collecting various natural language conversation datasets according to an embodiment of the present invention described above is implemented as a computer program (or application) to be executed in combination with a computer, which is hardware, and is a computer-readable recording medium. Can be stored in.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: task assignment device
110: communication module
120: memory
130: processor

Claims (10)

As a method performed by a computer,
Setting target values of a plurality of natural language conversation data sets;
Generating a plurality of channels corresponding to the plurality of natural language conversation datasets according to the target value;
Receiving a request to participate in a crowdsourcing project from a worker;
Randomly determining any one of the plurality of channels and entering the operator into the channel;
Requesting the operator to perform a task of inputting one utterance through the channel, and receiving the utterance input from the operator; And
When the operation of inputting the utterance is completed, leaving the worker from the channel,
Each natural language conversation dataset consists of two or more speech data,
Switching the state of the channel to a state in which no entry is allowed, and requesting an inspector to inspect the utterance input from the operator of the channel; And
Upon completion of the inspection of the utterance input from the operator of the channel, further comprising the step of switching the state of the channel to an entry enabled state,
After leaving the worker from the channel, randomly determining any one channel among the plurality of channels, entering the worker into the channel, and inputting one utterance to the worker through the channel Repetitively performing the steps of requesting execution and receiving the utterance from the worker, and the step of leaving the worker from the channel when the task of inputting the utterance is completed,
After leaving the worker from the channel, randomly determining any one channel among the plurality of channels, and entering the worker into the channel, the last utterance among the plurality of channels is input by the worker. Excluding the channel, randomly determining any one channel to enter the worker,
How to do a crowdsourcing project. The method of claim 1,
Providing a condition, situation, context or scenario for inputting the utterance to the operator through the channel,
How to do a crowdsourcing project. The method of claim 1,
The step of requesting the operator to perform a task of inputting one utterance through the channel, and receiving the utterance input from the operator,
To request to perform a task of inputting the first utterance of the channel or the next utterance following the previous utterance of the channel through the channel,
How to do a crowdsourcing project. The method of claim 1,
While the inspection result for the utterance input from the operator of the channel is input by the inspector, the state of the channel is maintained in the entry impossible state,
If the inspection result passes the inspection, the state of the channel is switched to an entry-enabled state, and if the inspection result is rejected, a rework request is made to the operator who has input the utterance,
How to do a crowdsourcing project. The method of claim 1,
While the inspection result for the utterance input from the operator of the channel is input by the inspector, the state of the channel is maintained in the entry impossible state,
If the inspection result passes, the state of the channel is switched to an entry-enabled state, and if the inspection result is rejected, the utterance input by the operator is deleted, and a new operator enters the channel, Receiving input of speech,
How to do a crowdsourcing project. The method of claim 1,
The step of entering the operator into the channel,
Entering into the channel a worker different from the worker who entered the last utterance in the channel,
How to do a crowdsourcing project. The method of claim 1,
Only one worker enters the channel, and does not enter two or more workers at the same time,
How to do a crowdsourcing project. The method of claim 1,
Switching a state of a channel in which a target value of speech data of a corresponding natural language conversation data set among the plurality of channels has been verified, to a state in which entry is not possible,
How to do a crowdsourcing project. A computer program stored in a computer-readable recording medium in combination with a computer as hardware and in order to execute the crowdsourcing project execution method of any one of claims 1 to 8. In a crowdsourcing project execution device for collecting various natural language conversation datasets,
Sending the work of the crowdsourcing project to a plurality of workers, receiving work results from the plurality of workers, sending the work results input from the plurality of workers to a plurality of inspectors, and receiving the inspection results from the plurality of inspectors Communication module;
A memory storing a computer program for collecting the natural language conversation data set; And
By executing the computer program stored in the memory, target values of a plurality of natural language conversation datasets are set, a plurality of channels corresponding to the plurality of natural language conversation datasets are generated according to the target values, and the crowd of the worker According to a request to participate in a sourcing project, randomly determining any one channel among the plurality of channels, entering the worker into the channel, requesting to perform a task of inputting one utterance to the worker through the channel, A processor that receives the utterance from the worker and leaves the worker from the channel when the task of inputting the utterance is completed,
Each natural language conversation dataset consists of two or more speech data,
The processor converts the state of the channel to a non-entry state, requests an inspector to inspect the utterance input from the operator of the channel, and when the utterance received from the operator of the channel is inspected, the channel Switch the state of the to be able to enter,
After leaving the worker from the channel, the processor randomly determines any one channel among the plurality of channels, enters the worker into the channel, and inputs one utterance to the worker through the channel. Requesting to perform a task, receiving the utterance from the worker, and when the task of inputting the utterance is completed, repeatedly performing the resigning of the worker from the channel, and
After leaving the operator from the channel, randomly determining any one channel among the plurality of channels and entering the operator into the channel means that the last utterance among the plurality of channels is input by the operator. Except for, randomly determining any one channel to enter the worker,
A device for performing crowdsourcing projects.

Images