KR102169334B1 - Method for adjusting inspection standards of inspectors by comparing inspection results of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of adjusting inspection criteria of an inspector through a comparison between inspection results of a crowdsourcing-based project for creating artificial intelligence learning data. The method includes the following steps of: requesting job fulfillment by assigning a plurality of jobs of a crowdsourcing-based project (hereinafter referred to as ″project″) to a plurality of workers; receiving a plurality of job results from the plurality of workers; creating a plurality of inspector groups including a plurality of inspectors; requesting inspection fulfillment by assigning the plurality of job results to the plurality of inspector groups; receiving a plurality of inspection results about the plurality of job results from the plurality of inspectors as an inspection pass or return; selecting an adjustment target inspector from among the plurality of inspectors based on the plurality of inspection results; and changing a minimum inspection time of the adjustment target inspector.

Description

{METHOD FOR ADJUSTING INSPECTION STANDARDS OF INSPECTORS BY COMPARING INSPECTION RESULTS OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for adjusting the inspection criteria of an inspector by comparing inspection results of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

When one project is opened, a plurality of tasks are assigned to each of a plurality of workers, and each worker performs a plurality of assigned tasks to provide work results. And the provided work results are assigned to a plurality of inspectors, and the plurality of inspectors perform inspection on the work results.

At this time, in the process of performing the inspection by a plurality of inspectors, the inspection results may be different due to different inspection standards among inspectors despite the result of the work performed by the same operator. This occurs because, while the work result by the operator is verified by the inspector, the inspection result by the inspector is not verified.

In particular, if the number of inspectors participating in the project is significant, and if the inspection criteria of each inspector are all different, it may lead to a result of deteriorating the quality of the final result of the project, contrary to the intention of the inspector who conducted the inspection sincerely. .

Therefore, there is a need for a way to improve the quality of the final result of the project by verifying inaccurate inspection results caused by different inspection standards among inspectors and unifying the inspection standards among inspectors.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of adjusting the inspection criteria of an inspector by comparing inspection results of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above problems, the method of adjusting the inspection criteria of the inspector by comparing the inspection results of a crowdsourcing-based project according to an aspect of the present invention is a method performed by a computer, and includes a crowdsourcing-based project (hereinafter, "project"). Assigning a plurality of tasks to a plurality of workers and requesting the task to be performed, receiving a plurality of work results from the plurality of workers, generating a plurality of inspector groups including a plurality of inspectors, the plurality of Allocating work results to the plurality of inspector groups to request inspection performance, receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting inspection, and receiving the plurality of inspection results On the basis of, the step of selecting an inspector to be adjusted from among the plurality of inspectors and changing a minimum inspection time of the inspectors to be adjusted, the step of creating the plurality of inspector groups, the total inspectors participating in the project In the case of n (where n is a natural number of k or more), k people (however, k is a natural number of 3 or more) are selected from the n number of examiners in no order. The step of generating and requesting to perform the inspection includes requesting to perform inspection by assigning different work results of the same worker to a plurality of inspectors belonging to the same inspector group in the same number, and selecting the inspector to be adjusted, Calculating the number of initial inspection passes during the unit period for each inspector in each inspector group, and the calculated initial inspection compared to the total number of work results assigned to each inspector during the unit period for each inspector in each inspector group Calculating the ratio of the number of passes as an inspection pass throughput, calculating the average inspection pass throughput of all inspectors belonging to each inspector group for each inspector group based on the inspection pass throughput, and each inspector group The inspection pass rate of the inspector and the average inspection Including the step of selecting an inspector whose absolute value of the deviation of the pass processing rate exceeds a reference ratio as an inspector to be adjusted, and receiving the plurality of inspection results as inspection passing or rejection, the minimum inspection time has elapsed. After activating the inspection result input interface, the inspection result is inputted from the inspector.

In some embodiments of the present invention, the method further includes performing a verification process for a plurality of inspection results received from an inspector to be adjusted, and the step of performing the verification process includes: Requesting another inspector of the inspector group to perform re-examination by reassigning a job result (hereinafter, referred to as re-examination subject job result) corresponding to at least some of the inspection results received from the plurality of inspection results received from the adjustment target inspector, The step of receiving the re-examination result of the re-examination target work result from the inspector as passing or rejecting the inspection, and the inspection cost for the work result to other inspectors who input the re-examination result different from the inspection result received from the reconciliation target inspector. It may include the step of processing the payment.

In some embodiments of the present invention, when the inspection pass processing rate of the adjustment subject inspector is higher than the average inspection pass throughput rate of the inspector group to which the adjustment subject inspector belongs, the step of requesting the re-examination may include the adjustment to the other inspector. You can request re-examination by rearranging the work result corresponding to the inspection result received by the target inspector as the inspection pass.

In some embodiments of the present invention, when the inspection pass throughput rate of the adjustment subject inspector is lower than the average inspection pass throughput rate of the inspector group to which the adjustment subject inspector belongs, the step of requesting the re-review may include the adjustment to the other inspectors. You can request re-examination by rearranging the work result corresponding to the inspection result received as a reject by the target inspector.

In some embodiments of the present invention, the step of performing the verification process includes receiving a reason for changing the verification result from another inspector who has entered a re-examination result different from the inspection result received from the adjustment target inspector, and changing the inspection result It may further include the step of providing the reason to the adjustment target inspector.

In some embodiments of the present invention, in the requesting to perform the re-examination, the re-examination target work result may be distributed and rearranged according to the inspection pass processing rate of each of the other inspectors in the selected inspector group. have.

In some embodiments of the present invention, the method corresponds to the payment processing of m cases of the inspection cost for the corresponding work result to another inspector who inputs a re-examination result different from the inspection result input from the inspector to be adjusted. After the process, it may further include the step of not paying the inspection cost for the results of the m work performed by the inspector to be adjusted.

In some embodiments of the present invention, when the inspection pass processing rate of the adjustment subject inspector is higher than the average inspection pass throughput rate of the inspector group to which the adjustment subject inspector belongs, changing the minimum inspection time may include: It further comprises the step of calculating an average inspection time required for a unit period, based on the average inspection time and a deviation between the inspection pass processing rate and the average inspection pass processing rate of the adjustment target inspector, the minimum inspection of the adjustment target inspector You can increase the time.

In some embodiments of the present invention, when the inspection pass processing rate of the adjustment subject inspector is lower than the average inspection pass throughput rate of the inspector group to which the adjustment subject inspector belongs, the step of changing the minimum inspection time comprises: It further comprises the step of calculating an average inspection time required for a unit period, based on the average inspection time and a deviation between the inspection pass processing rate and the average inspection pass processing rate of the adjustment target inspector, the minimum inspection of the adjustment target inspector It can reduce the time.

A computer program according to another aspect of the present invention for solving the above-described problem executes a method of adjusting the inspection criteria of an inspector by comparing inspection results of the crowdsourcing-based project by combining with a computer as hardware, and a computer-readable recording medium Can be stored in.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, it is possible to extract examiners having different inspection criteria by comparing the inspection results of a plurality of inspectors participating in the project. In addition, it is possible to change the time limit required for performing the inspection so that inspectors with different inspection criteria can adjust the inspection criteria the same or similar to other inspectors.

In addition, it is possible to induce the relaxation of the inspection criteria of inspectors whose inspection pass rate is particularly low due to the strict inspection criteria, thereby preventing the inspection bottleneck caused by the inspector, and finally preventing the delay of the entire project schedule.

In addition, it is possible to increase the accuracy of the inspection result of the inspector and finally improve the quality of the final product of the project by inducing the increase of the inspection standard of the inspector with a particularly high inspection pass rate due to the low inspection standard.

In addition, by re-examining the inspection results of inspectors with different inspection criteria, it is possible to verify whether inaccurate inspection results caused by different inspection criteria have occurred. In other words, it is possible to verify not only the work result of the operator but also the inspection result of the inspector.

In addition, by providing appropriate compensation to the inspector who performed the re-examination, appropriate motivation may be given to the inspector who performs the verification of the inspection result.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart schematically showing a method of adjusting the inspection criteria of an inspector through comparison of inspection results of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a diagram illustrating a method of creating a plurality of inspector groups including a plurality of inspectors according to an embodiment of the present invention.
5 is a diagram illustrating a method of allocating work results of the same worker to inspectors belonging to the same inspector group according to an embodiment of the present invention.
6 is a flow chart schematically showing a method of selecting an inspector 36 to be adjusted according to an embodiment of the present invention.
7 is a diagram for explaining a method of calculating an inspection pass processing rate for each inspector in an inspector group and selecting an inspector to be adjusted.
8 is a flowchart illustrating a step of performing a verification process for a plurality of inspection results received from an inspector to be adjusted according to an embodiment of the present invention.
9 is a flowchart schematically illustrating a method of performing a verification process for a plurality of inspection results input from an inspector to be adjusted according to an exemplary embodiment of the present invention.
FIG. 10 illustrates a request for re-examination by reassigning a work result corresponding to an inspection result inputted as an inspection pass by an inspector to be adjusted to another inspector in the inspector group to which the inspector to be adjusted belongs according to an embodiment of the present invention. It is a figure shown.
11 is a view for explaining a method of paying an inspection cost for a corresponding work result to another inspector who inputs a re-examination result different from the inspector to be adjusted according to an embodiment of the present invention.
12 is a view for explaining an apparatus for adjusting an inspection standard of an inspector by comparing inspection results of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of adjusting the inspection criteria of an inspector through comparison of inspection results of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 11. Details of overlapping content related to the crowdsourcing service configuration and the crowdsourcing-based project process described above with reference to FIGS. 1 and 2 will be omitted.

Meanwhile, the steps illustrated in FIGS. 3, 6, 8 and 9 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto. .

3 is a flowchart schematically illustrating a method of adjusting an inspector's inspection criteria by comparing inspection results of a crowdsourcing-based project according to an embodiment of the present invention.

First, referring to FIG. 3, a plurality of tasks of a project are assigned to a plurality of workers, and a task is requested (S110). Then, a plurality of work results are input from a plurality of workers (S120).

Thereafter, before requesting the inspector to perform inspection on the plurality of work results, a plurality of inspector groups including a plurality of inspectors are created for a plurality of inspectors who are authorized to inspect the project (S130).

Hereinafter, a specific method of generating a plurality of examiner groups according to an embodiment of the present invention will be described with reference to FIG. 4.

4 is an exemplary diagram illustrating a method of creating a plurality of inspector groups including a plurality of inspectors according to an embodiment of the present invention.

In an embodiment of the present invention, in step S130, when the total number of inspectors participating in the project is n (however, n is a natural number greater than or equal to k), k persons among the n inspectors in any order (however, k is a natural number greater than or equal to 3) You can create multiple groups of inspectors corresponding to the number of all cases by selecting

Specifically, a plurality of inspector groups may be created by a combination, which is a method of selecting k persons out of order among n inspectors. That is, when a plurality of examiner groups belonging to each of k members are generated for n examiners, a total of nCk examiner groups are created. In this way, the inspectors participating in the project can compare the inspection results with all inspectors except for themselves.

Referring to Fig. 4, there are four inspectors who are authorized to inspect the project. At this time, as a method of creating a plurality of inspector groups including 3 inspectors (i.e., when n is 4 and k is 3), ₄ C ₃ inspector groups are created to compare the inspection results between all inspectors. do. ₄ C ₃ QC staff a plurality of groups corresponding to the number of cases shown in the first through fourth QC staff group shown in Fig.

Referring to FIG. 3 again, the plurality of task results are then assigned to the plurality of inspector groups to request the inspection (S140).

In this case, according to an embodiment of the present invention, different work results of the same worker may be assigned to a plurality of inspectors belonging to the same inspector group in the same number to request the inspection.

Specifically, each worker is matched for each inspector group generated in step S130, and a plurality of work results input by the matched worker are distributed and assigned in the same number to request inspection. At this time, the number of identical work results to which a plurality of inspectors included in the same group are assigned may be determined based on the total number of work and the total number of inspectors of the project.

5 is an exemplary view showing a method of allocating work results of the same worker to inspectors belonging to the same inspector group according to an embodiment of the present invention.

Referring to FIGS. 4 and 5, a first group of the plurality of inspector groups generated in step S130 includes inspector 1, inspector 2, and inspector 3. At this time, a total of 9 work results of worker 1 matched to the first group. Therefore, three inspectors belonging to the first group may be assigned three different work results each and request to perform the inspection.

In addition, when 9 new work results are input by worker 1, the work results of worker 1 may be assigned to a second group among a plurality of inspector groups. According to this, three different work results are assigned to inspector 1, inspector 2, and inspector 4, respectively.

Likewise, the first inspector group may be matched with worker 2 following the 9 work results of worker 1 to receive the assignment of 9 work results of worker 2.

That is, it is sufficient that each inspector included in the same inspector group receives the same number of different work results of the same worker and performs the inspection. This is because, as will be described later, the inspection criteria can be compared between each inspector included in one inspector group.

Of course, the comparison of the inspection criteria is based on the premise that the work quality of the worker who enters the work result does not change rapidly over time.

On the other hand, in an embodiment of the present invention, only when the number of work results input by the worker exceeds the preset reference number, different work results of the corresponding worker can be assigned to the same number and request to perform inspection. have.

For example, if the number of preset criteria is 20, only the worker who has entered a plurality of job results exceeding 20, matches any one of a plurality of inspector groups with the worker, and The results of a plurality of tasks are allocated to a plurality of inspectors belonging to the matched inspector group in the same number.

Referring back to FIG. 3, after step S140, a plurality of inspection results for a plurality of work results are inputted from a plurality of inspectors as passing or rejecting the inspection (S150). At this time, a plurality of inspectors belonging to the same inspector group inputs the inspection results as the same number of inspection results for the work results related to the worker matched to the inspector group.

When a plurality of inspection results are input, an inspector to be adjusted is selected from among the plurality of inspectors based on the input (S160).

Specifically, after the work results of the same worker are distributed and assigned, the inspectors 36 to be adjusted are selected based on the inspection results of the plurality of inspectors who have performed the inspection.

Herein, the subject inspector 36 means an inspector whose inspection criteria are different among a plurality of inspectors belonging to the same inspector group on the basis of the inspection result, and thus the need to adjust the inspection criteria is required.

On the other hand, an inspector may belong to multiple inspector groups. For example, assume that inspector 1 belongs to the first inspector group and the second inspector group. In this case, when the examiner 1 is selected as the adjustment target examiner 36 in the first examiner group, it is common that the second examiner group is also selected as the adjustment target examiner 36. This is because most of the inspectors participating in the project have the same or extremely similar inspection criteria. In other words, the inspector to be adjusted statistically responds to the outlier.

However, even if examiner 1 is selected as the adjustment target examiner 36 in the first examiner group, it is not necessarily selected as the adjustment target examiner 36 in the second examiner group. This is because the workers who performed the work results assigned to each tester group are different, and according to the work performance results of the workers, the tester's test criteria may also be different. Therefore, by assigning only the results of work by the same operator to a plurality of inspectors belonging to the same inspector group and requesting to perform the inspection, the inspection results are repeatedly compared to induce a unified inspection standard among multiple inspectors in the same inspector group.

Hereinafter, a specific method of selecting an inspector 36 to be adjusted based on a plurality of inspection results will be described with reference to FIG. 6.

6 is a flow chart schematically showing a method of selecting an inspector to be adjusted according to an embodiment of the present invention.

Referring to FIG. 6, first, the number of first inspection passes during a unit period is calculated for each inspector in each inspector group (S171).

The number of first inspection passes refers to the number of work results in which the inspection result is input as the inspection pass when performing the initial inspection among the plurality of work results performed by the inspector. In other words, it means the number of job results that have not been rejected and requested the operator to perform re-inspection.

For example, assume that an inspector has been assigned 10 work results and performed the inspection. Among them, the inspector entered the inspection results for the eight work results as inspection passed and the inspection results for the remaining two work results as rejection after the initial inspection was performed. In addition, as a result of re-reviewing the rework results of the remaining two cases that were rejected, the inspection results for the rework results of one case were passed as inspection, and the inspection results of the rework results of the remaining one case were entered as rejection. At this time, the number of first inspection passes by the inspector corresponds to eight cases.

Referring back to Fig. 6, after calculating the number of initial inspection passes during the unit period in step S171, the calculated initial inspection passes compared to the total number of work results assigned to each inspector during the unit period for each inspector in each inspector group. The ratio of the number of cases is calculated as the inspection pass processing rate (S172).

FIG. 7 is a diagram for explaining a method of calculating an inspection pass processing rate for each inspector in a specific inspector group.

Referring to FIG. 7, it can be seen that the inspector 1, the inspector 2, the inspector 3, the inspector 4, and the inspector 5 belong to the same inspector group. At this time, assuming that the unit period is 1 day, the rate of passing inspection of inspector 1 is calculated as 0.6, which is the ratio of 6 cases, which is the number of first inspection, to 10 cases, which is the total number of work results assigned to inspector 1. Similarly, when calculating each inspector's pass through, the pass rate of inspector 2 and the pass rate of inspector 3 are 0.4, the pass rate of inspector 4 is 0.5, and the pass rate of inspector 5 is 0.8. You can see that it is.

Referring back to FIG. 6, after calculating the inspection pass processing rate of each inspector (S172), the average inspection pass processing rate of all inspectors belonging to each inspector group for each inspector group is calculated based on this (S173). In addition, for each inspector group, an inspector whose absolute value of the deviation between the inspection pass processing rate and the average inspection pass processing rate of each inspector exceeds the reference ratio is selected as the subject inspector 36 (S174).

That is, in order to extract inspectors with different inspection criteria within the same inspector group, the inspection pass rate of each inspector is compared with the average inspection pass rate of the inspector group. And, if the inspection pass processing rate is significantly out of the standard of the average inspection pass processing rate of the inspector group, it is determined that the inspection criteria of the inspector of the inspection pass processing rate are different from those of other inspectors, and the subject inspector 36 is selected.

Referring to FIG. 7, referring to FIG. 7, the average inspection pass throughput of the inspector group shown in FIG. 7 is calculated as 0.54. At this time, assuming that the reference ratio is 0.2 (20%), if the absolute value of the deviation between the inspection pass rate and the average verification pass rate (0.54) exceeds 0.2, it is selected as the inspector 36 to be adjusted. Accordingly, the inspector 5 whose absolute value of the deviation between the inspection pass processing rate and the average inspection pass processing rate is 0.26 is selected as the inspector 36 to be adjusted for the corresponding inspector group.

Since the absolute value of the deviation between each inspector's inspection pass rate and the average inspection pass rate is used, even if a specific inspector's inspection pass rate is less than the average inspection pass throughput rate, it can be selected as the subject inspector 36 for adjustment of the relevant inspector group. have. In FIG. 7, the inspection pass rate of the inspector 2 and inspector 3 is less than the average inspection pass rate, but the absolute value of the deviation is less than 0.2, so that the inspector to be adjusted is not selected.

On the other hand, if the inspection criteria of a plurality of inspectors belonging to the same group are the same or similar, the absolute value of the deviation between the inspection pass rate and the average inspection pass rate of each inspector exists within the standard ratio, so that the subject inspector 36 is It may not be selected.

Referring back to FIG. 3, when the adjustment target examiner 36 is selected in step S160, the minimum inspection time of the selected adjustment target examiner 36 is changed (S170).

In each project, in order to prevent the inspector from performing unscrupulous inspection, it is possible to set the minimum inspection time that should be taken to perform inspection of one work result. For example, if the minimum inspection time is set to 10 seconds, the inspector must perform inspection on one work result for at least 10 seconds. It is not a problem to take more than 10 seconds. And after the minimum inspection time of 10 seconds has passed, the inspector can complete the inspection. In other words, you can enter the inspection result.

On the other hand, in step S170, by adjusting the minimum inspection time of the adjustment subject inspector 36, the adjustment subject inspector 36 is induced to adjust the inspection criteria.

For example, it is assumed that the inspection pass processing rate of the adjustment subject inspector 36 is lower than the average inspection pass processing rate of the group to which the adjustment subject inspector 36 belongs. At this time, the fact that the inspection pass rate of the subject inspector 36 is lower than the average inspection pass rate means that the inspection criteria of the subject inspector 36 are different from the inspection criteria of other inspectors in the group, as well as the adjustment. It also implies that the subject inspector (36) is performing the inspection too closely. If such a situation is repeated, a bottleneck of the inspection performance occurs from the inspector 36 to be adjusted, which may lead to a problem that the schedule of the entire project is delayed.

Therefore, by changing the minimum inspection time of the adjustment subject inspector 36, the adjustment subject inspector 36 changes the inspection time required to correspond to the changed minimum inspection time, so that the existing inspection standards are changed to other inspectors and inspection standards. You might be able to induce adjustments similar to As a result, it is to induce the reviewer 36 to be adjusted to relax the inspection criteria within the quality range of the inspection result targeted by the project.

Hereinafter, a specific method of adjusting the minimum inspection time of the subject inspector 36 will be described.

In one embodiment of the present invention, when the inspection pass processing rate of the adjustment target examiner 36 is higher than the average inspection pass throughput rate of the examiner group to which the adjustment target examiner belongs, step S180 is performed on an average for a unit period of the adjustment target examiner 36 It further includes a step of calculating the time required for inspection (not shown), and the minimum inspection time for the inspector subject to adjustment can be increased based on the average inspection time required and the deviation between the inspection pass rate and the average inspection pass rate of the subject inspector. have.

For example, referring again to FIG. 7, the deviation between the inspection pass processing rate and the average inspection pass processing rate of the inspector 5, which is the subject inspector 36, is +0.26. At this time, assuming that the average inspection time of inspector 5 is 1 minute, inspector 5 should take an average inspection time of 1.26 minutes to perform inspection corresponding to the average inspection pass throughput. At this time, assuming that the minimum inspection time of inspector 5 is 20 seconds, the changed minimum inspection time of inspector 5 is as follows.

Changed minimum inspection time = 20 seconds X 1.26 minutes/1 minute

Therefore, the minimum inspection time for inspector 5 (the inspector subject to adjustment) is changed to 25.20 seconds.

On the other hand, in an embodiment of the present invention, when the inspection pass processing rate of the adjustment target examiner 36 is lower than the average inspection pass processing rate of the examiner group to which the adjustment target examiner 36 belongs, step S180 is performed by the adjustment target examiner 36 ) Further comprising a step (not shown) of calculating the average inspection time for a unit period, and the adjustment target based on the average inspection time required and the deviation between the inspection pass processing rate and the average inspection pass processing rate of the adjustment target inspector 36 It is possible to reduce the minimum inspection time of the inspector 36.

For example, it is assumed that the inspection pass processing rate of the subject inspector 36 is 0.3, the average inspection pass throughput rate of the inspector group to which the subject inspector belongs is 0.5, and the average inspection time of the inspector subject to adjustment 36 is 1 minute. . At this time, the deviation between the inspection pass processing rate and the average inspection pass processing rate of the subject inspector 36 is -0.2. In order for the subject inspector 36 to perform the inspection with the inspection pass processing rate of 0.5, it will take 48 seconds, which is equivalent to 80% of the average inspection time required. At this time, assuming that the minimum inspection time of the inspector 36 to be adjusted is 20 seconds, the changed minimum inspection time is as follows.

Changed minimum inspection time = 20 seconds X 48 seconds/60 seconds

Therefore, the minimum inspection time of the inspector 36 to be adjusted is changed to 16 seconds.

Meanwhile, in an exemplary embodiment of the present invention, in step S150, after the minimum inspection time has elapsed, the inspection result input interface may be activated to receive the inspection result from the inspector. Specifically, when the time required to perform the inspection by the inspector is less than the minimum inspection time, the inspection result input interface is deactivated until the minimum inspection time. That is, the inspector cannot input the inspection result until the minimum inspection time is elapsed, so that the inspector observes the minimum inspection time and takes the inspection execution time corresponding to the minimum inspection time, thereby inducing an appropriate inspection performance.

On the other hand, according to an embodiment of the present invention, a verification process for a plurality of inspection results input from the adjustment target inspector 36 may be performed. This is because the result of the inspection performance may be different due to the different inspection criteria of the inspector 36 to be adjusted from the other inspectors, and thus it must be corrected again. For example, if another inspector in the group to which the adjustment subject inspector 36 belongs has performed the inspection, the result of the work of sufficient quality to pass the inspection is assigned to the subject inspector 36 whose inspection standards are too strict and rejected. It corresponds.

Hereinafter, the verification process for the inspection result of the adjustment subject inspector 36 according to an embodiment of the present invention will be described in detail.

8 is a flowchart illustrating a step of performing a verification process for a plurality of inspection results received from an inspector to be adjusted according to an embodiment of the present invention.

9 is a flowchart schematically illustrating a method of performing a verification process for a plurality of inspection results input from an inspector to be adjusted according to an exemplary embodiment of the present invention.

Referring to FIG. 8, according to an embodiment of the present invention, the method of adjusting the review criteria of the reviewer through comparison of the review results of the above-described project proceeds a verification process for a plurality of inspection results input from the reviewer to be adjusted 36 It may further include a step (S180).

At this time, a specific method of performing the verification process is as follows.

First, referring to FIG. 9, a work result corresponding to at least some of the plurality of inspection results input from the adjustment subject inspector 36 to another inspector in the selected inspector group (hereinafter , Re-review target task result) is requested to perform re-review (S181).

On the other hand, according to an embodiment of the present invention, if the inspection pass processing rate of the adjustment target examiner 36 is higher than the average inspection pass processing rate of the examiner group to which the adjustment target examiner 36 belongs, step S181 is applied to another examiner. The work result corresponding to the inspection result received from the inspector 36 through the inspection pass may be rearranged and re-examination may be requested.

The fact that the inspection pass rate of the subject inspector 36 is higher than the average inspection pass rate of the inspector group means that the inspection performance rate of the inspector subject to adjustment 36 is relatively faster than the other inspectors in the inspector group. For this reason, in another aspect, it may mean that the inspection standard of the inspector 36 to be adjusted is set lower than that of the other inspectors in the inspector group.

Eventually, due to the low inspection standard of the inspector 36 to be adjusted, there may be a case in which the result of the work that would have been rejected if assigned to another inspector passes the inspection. This is not caused by poor inspection by the inspector 36 to be adjusted. It was only caused because the inspection criteria of the inspector 36 subject to adjustment were different from those of other inspectors. However, if this situation is repeated, it can lead to a deterioration in the quality of the final product of the project.

Therefore, in order to prevent such a situation from occurring, in the verification process step, only the work result corresponding to the inspection result received as the inspection pass from the adjustment target inspector 36 is reassigned to another inspector and re-review is requested.

FIG. 10 is a diagram showing a request for re-examination by reassigning a work result corresponding to an inspection result inputted by an inspector to be adjusted to another inspector of an inspector group to which an inspector to be adjusted belongs.

Referring back to FIG. 7, it can be seen that the average inspection pass throughput rate of the inspector group to which inspector 5 belongs is 0.54. At this time, the inspection pass rate of inspector 5 is 0.8, which is higher than the average inspection pass rate. In this case, according to the above-described embodiment of the present invention, among the inspection results input by the inspector 5, the work result corresponding to the inspection result input as the inspection pass is reassigned to another inspector in the inspector group and re-examination is requested.

Referring to FIG. 10, the number of inspection results received from inspector 5 is a total of 9 cases. At this time, out of the total number of inspection results of 9 cases, 8 work results inputted through inspection are reassigned to 4 other inspectors (Inspector 1, Inspector 2, Inspector 3, Inspector 4) of the inspector group and request re-inspection.

On the other hand, if the inspection pass processing rate of the adjustment subject inspector 36 is lower than the average inspection pass processing rate of the inspector group to which the adjustment subject inspector 36 belongs, step S181 is returned to another inspector from the adjustment subject inspector 36 You can request re-inspection by rearranging the work result corresponding to the received inspection result.

The fact that the inspection pass rate of the subject inspector 36 is lower than the average inspection pass rate of the inspector group means that, as described above, the inspection performance rate of the subject inspector 36 is relatively slower than that of other inspectors in the inspector group. . For this reason, in another aspect, it may mean that the inspection standard of the subject inspector 36 is set higher than that of the other inspectors in the inspector group. In other words, it means that the inspector 36 to be adjusted has strictly set the inspection criteria.

Therefore, there may be cases in which work results that would have passed the inspection had been assigned to other inspectors were rejected. That is, despite the quality of work results desired by the client of the project, the rejected work results may exist due to the strict inspection criteria of the inspector 36 to be adjusted. This can also affect the amount of final product of the project. Therefore, in order to prevent this, only the work result corresponding to the inspection result received as a rejection from the adjustment subject inspector 36 is reassigned to another inspector and re-examination is requested.

Meanwhile, according to an embodiment of the present invention, the re-examination target work result may be distributed and reassigned according to the inspection passing rate of each of the other inspectors in the selected inspector group by the inspector 36 to be adjusted.

Specifically, in order to prevent delays in the schedule of the entire project due to the progress of the verification process, more re-examination target work results are allocated to inspectors with a fast inspection performance than those with a low inspection performance speed. This is because, as described above, the inspection pass rate of inspectors is proportional to the inspection performance speed of each inspector.

Referring to FIG. 9 again, after a request to perform re-review is requested (S181), a re-review result for a work result subject to re-examination is input from another inspector as a pass or reject (182). In addition, the inspection cost for the work result is paid to another inspector who inputs a re-examination result different from the inspection result input from the adjustment subject inspector 36 (S183).

Specifically, for the inspection result in which the inspection result is incorrectly entered due to the different inspection criteria of the inspector 36 to be adjusted, the inspection cost for the work result is paid in exchange for the inspection performance to another inspector who has entered the correct inspection result. do.

FIG. 11 is a diagram for explaining a method of paying an inspection fee for a corresponding work result to another inspector who has input a re-examination result different from that of an inspector to be adjusted.

Referring to FIG. 11, as a result of requesting re-examination by rearranging the work result corresponding to the inspection result input by the inspector 5 selected as the inspector to be adjusted 36, inspector 3 and inspector 4 respectively Different retest results were entered. At this time, inspector 3 entered the inspection results as rejecting all of the two reassigned work results, and inspector 4 entered the inspection results for one of the two reassigned work results as rejection. Therefore, inspector 3 will be paid the inspection cost for two work results, and inspector 4 will be paid the inspection cost for one work result.

In the course of the verification process, other inspectors in the inspector group to which the subject inspector 36 belongs must separately perform the inspection in the other inspector groups to which they belong. In other words, it is necessary to perform an inspection on the results of work performed by other workers assigned from different inspector groups. Alternatively, the inspection process for the inspection results of the other inspectors to be adjusted 36 selected from the other inspectors group must proceed. Therefore, by providing reasonable compensation to other inspectors who performed accurate re-review in the course of the verification process, it will be possible to motivate them to perform re-review of the verification process.

In addition, although not clearly shown, as an embodiment of the present invention, the cost of inspection of m cases for the corresponding work result to another inspector who inputs a re-examination result different from the inspection result input from the adjustment target inspector 36 after step S183. In response to the payment processing, the step (not shown) of not paying the inspection cost for the results of m tasks performed by the adjustment target inspector 36 after the verification process may be further included.

Referring to FIG. 11, the inspection cost for the three work results paid to the inspector 3 and inspector 4, which input the re-examination result different from the inspector 5, is not paid to the inspector 5. In other words, the inspection cost is not redundantly paid for the inspection of the same work result. Alternatively, if the inspection fee has already been paid to inspector 5, the inspection fee may not be paid for the results of other three work that must be paid to inspector 5 in the future. Through this, it is possible to actively induce the adjustment of the inspection criteria of the inspector 36 to be adjusted, and above all, it is possible to prevent the inspection cost budget calculated for the project from being exceeded by the payment of inspection costs paid as the verification process proceeds.

On the other hand, although not clearly shown, as an embodiment of the present invention, step S180 is a step of receiving the reason for change of the inspection result from another inspector who inputs a re-examination result different from the inspection result input from the adjustment target inspector 36 (not shown) ) And a step (not shown) of providing the reason for the change of the inspection result to the adjustment target inspector 36.

Referring to FIG. 11 again, the reason for change of the inspection result is inputted from the inspector 3 and inspector 4, who input the re-examination result different from that of the inspector 5, and this is provided to inspector 5, which is the subject inspector 36. This is to provide the reason for the change of the inspection results so that inspector 5 can clearly understand the difference between the inspection criteria of other inspectors and his or her inspection criteria, and adjust his or her inspection criteria appropriately.

Meanwhile, in the above description, steps S110 to S183 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 12 to be described later can also be applied to the method of adjusting the inspection criteria of the inspector through comparison of inspection results of the crowdsourcing-based project of FIGS. 1 to 11.

Hereinafter, an apparatus for adjusting the inspection criteria of an inspector through comparison of inspection results of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIG. 12.

12 is a view for explaining an apparatus for adjusting an inspection standard of an inspector by comparing inspection results of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 12, an apparatus 200 (hereinafter, referred to as an apparatus) for adjusting the inspection criteria of an inspector through comparison of inspection results of a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The communication module 210 transmits the work results received from the plurality of workers 32 to the plurality of inspectors 34 to request inspection, and receives the inspection results from the plurality of inspectors 34. At this time, for the rejected inspection result, a corresponding rework is transmitted to the worker 32 to request rework, and the rework result is received from the worker 32.

The memory 220 stores a program for selecting an inspector 36 to be adjusted by comparing the inspection results of the project based on the data received from the communication module 210 and changing the minimum inspection time of the inspector 36 to be adjusted. do.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the method of adjusting the review criteria of the reviewer by comparing the review results of the crowdsourcing-based project described with reference to FIGS. 3 to 11 may be performed.

The device 200 described with reference to FIG. 12 may be provided as a component of the above-described server.

The method for adjusting the inspection criteria of the inspector through comparison of inspection results of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and is computer-readable. It can be stored on a recording medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
36: Inspector to be adjusted

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, “project”) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Generating a plurality of inspector groups including a plurality of inspectors;
Assigning the plurality of work results to the plurality of inspector groups and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Selecting an inspector to be adjusted from among the plurality of inspectors based on the plurality of inspection results; And
Including the step of changing the minimum inspection time of the inspector to be adjusted,
The step of generating the plurality of inspector groups,
If the total number of inspectors participating in the project is n (however, n is a natural number of more than k), k persons (however, k is a natural number of more than 3) are selected from the n number of inspectors in any order. Create a corresponding group of inspectors,
The step of requesting to perform the inspection,
Requests to perform the inspection by assigning the same number of different work results of the same worker to a plurality of inspectors belonging to the same inspector group,
The step of selecting an inspector to be adjusted,
Calculating the number of first inspection passes during a unit period for each inspector in each inspector group;
Calculating a ratio of the calculated number of first inspection passes to the total number of work results assigned to each inspector during the unit period for each inspector in each inspector group as an inspection pass processing rate;
Calculating an average inspection pass processing rate of all inspectors belonging to each inspector group for each inspector group based on the inspection pass throughput rate; And
For each inspector group, selecting an inspector as an adjustment target inspector in which an absolute value of a deviation between the inspection pass processing rate and the average inspection pass processing rate of each inspector exceeds a reference ratio,
The step of receiving the plurality of inspection results as inspection pass or rejection,
After the minimum inspection time has elapsed, the inspection result input interface is activated to receive the inspection result from the inspector,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 1,
Further comprising the step of performing a verification process for the plurality of inspection results received from the adjustment target inspector,
The step of performing the verification process,
Perform re-review by rearranging the work results corresponding to at least some of the plurality of inspection results input from the adjustment target inspector (hereinafter, the work results subject to re-examination) to other inspectors in the inspector group selected by the adjustment subject inspector Requesting;
Receiving a re-examination result for the re-examination target work result from the other inspector as passing or rejecting the inspection; And
Comprising the step of paying an inspection cost for the work result to another inspector who inputs a re-examination result different from the inspection result received from the adjustment target inspector,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 2,
When the inspection pass processing rate of the adjustment target inspector is higher than the average inspection pass processing rate of the inspector group to which the adjustment target inspector belongs,
The step of requesting to perform the re-examination,
Requesting the other inspector to perform re-examination by rearranging the work result corresponding to the inspection result received as the inspection pass from the adjustment target inspector,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 2,
When the adjustment target inspector's inspection pass throughput is lower than the average inspection pass throughput rate of the inspector group to which the adjustment target inspector belongs,
The step of requesting to perform the re-examination,
Requesting the other inspector to perform re-examination by rearranging the work result corresponding to the inspection result received as a rejection from the subject inspector to be adjusted,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 2,
The step of performing the verification process,
Receiving a reason for changing the inspection result from another inspector who inputs a re-examination result different from the inspection result received from the adjustment subject inspector; And
Further comprising the step of providing a reason for changing the inspection result to the inspection subject to be adjusted,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 2,
The step of requesting to perform the re-examination,
The adjustment target examiner distributes and rearranges the re-examination target work result according to the inspection pass processing rate of each of the other inspectors in the selected inspector group,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 2,
In response to the payment of m inspection costs for the results of the work to another inspector who inputs a re-examination result that is different from the inspection result input from the inspector to be adjusted, m inspected by the inspector to be adjusted after the verification process Further comprising the step of not paying the inspection fee for the proposed work result,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 1,
When the inspection pass processing rate of the adjustment target inspector is higher than the average inspection pass processing rate of the inspector group to which the adjustment target inspector belongs,
The step of changing the minimum inspection time,
Further comprising the step of calculating the average inspection time required for the unit period of the inspection subject to be adjusted,
Increasing the minimum inspection time of the adjustment target inspector based on the average inspection time required and a deviation between the inspection pass processing rate and the average inspection pass processing rate of the adjustment target inspector,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. The method of claim 1,
When the adjustment target inspector's inspection pass throughput is lower than the average inspection pass throughput rate of the inspector group to which the adjustment target inspector belongs,
The step of changing the minimum inspection time,
Further comprising the step of calculating the average inspection time required for the unit period of the inspection subject to be adjusted,
Decreasing the minimum inspection time of the adjustment target inspector based on the average inspection time required and a deviation between the inspection pass processing rate and the average inspection pass processing rate of the adjustment target inspector,
How to adjust the inspection standards of inspectors by comparing inspection results of crowdsourcing-based projects. A computer program stored in a computer-readable recording medium to perform a method of adjusting the inspection criteria of an inspector by comparing the inspection results of the crowdsourcing-based project according to any one of claims 1 to 9 in combination with a computer.

Images