JP7236694B2 - Information processing method and information processing system - Google Patents

Description

The present disclosure relates to an information processing method and an information processing system.

In recent years, efforts have been made to detect objects from images using deep learning. In order to accurately detect an object from images, for example, a large number of images are collected and an operator creates training data from the large number of images. Therefore, the accuracy of object detection for images varies depending on the skill of the operator.

As a technique related to worker skill, for example, there is a technique disclosed in Patent Document 1. The skill evaluation device of Patent Literature 1 compares the position and area of the mark with learning information to calculate the degree of similarity in an operation of marking a portion including a person's face in an image as target content. The skill evaluation device evaluates that a task with a higher degree of similarity to learning information was performed by a highly skilled worker. As a result, a highly skilled worker can be selected from a plurality of workers. Also, the evaluation result is fed back to the worker.

JP 2015-200985 A

With the conventional technology disclosed in Patent Document 1, it is difficult to efficiently improve a worker's skill according to individual characteristics.

Therefore, the present disclosure provides an information processing method and an information processing system that can efficiently improve annotation skills of workers according to individual characteristics.

In order to achieve the above object, an information processing method according to an embodiment of the present disclosure provides an annotation index, which is an index for evaluating the accuracy of a work in which a worker adds annotation information to an object appearing in an image. Acquiring skill -related information , acquiring an annotation image in which the object specified from the annotation skill-related information is captured, acquiring an annotation result of the annotation image by the worker, and acquiring the annotation image. updating the information related to the annotation skill using the correct annotation information corresponding to the object appearing in the image and the annotation result.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media.

According to the information processing method and the like of the present disclosure, it is possible to efficiently improve the annotation skills of workers according to their individual characteristics.

1 is a block diagram illustrating an information processing system and an input/output unit according to Embodiment 1. FIG. FIG. 2 is a diagram illustrating transmission of information in the information processing system according to the first embodiment. 3A is a diagram illustrating first annotation information according to Embodiment 1. FIG. 3B is a diagram illustrating annotation skills according to Embodiment 1. FIG. 3C is a diagram illustrating second annotation information according to Embodiment 1. FIG. FIG. 4 is a diagram illustrating the operation of the information processing system according to the first embodiment. FIG. 5 is a diagram exemplifying the work of adding annotation information to an annotation image. FIG. 6 is a block diagram illustrating an information processing system and an input/output unit according to the second embodiment. FIG. 7 is a diagram illustrating transmission of information in the information processing system according to the second embodiment. FIG. 8 is a diagram illustrating the operation of the information processing system according to the second embodiment.

An information processing method according to one embodiment of the present disclosure uses a computer to acquire annotation skills of a worker who assigns annotation information to sensing data, and an annotation having characteristics of sensing data specified from the annotation skills. sensing data for annotation is acquired, an annotation result for the sensing data for annotation by the worker is acquired, and the annotation skill is updated using correct annotation information corresponding to the feature of the sensing data and the annotation result.

In this way, by determining the sensing data for annotation according to the annotation skill of the worker, annotation training according to the individual characteristics of the worker becomes possible. For example, sensing data that the operator is not good at is determined as sensing data for annotation. In addition, by automatically updating annotation skills, it is possible to continuously improve annotation skills. Therefore, the annotation skill of the worker can be efficiently improved according to the individual characteristics.

An information processing method according to an aspect of the present disclosure includes a first acquisition unit that acquires sensing data for annotation that has characteristics of sensing data that are specified from annotation skills of a worker who imparts annotation information to sensing data; a second acquisition unit that acquires an annotation result for the annotation sensing data by the worker; and a processing unit that updates the annotation skill using correct annotation information corresponding to features of the sensing data and the annotation result. .

This information processing method also has the same effect as described above.

In the information processing method according to one aspect of the present disclosure, the annotation skill includes a skill related to designation of the annotation information.

In this way, by targeting skills directly related to the quality of annotation information, it is possible to effectively improve the quality of a learning model that learns using teacher data generated from annotations.

In the information processing method according to an aspect of the present disclosure, the designation of the annotation information includes at least one of position designation, size designation, and shape designation of the annotation.

In this way, the operator can be trained to perform at least one of annotation position specification, size specification, and shape specification. For this reason, the accuracy of the task of adding annotation information to the sensing data for annotation is improved by training the operator to specify annotations that the operator is not good at.

In the information processing method according to an aspect of the present disclosure, specifying the annotation information includes specifying a label of the annotation.

In this way, the operator can be trained to designate annotations for each annotation label (for example, class). Therefore, the accuracy of the task of adding annotation information to the sensing data for annotation is improved by training the annotation class that the operator is not good at.

In the information processing method according to an aspect of the present disclosure, the annotation result includes the time taken to specify the annotation information, and the annotation skill includes the time taken to specify the annotation information.

According to this, the worker can perform training for improving the time required for the work for specifying the annotation. Therefore, it is possible to improve the skill of the worker, ie, the speed of the task of adding annotation information to the sensing data for annotation.

In the information processing method according to an aspect of the present disclosure, the sensing data is image data, and the annotation sensing data is annotation image data.

According to this, it is possible to efficiently improve annotation skills even for image data having a large amount of information and having individual differences in improvement of annotation skills.

An information processing method according to an aspect of the present disclosure specifies the correct annotation information from the annotation skill, and includes the annotation in which an object corresponding to the specified correct annotation information is displayed from among the plurality of annotation image data. The image data for annotation is acquired by selecting the image data for annotation.

According to this, it is possible to select image data for annotation with teacher data that leads to improvement of annotation skill. For example, image data for annotation that is required to add annotation information that the operator is not good at is selected. Workers can be trained for work that they are not good at.

An information processing method according to an aspect of the present disclosure specifies the correct annotation information from the annotation skill, and generates image data showing an object corresponding to the specified correct annotation information, thereby obtaining the annotation image data. get.

According to this, the operator can be trained even if there is no image data for annotation suitable for improving the annotation skill.

In the information processing method according to an aspect of the present disclosure, generating the image data includes synthesizing image data showing an object corresponding to the identified correct annotation information with another image data.

According to this, it is possible to efficiently generate annotation image data with teacher data.

The information processing method according to an aspect of the present disclosure further presents the operator with a difference between the correct annotation information and the annotation result.

According to this, the worker can sensuously understand his own annotation skill from the difference between the correct annotation information and the annotation result.

The information processing method according to an aspect of the present disclosure further presents the worker with the updated annotation skill.

According to this, the worker can grasp the state of his own annotation skill. For example, it is possible to let the worker know whether the annotation skill has improved, whether it has not changed, or whether it has deteriorated. Accordingly, it is possible to directly or indirectly encourage the worker to take actions that improve the annotation skill.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below is a specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as arbitrary constituent elements.

Each figure is a schematic diagram and is not necessarily strictly illustrated. Moreover, in each figure, the same code|symbol is attached|subjected to the substantially same structure, and the overlapping description is abbreviate|omitted or simplified.

An information processing method and an information processing system according to embodiments of the present disclosure will be described below.

(Embodiment 1)
[composition]
FIG. 1 is a block diagram illustrating an information processing system 1 and an input/output unit 40 according to the first embodiment. FIG. 2 is a diagram illustrating transmission of information in the information processing system 1 according to the first embodiment.

As shown in FIGS. 1 and 2, an operator (also referred to as an annotator) annotates sensing data for machine learning. For example, the sensing data is image data, and an operator may select an image captured by an imaging device or an image synthesized by CG (computer graphics) (hereinafter referred to as an annotation image), and an object appearing in the annotation image. Adds annotation information about . The information processing system 1 evaluates the accuracy of annotation information added to annotation images. The information processing system 1 evaluates the worker (determines annotation skill) according to the annotation image to which the worker has attached the annotation information, and assigns the annotation image according to the evaluation to the worker. let it happen Objects may be, for example, people, animals, vehicles, equipment, or anything else.

Annotation here means adding relevant metadata as annotations (labels) to a specific object associated with an annotation image, and is described in text. An annotation image is an image in which an object to be worked by a worker is reflected, and is an image for the worker to add annotation information to the object.

The information processing system 1 is connected to an input/output unit 40 for an operator to add annotation information. The input/output unit 40 is, for example, a personal computer, and receives input from the operator via a screen such as a browser displayed on the personal computer. The information processing system 1 includes a first processing section 10 , a second processing section 20 and a third processing section 30 .

The first processing unit 10 has a storage unit 11 that stores information indicating an image for annotation, first annotation information, and information indicating an annotation skill of an operator. The first annotation information referred to here is exemplary annotation information (correct data). The first annotation information is an example of correct annotation information.

Note that, when an annotation image including an object corresponding to the annotation skill does not exist in the storage unit 11, the first processing unit 10 generates an annotation image including an object corresponding to the annotation skill. The generation of this annotation image includes synthesizing an image showing an object corresponding to the identified second annotation information with another image.

An example of the first annotation information will be described with reference to FIG. 3A. 3A is a diagram illustrating first annotation information according to Embodiment 1. FIG.

As shown in FIG. 3A, the first annotation information includes the position on the x and y coordinates where the object is arranged, the size of the object, and the size of the object in the annotation image in which the object (corresponding to the class in FIG. 3A) is captured. , the shape of an object, and the time required to add annotation information. When a plurality of objects exist in one annotation image, the first annotation information includes each text data corresponding to the plurality of objects.

An example of information indicating annotation skills will be described with reference to FIG. 3B. 3B is a diagram illustrating annotation skills according to Embodiment 1. FIG.

As shown in FIG. 3B, the annotation skill is a skill related to the designation (addition) of annotation information, that is, in the task of an operator adding annotation information to an annotation image, the accuracy of each object, the accuracy rate of options, It is the skill of the worker that indicates the work time required for adding annotation information. The annotation skill is, for example, an index that evaluates the accuracy of a worker's task of adding annotation information to an annotation image. An annotation skill is determined for each object. Accuracy means the accuracy of annotation information, such as the position, range (size), shape, etc., at which annotation information is assigned to an object included in an annotation image.

The first processing unit 10 acquires the annotation skill of a worker who gives annotation information. The first processing unit 10 outputs to the input/output unit 40 an image for annotation having image features specified from the annotation skill of the operator. A feature of an image is, for example, an object reflected in the image. The input/output unit 40 acquires an image for annotation and displays the image for annotation in order to allow the operator to attach annotation information.

Further, when acquiring information indicating the annotation skill of the worker evaluated by the evaluation unit 33 described later, the first processing unit 10 updates the annotation skill of the worker. Note that the first processing unit 10 may acquire information indicating the annotation skill of the worker evaluated by another device from another device.

The second processing unit 20 has a first acquisition unit 21 and a provision unit 22 .

The first acquisition unit 21 acquires, through the input/output unit 40 , an input for adding annotation information to the annotation image by the operator.

The adding unit 22 adds annotation information to the annotation image in response to an input for adding annotation information to the annotation image acquired by the first acquiring unit 21 via the first acquiring unit 21 . A completed image and second annotation information, which will be described later, are generated. The second annotation information is annotation information added to the annotation image by the operator, and is text data corresponding to the annotated image. The second annotation information also includes the time taken to designate (assign) the annotation information. The adding unit 22 outputs the generated information indicating the annotated image and the second annotation information to the third processing unit 30 . The second annotation information is an example of annotation results.

The second annotation information will be explained using FIG. 3C. 3C is a diagram illustrating second annotation information according to Embodiment 1. FIG.

As shown in FIG. 3C, the second annotation information includes the position, size, and shape of the object input by the operator on the x and y coordinates, the time required to add the annotation information, etc. to the image for annotation. It is text data. The second annotation information is each text data corresponding to a plurality of objects when a plurality of objects exist in one annotation image.

The third processing unit 30 has a second acquisition unit 32 and an evaluation unit 33 .

The second acquisition unit 32 acquires the annotated image and the second annotation information from the second processing unit 20 and outputs the second annotation information to the evaluation unit 33 . The second acquisition unit 32 also acquires first annotation information to be compared with the second annotation information from the storage unit 11 of the first processing unit 10 and outputs the acquired first annotation information to the evaluation unit 33 .

The evaluation unit 33 evaluates the second annotation information. Specifically, the evaluation unit 33 determines the position, size, shape, etc. of the object in the second annotation information of FIG. 3C and the position, size, shape, etc. of the object in the first annotation information of FIG. 3A. Compare and calculate each difference (similarity). The evaluation unit 33 generates information indicating the annotation skill of the worker based on the calculated difference. The evaluation unit 33 outputs information indicating the generated new annotation skill to the first processing unit 10 . Information indicating annotation skills stored in the storage unit 11 of the first processing unit 10 is automatically updated.

If the difference (similarity) calculated by the evaluation unit 33 is equal to or greater than a prescribed threshold value, it can be said that the image has been annotated with good accuracy. You may

[motion]
Next, the operation of the information processing system 1 according to this embodiment will be described with reference to FIG.

FIG. 4 is a diagram illustrating the operation of the information processing system 1 according to the first embodiment.

In this information processing system 1, it is assumed that the annotation skill of the worker is stored in advance in the storage unit 11 of the first processing unit 10. FIG. Also, the operator performs a task of adding annotation information to the annotation image via the input/output unit 40 .

As shown in FIG. 4, first, the first processing unit 10 acquires the annotation skill of the worker from the storage unit 11 (S11).

Next, the first processing unit 10 outputs an image for annotation having image features specified from the acquired annotation skill (S12). Specifically, the first processing unit 10 searches the storage unit 11 for an annotation image showing an object corresponding to the annotation skill. The first processing unit 10 selects the annotation image when there is an annotation image showing an object corresponding to the annotation skill. Further, when the storage unit 11 does not have an annotation image including an object corresponding to the annotation skill, the first processing unit 10 generates an annotation image including an object corresponding to the annotation skill. The first processing unit 10 outputs the selected or generated annotation image to the input/output unit 40 .

Next, the operator inputs annotation information via the annotation image displayed on the input/output unit 40 . Here, an example of work for adding annotation information to an annotation image will be described with reference to FIG. FIG. 5 is a diagram exemplifying the work of adding annotation information to an annotation image.

For example, in the input/output unit 40,
(1) "hands up" or "hands down"
(2) "legs open" or "legs closed"
Display mutually contradictory and independent options. The worker selects one of these options to perform work.

Also, the operator sets the range of the person appearing in the annotation image via the input/output unit 40 in order to add annotation information to the position of the object (the person in the figure) in the annotation image. Also, the position of the person's head is set as a point, and the skeleton of the person's limbs, etc. is set as a line.

As shown in FIG. 4 , the input/output unit 40 outputs the input received by the operator to the second processing unit 20 . When the first acquisition unit 21 of the second processing unit 20 acquires an input for adding annotation information to the annotation image by the operator, the addition unit 22 of the second processing unit 20 acquires the annotation information for the annotation image. Annotation information is added to the image to generate an annotated image and second annotation information. Then, the second processing unit 20 outputs the annotated image and the second annotation information to the third processing unit 30 (S13).

Next, the second acquisition unit 32 of the third processing unit 30 acquires the second annotation information from the second processing unit 20 and the first annotation information from the storage unit 11 of the first processing unit 10, and acquires the second annotation. The information and the first annotation information are output to the evaluation unit 33 . The evaluation unit 33 compares the position, size, shape, etc. of the object in the second annotation information of FIG. 3C with the position, size, shape, etc. of the object in the first annotation information of FIG. A difference (similarity) is calculated. This difference is performed for each object in the second annotation information. The evaluation unit 33 generates information indicating the annotation skill of the worker based on each calculated difference. The evaluation unit 33 outputs information indicating the annotation skill of the worker to the first processing unit 10, and updates the information indicating the annotation skill stored in the storage unit 11 of the first processing unit 10 (S14).

The evaluation unit 33 determines whether there is a predetermined object (work) that the worker is not good at in the work of adding annotation information to the annotation image (S15). Specifically, the evaluation unit 33 identifies whether or not there is a skill whose annotation skill is less than a predetermined threshold in the information indicating the annotation skill of the worker.

If there is no skill whose annotation skill is less than the predetermined threshold value (NO in S15), the information processing system 1 returns to step S11 because there is no work that the worker is not good at.

On the other hand, if there is a skill whose annotation skill is equal to or higher than the predetermined threshold (YES in S15), there is work that the worker is not good at. is generated, and information indicating the evaluation result is presented to the operator (S16). That is, the evaluation unit 33 outputs information indicating the evaluation result to the input/output unit 40 or an external display device such as a personal computer.

Here, an example of the evaluation result presented to the worker will be given.

The third processing unit 30 presents an annotated image serving as a model, options serving as a model, etc., an annotated image worked by the worker, options selected by the worker, and the like. In addition, the third processing unit 30 presents the difference (difference) between the annotation worked by the worker and the model annotation. This difference is a statistical index that evaluates the entire objects included in the annotation image, an individual index that evaluates each object, an absolute index that evaluates when the goal is achieved, and a relative index for multiple objects. is indicated by The third processing unit 30 presents the work that the worker is good at or the work that the worker is not good at, based on the index described above.

In addition, the third processing unit 30 calculates the work time for each object reflected in the annotation image, comparison of annotation skills with other workers, annotation skill history of workers, ranking among a plurality of workers, It presents the work unit price of the worker, practice problems according to the worker, and the like.

The information processing system 1 then terminates this process and returns the flow to the beginning.

[Effects, etc.]
Thus, in the information processing system 1, annotation images including annotation images corresponding to the annotation skill of the operator are determined. This makes it possible to train annotations according to the individual characteristics of workers. For example, an image that the operator is not good at is determined as an annotation image.

The evaluation unit 33 also updates the annotation skill of the worker stored in the storage unit 11 of the first processing unit 10 . Therefore, it is possible to determine an image for annotation including an image corresponding to the new (updated) annotation skill of the operator. As a result, the operator can continuously improve his or her annotation skills by performing training for adding annotation information to images for annotation according to his or her own annotation skills. Therefore, the annotation skill of the worker can be efficiently improved according to the individual characteristics.

(Embodiment 2)
[composition]
The information processing method of the present embodiment and the configuration of the information processing system 200 are the same as those of the first embodiment unless otherwise specified, and the same reference numerals are assigned to the same configurations, and detailed description of the configuration will be given. omitted.

FIG. 6 is a block diagram illustrating the information processing system 200 and the input/output unit 40 according to the second embodiment. FIG. 7 is a diagram illustrating transmission of information in the information processing system 200 according to the second embodiment.

As shown in FIGS. 6 and 7, the information processing system 200 according to the present embodiment mixes an evaluation image for evaluating the annotation skill of the worker with a plurality of annotation images presented to the worker. do. The information processing system 200 generates a plurality of annotation images in a manner that looks the same as the annotation image so that the worker does not notice that the image is for evaluating his or her own annotation skills. Blend into the image. Accordingly, the information processing system 200 can present a plurality of annotation images and evaluation images to the operator.

The information processing system 200 further includes a first image selection section 210 , a second image selection section 220 and a management section 230 in addition to the first processing section 10 , the second processing section 20 and the third processing section 30 .

The first processing unit 10 has a storage unit 11 that stores information indicating evaluation images and first annotation information. When the first processing unit 10 acquires the information indicating the annotation skill of the worker from the management unit 230, the first processing unit 10 acquires an evaluation image corresponding to the annotation skill. For example, the first processing unit 10 can be configured to look the same as the annotation image presented to the worker by the first image selection unit 210 (so that the worker does not notice that the evaluation image is different from the annotation image). 2), to process the evaluation image to be presented to the operator. Further, when the evaluation image does not exist in the storage unit 11, the first processing unit 10 newly generates an evaluation image to be presented to the operator or processes the annotation image as an evaluation image. Thus, the first processing unit 10 acquires the evaluation image.

The first image selection unit 210 holds a plurality of images for annotation that allow the operator to perform the task of adding annotation information. The first image selection unit 210 has a storage unit 221 that stores annotation images. The first image selection unit 210 acquires the annotation skill of the operator from the management unit 230 and determines an annotation image according to the annotation skill. The first image selection unit 210 acquires an evaluation image from the first processing unit 10, mixes the acquired evaluation image with a plurality of annotation images, and inputs/outputs the evaluation image and the plurality of annotation images. Output to unit 40 .

The second image selection unit 220 acquires information indicating an annotated image generated by the second processing unit 20 and second annotation information via the input/output unit 40 . The second image selection unit 220 selects, from information indicating annotated images and second annotation information, information indicating an annotated image for evaluation that has been worked as an image for evaluation, and the second annotation for evaluation. information to the third processing unit 30 .

The evaluation unit 33 of the third processing unit 30 evaluates the second annotation information and the first annotation information for evaluation. The evaluation unit 33 generates information indicating the annotation skill of the worker and outputs the information to the management unit 230 .

Note that the second image selection unit 220 may transmit information indicating annotated images other than those for evaluation and the second annotation information to the external device as a finished product.

The management unit 230 includes a storage unit 231 that stores information indicating annotation skills, and manages annotation skills of workers. When acquiring the annotation skill of the worker, the management unit 230 updates the annotation skill stored in the storage unit 231 .

Also, the management unit 230 selects a worker according to the annotation skill. For example, a worker with low annotation skill as a whole may be selected, and the worker may be made to actively add annotation information to the annotation image.

[motion]
Next, the operation of the information processing system 200 according to this embodiment will be described.

FIG. 8 is a diagram illustrating the operation of the information processing system 200 according to the second embodiment.

In this information processing system 200, it is assumed that the operator's annotation skill is stored in the management unit 230 in advance.

As shown in FIG. 8, first, the first processing unit 10 acquires the annotation skill of the worker from the management unit 230 (S21).

Next, the first processing unit 10 outputs an evaluation image corresponding to the acquired annotation skill (S22). Specifically, the first processing unit 10 searches the storage unit 11 for an annotation image showing an object corresponding to the annotation skill. The first processing unit 10 selects this evaluation image when there is an evaluation image showing an object corresponding to the annotation skill. In addition, when the storage unit 11 does not have an evaluation image showing an object corresponding to the annotation skill, the first processing unit 10 generates an evaluation image showing an object corresponding to the annotation skill, or generates an evaluation image showing an object corresponding to the annotation skill. Based on the annotation image acquired from 210, the annotation image to be presented to the operator is processed into an evaluation image so that the operator does not recognize it as an evaluation image. The first processing unit 10 outputs information indicating the selected, generated or processed evaluation image to the first image selection unit 210 .

The first image selection unit 210 acquires the annotation skill of the operator from the management unit 230 via the first processing unit 10, and selects an image for annotation according to the annotation skill. The first image selection unit 210 acquires an evaluation image from the first processing unit 10, mixes the acquired evaluation image with a plurality of annotation images, and inputs/outputs the evaluation image and the plurality of annotation images. Output to the unit 40 (S23).

The input/output unit 40 outputs the input received by the operator to the second processing unit 20 . The second processing unit 20 generates an annotated image and second annotation information to which an operator has added annotation information to the annotation image. The second processing unit 20 then outputs the annotated image and the second annotation information to the second image selection unit 220 (S24).

The second image selection unit 220 acquires information indicating annotated images and second annotation information. The second image selection unit 220 selects, from information indicating annotated images and second annotation information, information indicating an annotated image for evaluation that has been worked as an image for evaluation, and the second annotation for evaluation. information to the third processing unit 30 (S25).

Next, when the third processing unit 30 acquires the information indicating the annotated image for evaluation and the second annotation information for evaluation, the evaluation unit 33 of the third processing unit 30 obtains the second annotation information for evaluation. Evaluate the annotation information and the first annotation information. That is, the evaluation unit 33 generates information indicating the annotation skill of the worker based on the difference between the second annotation information for evaluation and the first annotation information for evaluation. The evaluation unit 33 outputs information indicating the annotation skill of the worker to the first processing unit 10, and updates the information indicating the annotation skill stored in the storage unit 11 of the first processing unit 10 (S26). That is, the evaluation unit 33 determines whether there is a predetermined object (work) that the worker is not good at in the work of adding annotation information to the annotation image (S27). Specifically, the evaluation unit 33 identifies whether or not there is a skill whose annotation skill is less than a predetermined threshold in the information indicating the annotation skill of the worker.

If there is no skill whose annotation skill is less than the predetermined threshold (NO in S27), there is no work that the worker is not good at, so the information processing system 200 returns to step S21.

On the other hand, if there is a skill whose annotation skill is equal to or higher than the predetermined threshold (YES in S27), there is work that the worker is not good at. is generated, and information indicating the evaluation result is presented to the operator (S28).

The evaluation unit 33 outputs information indicating the annotation skill to the management unit 230 , and the management unit 230 updates the annotation skill stored in the storage unit 231 when acquiring the information indicating the annotation skill of the worker. The information processing system 200 then terminates this process and returns the flow to the beginning.

[Effects, etc.]
In this manner, in the information processing system 200, the evaluation image is processed or selected so as not to be noticed that the evaluation image is different from the annotation image. The information processing system 200 mixes this evaluation image with a plurality of annotation images and presents them to the operator. For this reason, the worker is less likely to notice that the evaluation image is included, so that the worker's annotation skill can be evaluated.

Other effects in this embodiment are the same as those in the first embodiment.

(Other modifications)
Although the information processing method and the information processing system according to the first and second embodiments of the present disclosure have been described above, the embodiments of the present disclosure are not limited to the first and second embodiments described above.

For example, in the above embodiments, an example in which sensing data to be annotated is image data has been described, but the sensing data is not limited to this. Sensing data to be annotated includes distance data obtained from a distance sensor, sound data obtained from a microphone, point cloud data obtained from radar such as LIDAR (Light Detection and Ranging), and scent data obtained from a scent sensor. It may be sensing data.

Further, for example, the information processing method according to the above embodiments can be implemented as a program for causing a computer to execute, or as a storage medium for storing the program.

Each processing unit included in the information processing system according to the above embodiment is typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them.

Further, circuit integration is not limited to LSIs, and may be realized by dedicated circuits or general-purpose processors. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connections and settings of circuit cells inside the LSI may be used.

In each of the above-described embodiments, each component may be configured by dedicated hardware, or realized by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.

In addition, the numbers used above are all examples for specifically describing the present disclosure, and the embodiments of the present disclosure are not limited to the illustrated numbers.

Also, the division of functional blocks in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be moved to other functional blocks. may Moreover, single hardware or software may process the functions of a plurality of functional blocks having similar functions in parallel or in a time-sharing manner.

Also, the order in which each step in the flowchart is executed is for illustrative purposes in order to specifically describe the present disclosure, and orders other than the above may be used. Also, some of the above steps may be executed concurrently (in parallel) with other steps.

The information processing method and the information processing system according to one or more aspects have been described above based on Embodiments 1 and 2, but Embodiments 1 and 2 of the present disclosure are limited to these aspects. not something. As long as it does not deviate from the spirit of the present disclosure, the embodiments 1 and 2 with various modifications that a person skilled in the art can think of, and the embodiment constructed by combining the components of different embodiments are also one or more aspects may be included within the range of

INDUSTRIAL APPLICABILITY The present disclosure can be applied to an evaluation system for understanding skills for adding annotation information to sensing data, a training system for improving skills for adding annotation information to sensing data, and the like.

1, 200 information processing system 10 first processing unit (processing unit)
21 first acquisition unit 32 second acquisition unit

Claims (10)

the computer
Acquire information related to annotation skill , which is an index that evaluates the accuracy of the work of annotating the object in the image by the worker ,
obtaining an annotation image showing the object specified from the information related to the annotation skill;
obtaining an annotation result for the annotation image by the worker;
An information processing method, wherein information related to the annotation skill is updated using correct annotation information corresponding to the object appearing in the annotation image and the annotation result. 2. The information processing method according to claim 1 , wherein the addition of the annotation information includes at least one of annotation position specification, size specification, and shape specification. The information processing method according to claim 1 , wherein the addition of the annotation information includes specification of a label for the annotation. The annotation result includes the time taken to add the annotation information,
4. The information processing method according to any one of claims 1 to 3 , wherein the information related to the annotation skill includes the time required for adding the annotation information. A plurality of the annotation images and the correct annotation information are stored in a storage unit of the computer,
2. The annotation image according to claim 1, wherein the annotation image is obtained by selecting, as the annotation image, one annotation image in which the object corresponding to the correct annotation information is shown, from among the plurality of annotation images. Information processing methods. The information processing method according to claim 1, wherein the image for annotation is acquired by generating an image for annotation in which the object corresponding to the correct annotation information is shown. A plurality of the annotation images and the correct annotation information are stored in a storage unit of the computer,
7. The method according to claim 6 , wherein the generation of the annotation image includes synthesizing one of the plurality of annotation images, in which the object corresponding to the correct annotation information is shown, with another annotation image. information processing method. The information processing method according to any one of claims 1 to 7, further comprising presenting a difference between the correct annotation information and the annotation result to the worker. The information processing method according to any one of claims 1 to 8 , further comprising presenting information related to the updated annotation skill to the worker. a first acquisition unit that acquires an image for annotation in which the object is identified from information related to annotation skill , which is an index for evaluating the accuracy of the work of adding annotation information to an object in the image by a worker ; ,
a second acquisition unit that acquires an annotation result for the annotation image by the operator;
An information processing system, comprising: a processing unit that updates information related to the annotation skill using correct annotation information corresponding to an object included in the annotation image and the annotation result.

Images