JP7334920B1 - Information processing system, information processing method and program - Google Patents

Abstract

[Problem] To reduce the labor required for annotation. [Solution] A processor of a computer system displays an operation screen that allows a user to associate an annotation class that is an annotation class registered by the user with a model class that is a class classified by a model. The first output unit outputs information on the annotation class and the model class in accordance with the user's operation to associate the annotation class and the model class on the operation screen generated by the terminal using the output information. A recording unit that records data in association with a storage device, an execution unit that executes a model on target input data according to user operations, and a model class detected by executing the model by referring to the storage device. an assignment unit that replaces and assigns an annotation class to be assigned, and a second output unit that outputs information regarding the assigned annotation class. [Selection diagram] Figure 2

Description

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of publication of the website March 3, 2022 2 Website address https://fastlabel. ai/news/20220303-auto-annotation

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of publication of the website March 3, 2022 2 Website address https://docs. fast label. ai/docs/auto-annotation https://docs. fast label. ai/docs/start-auto-annotation

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of publication of the website March 3, 2022 2 Website address https://prtimes. jp/main/html/rd/p/000000032.000065427. html

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of posting on the website March 3, 2022 2 Website address https://japan. zdnet. com/release/30654827/

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of publication of the website March 3, 2022 2 Website address https://wmr. tokyo/startup/2022/123152/

Application of Article 30, Paragraph 2 of the Patent Act 1 Date of publication of the website March 3, 2022 2 Website address https://ledge. ai/2022-03-03-1313238237622084de96d18/

Application of Article 30, Paragraph 2 of the Patent Law 1 Date of publication of the website March 4, 2022 2 Website address https://airobot-news. net/2022/03/04/fastlabel-7/

The present invention relates to an information processing system, an information processing method, and a program.

In recent years, social implementation of machine learning is progressing in many companies. In such a situation, there is a problem that, among the creation of teacher data, annotation in particular takes time and effort. Here, annotation is a process of attaching information called labels, tags or metadata to each piece of data in any form such as text, voice, image or moving image.

Techniques have been developed to support annotation for this problem. For example, in Patent Document 1, using an existing model prepared in advance, a prior label estimation unit for estimating a prior label for each of a plurality of elements and assigning it to each of the plurality of elements, each of the plurality of elements, and a plurality of generating a labeling work screen for updating the labels given to the elements of the labeling work screen showing each of the plurality of elements and the label given to each of the plurality of elements in association with each other; When the label given to the element is updated by the label assignment work screen output unit that outputs to the external input/output interface and the update operation via the label assignment work screen, the label that assigns the updated label to the element A labeling assistance device is disclosed that includes an updating unit.

International Publication No. 2022/185363

However, the granularity or name of the model class, which is the classification output by the model for input data, and the annotation class, which is the classification desired by the user, may differ. As a specific example of different granularity, there is a case where the model classes output by the model for the input data are "bus" and "truck", and the annotation class that the user wants to add is "car". be. In this way, there is a problem that the granularity or name differs between the model class and the annotation class, and the annotation is labor intensive.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing system, an information processing method, and a program that can reduce the labor required for annotation.

An information processing system according to a first aspect of the present invention provides an operation screen that allows a user to associate an annotation class, which is a class for annotation registered by a user, with a model class, which is a class classified by a model. and an operation screen generated by the terminal using the output information, in response to a user operation for associating the annotation class and the model class, the annotation class and a recording unit that records the model class in association with a storage device; an execution unit that executes the model for target input data in accordance with user's operation; and detection by executing the model with reference to the storage device. an assignment unit that replaces and assigns the assigned model class to the corresponding annotation class; and a second output unit that outputs information about the assigned annotation class.

According to this configuration, when the model is executed for the target input data, the annotation class desired by the user is output instead of the original model class of the model, so that the labor required for annotation can be reduced.

An information processing system according to a second aspect of the present invention is the information processing system according to the first aspect, wherein the model is an object detection model for detecting an object in an image, and the execution unit includes a user executes an object detection model on the target image data in response to the operation of , and the assigning unit refers to the storage device and executes the object detection for each object detected by executing the object detection model. The model class detected by executing the model is replaced and assigned to the corresponding annotation class, and the second output unit enables the user to visually identify the annotation class assigned to each object detected in the target image. Output information for display.

According to this configuration, the annotation class assigned to each object detected by the object detection model in the target image is displayed, so that the labor for assigning the annotation class can be reduced.

An information processing system according to a third aspect of the present invention is the information processing system according to the second aspect, wherein the annotation class can be selected not to overwrite the annotation class already assigned to the object included in the target image. The execution unit further includes a third output unit that outputs information for displaying a setting screen, and when the user selects not to overwrite the annotation class already assigned to the target image on the setting screen, executing a first object detection model in response to the operation of the allocating unit referring to the storage device to assign a first model class to each object detected by executing the first object detection model to a corresponding first annotation class, the execution unit executes a second object detection model in response to a user operation, and the allocation unit assigns the first annotation class, in addition to With reference to the storage device, the first model class is replaced with the corresponding first annotation class and assigned to each object detected by executing the first object detection model.

According to this configuration, both the annotation result using the first object detection model and the annotation result using the second object detection model are output. can be displayed. Therefore, the user can simultaneously grasp annotation results using a plurality of object detection models.

An information processing system according to a fourth aspect of the present invention is the information processing system according to any one of the first to third aspects, and displays an input screen on which status and/or tags of target input data can be input. The execution unit executes the model on target input data to which statuses and/or tags input by a user are added.

According to this configuration, the user can narrow down the data to be annotated by status and/or.

An information processing system according to a fifth aspect of the present invention is the information processing system according to any one of the first to fourth aspects, wherein the recording unit causes a user to associate an annotation class and a model class on the operation screen. In addition to the operation, the annotation class and the model class for the target model are recorded in a storage device in association with each other in response to a user operation for selecting a target model from a plurality of models, and the execution unit is configured to: A model selected by the user is executed for target input data in accordance with a user's operation, and the allocation unit refers to the storage device and detects a model class detected by executing the model selected by the user. is replaced with the corresponding annotation class and assigned.

According to this configuration, when a plurality of models can be selected, the annotation class and the model class of the target model are recorded in association with the storage device. A class can be assigned by replacing it with an annotation class.

An information processing method according to a sixth aspect of the present invention provides an operation screen that allows a user to associate an annotation class, which is a class for annotation registered by a user, with a model class, which is a class classified by a model. In a first output procedure for outputting information for displaying and an operation screen generated by a terminal using the output information, the annotation class and A recording procedure for recording the model class in association with a storage device, an execution procedure for executing the model for target input data in response to user operation, and detection by executing the model with reference to the storage device and a second output procedure for outputting information about the assigned annotation class.

According to this configuration, when the model is executed for the target input data, the annotation class desired by the user is output instead of the original model class of the model, so that the labor required for annotation can be reduced.

A program according to a seventh aspect of the present invention provides a computer with an operation that enables a user to associate an annotation class, which is a class for annotation registered by a user, with a model class, which is a class classified by a model. A first output procedure for outputting information for displaying a screen, in an operation screen generated by a terminal using the output information, in response to a user operation for associating an annotation class and a model class, the annotation class and A recording procedure for recording the model class in association with a storage device, an execution procedure for executing the model for target input data in response to a user's operation, and a model detected by executing the model with reference to the storage device A program for executing an assignment procedure for replacing and assigning a model class to a corresponding annotation class, and a second output procedure for outputting information about the assigned annotation class.

According to this configuration, when the model is executed for the target input data, the annotation class desired by the user is output instead of the original model class of the model, so that the labor required for annotation can be reduced.

According to one aspect of the present invention, when a model is executed for target input data, an annotation class desired by the user is output instead of the original model class of the model, so that the labor required for annotation can be reduced. .

1 is a schematic configuration diagram of an information processing system according to an embodiment; FIG. 1 is a schematic configuration diagram of a computer system according to this embodiment; FIG. It is an example of an annotation class creation screen. It is an annotation class list screen. It is an example of an operation screen for linking an annotation class and a model class. It is an example of a setting screen. It is an example of a pre-execution setting screen for automatic annotation. It is an example of a detection result screen.

Hereinafter, each embodiment will be described with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art.

In order to solve the above problems, the inventor of the present application provides an annotation class, which is a class for annotations registered by a user, and a model class, which is a class classified by a model, on an operation screen used by the user. is associated, and when the model is executed, the model class output by the model is replaced with the annotation class associated with the model class and output.

FIG. 1 is a schematic configuration diagram of an information processing system according to this embodiment. As shown in FIG. 1, an information processing system S includes terminals 1-1, . . . , 1-N (N is a natural number) used by users, the terminals 1-1, . a computer system 2 connected via.

Terminals 1-1, . . . , 1-N are terminals used by the user who uses the model. . Terminals 1-1, . . . , 1-N are collectively referred to as terminal 1.

The computer system 2 is used or managed, for example, by an administrator of an organization (including a company) that supports annotation of models. The computer system 2 executes processing in response to a request from the terminal 1 and provides the terminal 1 with information obtained after the processing. The computer system 2 may be a single computer, or may be composed of a plurality of computers. Here, as an example, the computer system 2 is described as one computer.

FIG. 2 is a schematic configuration diagram of a computer system according to this embodiment. As shown in FIG. 2, the computer system 2 comprises an input interface 21, a communication module 22, a storage device 23, a memory 24, an output interface 25 and a processor 26.
The input interface 21 receives input from the administrator of the computer system 2 and outputs an input signal to the processor 26 according to the received input.
The communication module 22 is connected to the communication network CN and communicates with the terminal 1 . Although this communication may be wired or wireless, it will be explained assuming that it is wired.

The storage device 23 is, for example, a storage, and stores programs and various data for the processor 26 to read and execute.
The memory 24 temporarily holds data and programs. The memory 24 is a volatile memory, such as a RAM (Random Access Memory).
The output interface 25 can be connected to an external device and can output a signal to the external device.

The processor 26 loads a program from the storage device 23 to the memory 24 and executes a series of instructions included in the program to perform a first output unit 261, a recording unit 262, an execution unit 263, an allocation unit 264, a second It functions as an output unit 265 and a third output unit 266 .

Examples of screens displayed on the terminal 1 will be described below with reference to FIGS. 3 to 8. FIG. FIG. 3 is an example of an annotation class creation screen. As shown in FIG. 3, on the annotation class creation screen G1, a select box B11 for selecting the type indicating the shape of marking the object to be annotated in the image, a text box B12 for entering the name of the annotation class, A text box B13 for inputting the annotation class value, a text box B14 for inputting an identification color for identifying the annotation, and a creation button B15 are provided. Also, when the create button B15 is pressed, an annotation class with the name "bicycle" and the value "bicycle" is created, for example.

FIG. 4 is an annotation class list screen. The annotation class list screen G2 in FIG. 4 displays a list of annotation classes registered by the user. As a list of this annotation class, there is shown a table consisting of sets of identification color, name, value, type, attribute, and last update date and time set at the time of creation. A "create annotation class" button B21 is also provided, and when this button B21 is pressed, the screen changes to the creation screen of FIG.

FIG. 5 is an example of an operation screen for linking annotation classes and model classes. The operation screen G3 in FIG. 5 displays a select box B31 for selecting a model, a radio button B32 for selecting an annotation class, a check box B33 for selecting a model class, and a save button B34.

In order to display this operation screen G3, the first output unit 261 allows the user to associate an annotation class, which is a class for annotation registered by the user, with a model class, which is a class classified by a model. information for displaying an operation screen (for example, see FIG. 5).

In FIG. 5, the user selects a model, selects one annotation class, selects one or more model classes linked to the selected annotation model, and presses the save button B34.

When the save button B34 is pressed, the recording unit 262 saves the annotation class on the operation screen G3 generated by the terminal 1 using the output information according to the user operation for associating the annotation class and the model class. and the model class concerned are recorded in the storage device 23 in association with each other. As a result, the annotation class and the model class are linked at 1:1 or 1:multiple.

More specifically, for example, the recording unit 262 stores information about the target model in response to a user operation of selecting a target model from a plurality of models in addition to a user operation of associating an annotation class and a model class on the operation screen G3. , the annotation class and the model class are recorded in association with the storage device. As a result, in the selected model, the annotation class and the model class are linked at 1:1 or 1:multiple. In this case, the execution unit executes the model selected by the user for the target input data according to the user's operation. Then, the assigning unit refers to the storage device, replaces the model class detected by the execution of the model selected by the user with the corresponding annotation class, and assigns it.

According to this configuration, when a plurality of models can be selected, the annotation class and the model class of the target model are recorded in association with the storage device. A class can be assigned by replacing it with an annotation class.

FIG. 6 is an example of a setting screen. A setting screen G4 in FIG. 6 displays a list of annotation classes and model classes that are linked by a user operation on the operation screen G3 in FIG. The setting screen G4 of FIG. 6 is provided with an "execute automatic annotation" button B41. When the "execute automatic annotation" button B41 is pressed, a pre-execution setting screen G5 shown in FIG. 7 is displayed as a pop-up, for example.

FIG. 7 is an example of a pre-execution setting screen for automatic annotation. In the pre-execution setting screen G5 of FIG. 7, for example, a select box B51 for selecting a model and a select box B52 for selecting a reliability score are provided. Only results exceeding the reliability score set here (for example, a value between 0 and 1 representing the probability of classification output by the model) are used as annotation data.

On the pre-execution setting screen G5, a link B56 of "execute automatic annotation" is provided. The execution unit 263 executes a model for target input data (for example, target image data) according to a user's operation (for example, selection of link B56 "execute automatic annotation" in FIG. 7).

The allocation unit 264 refers to the storage device 23 and replaces the model class detected by executing the model with the corresponding annotation class and allocates it.
The second output unit 265 outputs information about the assigned annotation class.

In this embodiment, the model is, for example, an object detection model for detecting objects in an image. The executing unit 263 executes the object detection model on the target image according to the user's operation. The assigning unit 264 refers to the storage device 23 and replaces the model class detected by executing the object detection model with the corresponding annotation class and assigns it to each object detected by executing the object detection model. The second output unit 265 outputs information for displaying the annotation class assigned to each object detected in the target image so that the user can visually identify the annotation class.

As a result, the annotation class assigned to each object detected by the object detection model in the target image is displayed, so that the labor for assigning the annotation class can be reduced.

As a result, for example, a detection screen shown in FIG. 8 is output. FIG. 8 is an example of a detection result screen. As shown in the detection result screen G6 of FIG. 8, in the target image, the extracted pedestrian regions R61 and R62 and the bicycle region R63 are indicated by rectangles in different colors. A legend (or label) is given. When the save button B64 in FIG. 8 is pressed, the information about this annotation is saved in the storage device 23. FIG.

Here, the user can attach a status and/or a tag to the data to be annotated on another operation screen (not shown).
Returning to FIG. 7, the pre-execution setting screen G5 of FIG. 7 is provided with a select box B53 for selecting a status. This allows the user to narrow down the data to be annotated by status. A select box B54 for selecting a tag is provided on the pre-execution setting screen G5 of FIG. This allows the user to narrow down the data to be annotated using tags.

To implement this function, the third output unit 266 outputs information for displaying an input screen on which the status and/or tag of the target input data can be input. Then, the execution unit 263 executes the model with respect to target input data to which statuses and/or tags input by the user are added.

A toggle button B55 for selecting whether or not to "overwrite existing annotations" is provided on the pre-execution setting screen G5 of FIG.
If "Overwrite existing annotations" is selected, the annotation class assigned to the objects previously detected by the first object detection model will be deleted, and the annotation class assigned to the objects detected by the second object detection model will be Only assigned annotation classes are printed.
On the other hand, if "overwrite existing annotations" is selected, the annotation class assigned to the object previously detected by the first object detection model remains without being deleted, and is Output with the annotation class assigned to the detected object.

In order to realize this function, the third output unit 266 provides a setting screen (for example, the pre-execution setting screen G5 in FIG. ) to display information.
When the user selects not to overwrite the annotation class already assigned to the target image on this setting screen (for example, the pre-execution setting screen G5 in FIG. 7), the execution unit 263 performs the first , and the assigning unit 264 refers to the storage device and assigns the first model class to the corresponding first annotation class for each object detected by the execution of the first object detection model. Assign by replacing. Furthermore, the execution unit 263 executes the second object detection model according to the user's operation, and the allocation unit 264 refers to the storage device 23 to perform the first annotation class in addition to the allocation of the first annotation class. For each object detected by execution of the object detection model, the first model class is assigned in place of the corresponding first annotation class.

As a result, both the annotation result using the first object detection model and the annotation result using the second object detection model are output, so the annotation results using a plurality of object detection models can be displayed at the same time. can be done. Therefore, the user can simultaneously grasp annotation results using a plurality of object detection models.

As described above, the information processing system S according to the present embodiment provides an operation screen that allows the user to associate annotation classes, which are classes for annotations registered by the user, with model classes, which are classes classified by models. A first output unit for outputting information for display is provided.
The information processing system S associates the annotation class and the model class with the storage device according to a user operation for associating the annotation class and the model class on the operation screen generated by the terminal using the output information. and a recording unit for recording.
The information processing system S includes an execution unit 263 that executes a model on target input data in accordance with a user's operation.
The information processing system S includes an allocation unit 264 that refers to the storage device 23 and replaces the model class detected by executing the model with the corresponding annotation class and allocates it.
The information processing system S includes a second output unit 265 that outputs information about annotation classes assigned to target input data.

According to this configuration, when the model is executed for the target input data, the annotation class desired by the user is output instead of the original model class of the model, so that the labor required for annotation can be reduced.

At least part of the computer system 2 described in the above embodiment may be configured with hardware or software. When configured with software, a program that implements at least part of the functions of the computer system 2 may be stored in a computer-readable recording medium and read and executed by a computer. The recording medium is not limited to a detachable one such as a magnetic disk or an optical disk, and may be a fixed recording medium such as a hard disk device or memory.

Also, a program that implements at least part of the functions of the computer system 2 may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired line or wireless line such as the Internet, or stored in a recording medium and distributed.

Furthermore, the computer system 2 may be operated by one or more information devices. When a plurality of information devices are used, one of them may be a computer, and the computer may implement a function as at least one means of the computer system 2 by executing a predetermined program. A part or all of the functions of the computer system 2 may be realized by the terminal 1 .

Further, in the method invention, all processes (steps) may be automatically controlled by a computer. Also, while a computer is causing each step to be performed, progress control between steps may be manually performed. Furthermore, at least a part of all steps may be performed manually.

As described above, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the present invention at the implementation stage. Further, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, components across different embodiments may be combined as appropriate.

1 terminal 2 computer system 21 input interface 22 communication module 23 storage device 24 memory 25 output interface 26 processor 261 first output unit 262 recording unit 263 execution unit 264 allocation unit 265 second output unit 266 third output unit S information processing system

Claims (7)

a first output unit that outputs information for displaying an operation screen that allows the user to associate an annotation class, which is a class for annotations registered by a user, with a model class, which is a class classified by a model; ,
a recording unit that associates and records the annotation class and the model class in a storage device according to a user operation for associating the annotation class and the model class on the operation screen generated by the terminal using the output information; ,
an execution unit that executes a model on target input data according to a user's operation;
an allocation unit that refers to the storage device and allocates the model class detected by executing the model by replacing it with the corresponding annotation class;
a second output unit that outputs information about the assigned annotation class;
An information processing system comprising the model is an object detection model for detecting objects in an image;
The execution unit executes an object detection model on target image data according to a user's operation,
The allocation unit replaces the model class detected by executing the object detection model with the corresponding annotation class and allocates it to each object detected by executing the object detection model by referring to the storage device;
The information processing system according to claim 1, wherein the second output unit outputs information for displaying annotation classes assigned to respective objects detected in the target image so that the user can visually identify them. further comprising a third output unit that outputs information for displaying a setting screen that can be selected so as not to overwrite the annotation class already assigned to the object included in the target image;
If the user selects not to overwrite the annotation class already assigned to the target image on the setting screen,
The execution unit executes a first object detection model according to a user's operation,
The assigning unit replaces the first model class with the corresponding first annotation class and assigns it to each object detected by executing the first object detection model by referring to the storage device;
The execution unit executes a second object detection model according to a user's operation,
The assigning unit, in addition to assigning the first annotation class, refers to the storage device and assigns the first model class to each object detected by executing the first object detection model. The information processing system according to claim 2, wherein the annotation class is replaced and assigned. Further comprising a third output unit that outputs information for displaying an input screen on which the status and/or tags of the target input data can be input,
The information processing system according to any one of claims 1 to 3, wherein the execution unit executes the model with respect to target input data to which statuses and/or tags input by a user are added. In addition to a user operation of associating an annotation class and a model class on the operation screen, the recording unit stores the target model with the annotation class in response to a user operation of selecting a target model from a plurality of models. record the model class in association with a storage device;
The execution unit executes a model selected by the user for the target input data in accordance with a user's operation,
The information according to any one of claims 1 to 3, wherein the assigning unit refers to the storage device and assigns a model class detected by executing the model selected by the user by replacing it with a corresponding annotation class. processing system. A first output unit outputs information for displaying an operation screen that allows a user to associate an annotation class, which is a class for annotation registered by the user, with a model class, which is a class classified by a model. a first output procedure to
A recording unit records the annotation class and the model class in association with the storage device according to a user operation for associating the annotation class and the model class on the operation screen generated by the terminal using the output information. a recording procedure to
an execution procedure in which the execution unit executes a model on target input data in accordance with a user's operation;
an allocation procedure in which the allocation unit refers to the storage device and allocates the model class detected by executing the model by replacing it with the corresponding annotation class;
a second output step in which a second output unit outputs information about the assigned annotation class;
An information processing method comprising: to the computer,
A first output procedure for outputting information for displaying an operation screen that enables the user to associate an annotation class, which is a class for annotation registered by a user, with a model class, which is a class classified by a model;
A recording procedure for recording the annotation class and the model class in association with a storage device in response to a user operation for associating the annotation class and the model class on the operation screen generated by the terminal using the output information;
Execution procedure for executing the model on the target input data according to the user's operation,
an assignment procedure that refers to the storage device and replaces and assigns the model class detected by executing the model to the corresponding annotation class;
a second output procedure for outputting information about the assigned annotation class;
program to run the