JP2023166812A - Information processing method, computer program and information processing device - Google Patents

Abstract

To provide an information processing method that can be expected to support annotation work for images, a computer program, and an information processing device.SOLUTION: In an information processing method according to the present embodiment, an information processing device acquires a plurality of captured images of the same object, each captured using a different wavelength of light, and based on at least one of the plurality of acquired captured images, attaches related information to at least one of the plurality of captured images. Also, the information processing device may receive designation of an image region for at least one captured image of the plurality of acquired captured images, extract a similar region to the accepted image region from the captured image, and attach information on the image region and the similar region as the related information.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing method, computer program, and information processing apparatus that support annotation work on images.

2. Description of the Related Art In recent years, machine learning and deep learning technologies have made remarkable progress, and high-performance devices and products that use learning models generated by these machine learning methods are widely used. Among methods such as machine learning, a method called supervised learning requires learning data (so-called teacher data) in which input information and output information for a learning model are associated. For example, an image that is input to a learning model is associated with information such as the classification results of things captured in this image or image areas where objects are captured in this image, and the training model is used as learning data. Used for learning. This process of generating learning data, that is, associating output information with input information, is called annotation, and is often a manual process.

In Patent Document 1, multiple annotations based on inputs from multiple workers to the same assignment target are acquired, the acquired multiple annotations are compared, and based on the comparison, annotations are shared between multiple workers. An information processing device has been proposed that determines whether there is a deviation in the application standards and issues a warning if there is a deviation in the application standards between a plurality of workers.

Japanese Patent Application Publication No. 2021-196905

Generally, machine learning requires a large amount of learning data, and the amount of manual annotation work is enormous. In addition, a worker who performs annotation work requires a certain level of knowledge about the data being handled, and not just anyone can perform annotation work easily. Additionally, if the data being handled is highly confidential, the number of people who can perform annotation work may be limited.

The present invention has been made in view of the above circumstances, and its purpose is to provide an information processing method, a computer program, and an information processing device that can be expected to support annotation work on images. be.

In an information processing method according to an embodiment, an information processing device obtains a plurality of captured images of the same object based on different wavelengths of light, and at least one of the obtained plurality of captured images. Based on the photographed images, relevant information is added to at least one photographed image among the plurality of photographed images.

According to one embodiment, it can be expected to support annotation work on images.

FIG. 1 is a schematic diagram for explaining the configuration of an information processing system according to the present embodiment. FIG. 2 is a schematic diagram for explaining the configuration of a rotating filter. FIG. 1 is a block diagram showing the configuration of an information processing device 1 according to the present embodiment. FIG. 2 is a schematic diagram showing an example of a composite image. FIG. 2 is a schematic diagram showing an example of an infrared region image. FIG. 2 is a schematic diagram for explaining an example of annotation work. FIG. 2 is a schematic diagram for explaining an example of annotation work. FIG. 2 is a schematic diagram for explaining an example of learning data. 3 is a flowchart showing the procedure of annotation processing performed by the information processing apparatus according to the present embodiment. 3 is a flowchart showing the procedure of annotation processing performed by the information processing apparatus according to the present embodiment.

A specific example of an information processing system according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all changes within the meaning and scope equivalent to the scope of the claims.

<System configuration>
FIG. 1 is a schematic diagram for explaining the configuration of an information processing system according to this embodiment. The information processing system according to this embodiment includes an information processing device 1, a camera 3, a rotating filter 7, and the like. In the information processing system according to the present embodiment, the information processing device 1 acquires a captured image of an object captured by the camera 3 via the rotating filter 7, and the user uses the information processing device 1 to It is a system for annotation.

Annotation is the process or work of adding related information to some data, and particularly in the field of artificial intelligence, it is the process or work of creating teacher data for supervised learning. In annotation work, for example, one task is to add data (information such as correct values or correct answer labels) that the learning model should output to data such as numerical values, images, or character strings that are input to the learning model. Or by multiple users. The information processing system according to this embodiment is a system that supports annotation work that uses images as input data.

Further, the information processing system according to the present embodiment adds information indicating a specific area included in the image, such as the area of the object captured in the image or the area of an abnormality included in the object. Specifically, it supports annotation work that adds information such as labels to each pixel of an image. Learning data in which such information is attached to an image is used for machine learning of a learning model that performs so-called image segmentation. Such an annotation work can be performed, for example, by a user operating an input device such as a mouse or a pen tablet on an image displayed on a display to select a specific area in the image.

However, in the information processing system according to the present embodiment, the data added to the input data by annotation is, for example, a binary classification result indicating the presence or absence of an abnormality of the object shown in the image, or the data added to the input data by annotation. The data may be data in which one value is assigned to one image, such as a multi-value classification result for classifying an object into one of a plurality of classes.

In the information processing system according to the present embodiment, a plurality of photographed images of the same object are acquired based on different wavelengths of light by photographing using the camera 3 and the rotating filter 7, and the plurality of acquired images are Annotation work is done based on this. FIG. 2 is a schematic diagram for explaining the configuration of the rotating filter 7. As shown in FIG. The rotary filter 7 has a disc-shaped main body in which a plurality of types of filters 7a to 7f are arranged side by side in the circumferential direction, and a rotating shaft of a motor is fixed at the center of the main body. The rotating filters 7 are installed such that any one of the filters 7a to 7f is located in front of the lens of the camera 3. The rotation of the motor of the rotary filter 7 is controlled by the camera 3, and the camera 3 controls the rotation of the motor to rotate the main body of the rotary filter 7, thereby rotating the filters 7a to 7a located in front of the lens of the camera 3. 7f can be switched. That is, the camera 3 can select one of the plurality of types of filters 7a to 7f included in the rotating filter 7 to photograph the object to be inspected.

Furthermore, the camera 3 according to the present embodiment is a camera that can receive not only visible light but also infrared light to take pictures. The plurality of types of filters 7a to 7f included in the rotary filter 7 include, for example, a red filter that passes red light, a blue filter that passes blue light, a green filter that passes green light, and a first wavelength (for example, 750 nm) filter. A first infrared filter that passes infrared rays of a second wavelength (for example, 800 nm), a second infrared filter that passes infrared rays of a second wavelength (for example, 800 nm), and the like may be included. In addition, the filters 7a to 7f of the rotating filter 7 are filters that enable the camera 3 to perform normal photography virtually without a filter (allowing the camera 3 to perform photography equivalent to a normal camera that does not receive infrared rays). (e.g., a filter that blocks infrared rays included in ordinary cameras).

The camera 3 according to the present embodiment rotates the rotary filter 7 to sequentially photograph the object to be inspected through a plurality of types of filters 7a to 7f, and outputs the plurality of photographed images to the information processing device 1. . For example, if the rotary filter 7 is provided with six types of filters 7a to 7f, the camera 3 can acquire six types of photographed images by sequentially photographing the object to be inspected through each of the filters 7a to 7f. Can be done. However, the camera 3 does not need to take pictures using all the filters 7a to 7f included in the rotating filter 7. For example, even if the rotating filter 7 is provided with six types of filters 7a to 7f, Photographing may be performed sequentially using one or fewer filters 7a to 7f to obtain five or fewer types of photographed images.

Note that in this embodiment, a configuration is adopted in which a plurality of types of filters are changed using the rotating filter 7, but the configuration for changing filters is not limited to the configuration of the rotating filter 7. For example, the configuration may be such that one or more filters are switched by sliding with respect to the lens of the camera 3, or the filters may be switched by other methods. Furthermore, the filter switching is not limited to the configuration in which it is performed under the control of the camera 3, and may be performed automatically under the control of the information processing device 1, for example, or may be performed manually by the user. In the case of a configuration in which the user manually switches filters, for example, a configuration in which the user individually attaches and detaches the filters to and from the lens of the camera 3 may be used.

The information processing device 1 acquires a plurality of images taken by the camera 3 through the plurality of filters 7a to 7f of the rotating filter 7, and stores these plurality of images as one set of images. In this embodiment, it is assumed that images are transferred from the camera 3 to the information processing device 1 via, for example, wired or wireless communication, but the present invention is not limited to this. For example, images may be sent and received from the camera 3 to the information processing device 1 via a recording medium such as a memory card or an optical disk, and the camera 3 and the information processing device 1 may send and receive images via another device such as a server device. Alternatively, the camera 3 and the information processing device 1 may be an integrated device.

The information processing device 1 displays one or more images of a set of images to a user performing an annotation work, and prompts the user to specify a specific area included in the image. accept. The information processing device 1 generates data such as labels to be attached to pixels included in the area specified by the user's operation (and pixels included in the unspecified area), and stores the generated data together with the image. do.

<Device configuration>
FIG. 3 is a block diagram showing the configuration of the information processing device 1 according to the present embodiment. The information processing device 1 according to the present embodiment includes a processing section 11, a storage section 12, a communication section (transceiver) 13, a display section 14, an operation section 15, and the like. The information processing device 1 according to the present embodiment may be configured using a general-purpose information processing device such as a personal computer or a tablet terminal device. Note that although the present embodiment will be described assuming that the processing is performed by one information processing device 1, the processing may be performed in a distributed manner by a plurality of information processing devices.

The processing unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. It is configured using The processing unit 11 reads out and executes a program 12a stored in the storage unit 12, thereby performing a process of acquiring a plurality of photographed images taken by the camera 3 via the rotating filter 7, and processing of acquiring these plurality of photographed images. Performs various processing such as processing to support the work of the annotations used.

The storage unit 12 is configured using a large-capacity storage device such as a hard disk, for example. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for processing by the processing unit 11. In this embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. The storage unit 12 is also provided with a learning data storage unit 12b that stores learning data used for machine learning that is generated by the user performing annotation work.

In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit. 12. However, the program 12a may be written into the storage unit 12, for example, during the manufacturing stage of the information processing device 1. Further, for example, the program 12a may be distributed by a remote server device or the like, and the information processing device 1 may obtain it through communication. For example, the program 12a may be recorded on the recording medium 99 and read by a writing device and written into the storage unit 12 of the information processing device 1. The program 12a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99.

The learning data storage unit 12b associates one or more photographed images taken by the camera 3 with information added by the user's annotation work and stores them as learning data. Note that when the camera 3 photographs the same object using, for example, six different wavelengths of light, and the information processing device 1 acquires six photographed images, the learning data after the user's annotation work is , it is sufficient that at least one of the six captured images is included, and it is not necessary to include all of the six captured images. That is, the learning data need only include at least images necessary for machine learning of the learning model (images input to the learning model), and do not need to include images unnecessary for machine learning.

The communication unit 13 is connected to the camera 3 via a communication cable, for example, and exchanges data such as captured images and control information with the camera 3. The communication unit 13 may exchange data based on a communication standard such as USB (Universal Serial Bus) or LAN (Local Area Network). Further, the communication unit 13 may exchange data with the camera 3 by wireless communication without using a communication cable.

The display unit 14 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 11. The operation unit 15 accepts user operations and notifies the processing unit 11 of the accepted operations. For example, the operation unit 15 receives user operations using input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 14 . Further, for example, the operation unit 15 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing apparatus 1.

In addition, the information processing device 1 according to the present embodiment includes a captured image acquisition section 11a, an annotation processing section 11b, and a display processing section 11c, by the processing section 11 reading out and executing the program 12a stored in the storage section 12. etc. are implemented in the processing unit 11 as software-like functional units. In this figure, as functional units of the processing unit 11, functional units related to processing that supports annotation based on images taken by the camera 3 are illustrated, and functional units related to other processes are omitted.

The photographed image acquisition unit 11a transmits and receives data to and from the camera 3 through the communication unit 13, thereby acquiring the photographed image of the object photographed by the camera 3, that is, through the plurality of filters 7a to 7f of the rotating filter 7. Performs processing to acquire multiple captured images. The photographed image acquisition unit 11a temporarily stores the acquired photographed image in the storage unit 12.

The annotation processing unit 11b performs processing to support the user's annotation work on images captured by the camera 3. For example, the annotation processing section 11b displays a plurality of images taken based on different wavelengths of light side by side on the display section 34, and receives a selection from the user as to which image is to be used for annotation. The annotation processing unit 11b enlarges and displays the selected image on the display unit 34, and receives from the user a designation of an image area to be given a specific label from among the displayed images. The annotation processing unit 11b adds information regarding the designated image area to the photographed image as learning data, and stores this learning data in the learning data storage unit 12b.

At this time, the annotation processing unit 11b accepts a selection of a specific location in the displayed image from the user using an input device such as a mouse, obtains the pixel value of the accepted specific location from the image, and selects a pixel value within a predetermined range that includes this pixel value. An area where pixels with a pixel value of . The user performs operations such as approving or modifying this candidate area, and the annotation processing unit 11b accepts the approved or revised area as an image area to which a specific label is attached.

In addition, the annotation processing unit 11b first determines one or more candidate regions that are candidates for image regions to be attached with a specific label based on the plurality of images, and adds a boundary line, figure, etc. indicating the candidate region to the original image. may be displayed in a superimposed manner. The user performs operations such as selecting an appropriate one from the displayed candidate area or modifying or deleting an inappropriate area, and the annotation processing unit 11b can be accepted as an image region to be labeled with a specific label.

The display processing unit 11c performs processing to display various information such as images and characters on the display unit 14. In the present embodiment, the display processing unit 11c performs various displays such as displaying captured images, displaying boundaries or figures indicating areas, and displaying messages to the user in support of annotation by the annotation processing unit 11b. conduct.

<Annotation support>
The information processing device 1 according to the present embodiment acquires a plurality of photographed images taken by the camera 3 through each of the filters 7a to 7f of the rotating filter 7, and stores the plurality of acquired images as one set in the storage unit 12. Store and accumulate a large set of captured images. The user uses the information processing device 1 to perform annotation work on the accumulated captured images. As annotation work, the user may, for example, specify an area in a photographed image where a specific object is photographed, or specify an area where an abnormal part of the object photographed in a photographed image is photographed. conduct. Note that the user may perform annotation work such as inputting a label indicating the classification of the object shown in the captured image, or inputting a label indicating whether or not there is an abnormality in the object captured in the captured image. You can also go as

The information processing device 1 reads out one set of photographed images stored in the storage unit 12, for example, and displays a plurality of images included in this one set side by side on the display unit 34. The information processing device 1 receives from the user a selection of a captured image to be used for annotation work from among a plurality of captured images of the same object captured using different wavelengths of light. The information processing device 1 enlarges and displays the photographed image selected by the user on the display unit 34, and receives an operation for specifying a region of the photographed image using the operation unit 35.

Furthermore, the information processing device 1 according to the present embodiment not only generates a plurality of photographed images taken by the camera 3, but also generates a composite image by combining these plurality of photographed images, and generates one or more composite images together with the plurality of photographed images. The composite images may be displayed side by side for selection by the user. FIG. 4 is a schematic diagram showing an example of a composite image. In this figure, four images, an "RGB image," an "infrared region image," a "red region image," and a "composite image" are displayed side by side on the display unit 34 of the information processing device 1.

The "RGB image" is an image normally taken by the camera 3, that is, an image taken by the camera 3 after switching to one of the plurality of filters 7a to 7f included in the rotating filter 7, which performs shooting substantially without a filter. It is. The "infrared region image" is an image taken by the camera 3 through a filter that transmits infrared rays of a predetermined wavelength (a predetermined range of wavelengths) and blocks light of other wavelengths. The "red area image" is an image taken by the camera 3 through a so-called red filter, which is a filter that transmits red wavelengths of visible light and blocks light of other wavelengths. These three photographed images are images obtained by photographing with the camera 3.

The "synthesized image" is an image that the information processing device 1 synthesizes the "infrared region image" and "red region image" described above. Since both the "infrared region image" and the "red region image" are images taken by the camera 3, the size of the images is the same, and the information processing device 1 uses the "infrared region image" and the "red region image". By calculating the following equation (1) for each corresponding pixel, the pixel value of each pixel of the composite image is calculated, and a composite image is generated.

Pixel value of composite image = (pixel value of infrared region image - pixel value of red region image) / (pixel value of infrared region image + pixel value of red region image) ... (1)

The composite image generated using the above equation (1) using the pixel values of the infrared region image and the red region image is an NDVI (Normalized Difference Vegetation Index) image, which is based on the reflection of light by plants. Show characteristics. In the composite image of FIG. 4, regions with large pixel values (that is, white or near-white colors in the image) are regions where plants exist. For example, if the object to be annotated by the user is a plant, by displaying an NDVI composite image, the user can easily understand the location of the plant in the image.

Note that the composite image generated by the information processing device 1 is not limited to the above-mentioned NDVI image. The information processing device 1 includes, for example, an NDWI (Normalized Difference Water Index) that can visualize the presence or absence of water, an NDSI (Normalized Difference Snow Index) that can visualize snowfall, and An image such as NDSI (Normalized Difference Soil Index) showing the distribution of sand, concrete, etc. may be generated as a composite image. The information processing device 1 selects two appropriate images from among the plurality of images taken by the camera 3, and calculates these two images based on the following equation (2), which is an extension of the above equation (1). Can be synthesized. Further, the information processing device 1 may further combine the photographed image and the composite image, or may further combine the composite image and the composite image.

Pixel value of composite image = (pixel value of first image - pixel value of second image) / (pixel value of first image + pixel value of second image) ... (2)

Note that the information processing device 1 does not need to generate and display the composite image. What kind of composite image should be generated depends on the type of object to be annotated. For this reason, for example, the information processing device 1 receives settings such as whether or not to synthesize captured images and the combination of captured images to be synthesized from the user performing the annotation, and creates the synthesized image according to the settings. May be generated. For example, before starting the annotation work, the information processing device 1 appropriately generates composite images by combining multiple types, displays a list of the generated multiple types of composite images, and allows the user to select a composite image suitable for the user. , a subsequent composite image may be generated under the conditions of the selected composite image.

FIG. 5 is a schematic diagram showing an example of an infrared region image. In this figure, two images, an "RGB image" and an "infrared region image", obtained by photographing an apple as a target object, are displayed side by side on the display unit 34 of the information processing device 1. The user performs annotation to specify, for example, an abnormal location (such as a scratched or painful location) on the apple photographed in the image. In the RGB image, since there is a pattern on the surface of the apple, it is difficult for the user to determine which part is abnormal. On the other hand, in an infrared region image taken using infrared light of a specific wavelength, the abnormal part of the apple (the part indicated by the white arrow in the figure) is shown in a different gradation from the normal part. Therefore, the user can easily identify the abnormal location.

6 and 7 are schematic diagrams for explaining an example of annotation work. This figure shows an example in which a user performs annotation work to identify abnormalities on an avocado on an infrared region image taken of an avocado as a target object. The information processing device 1 displays, for example, a plurality of captured images of an object taken by the camera 3 based on a plurality of wavelengths side by side on the display unit 34, and selects a captured image to be used for an annotation work from among these captured images. is accepted from the user. In this example, the user selects an infrared region image, and the information processing device 1 enlarges and displays the infrared region image photographed based on a predetermined infrared light from among the plurality of photographed images on the display unit 34.

For example, the user uses an input device such as a mouse provided as the operation unit 35 of the information processing device 1 to perform an operation such as clicking on an abnormal part in the infrared region image of the avocado displayed on the display unit 34. conduct. The upper part of FIG. 6 shows how an abnormal location is specified with a mouse cursor (white arrow) in an infrared region image of an avocado. The information processing device 1 acquires the pixel value of a pixel at a position (coordinates) specified by the user in the infrared region image, calculates a candidate area for the abnormality including the position specified by the user, and uses this candidate area as the original. Displayed superimposed on the infrared region image. The lower part of FIG. 6 shows how the candidate area is displayed as a polygonal figure superimposed on the original infrared region image. For example, the information processing device 1 selects as candidates a collection of pixels that include a pixel specified by the user, are connected to this pixel, and have the same pixel value as this pixel (the difference in pixel value from this pixel is within a predetermined range). It can be a region.

For example, the user can perform adjustment operations such as changing the shape or size of the candidate area superimposed on the infrared region image, as necessary. The upper part of FIG. 7 shows the candidate area in the lower part of FIG. 6 which has been adjusted to reduce its size. After the user completes the adjustment work as necessary, the information processing device 1 accepts this candidate area as an annotation result area, that is, an avocado abnormal area. Furthermore, the information processing device 1 calculates one or more candidate regions having the same characteristics as the determined abnormal region from the infrared region image, and displays the one or more candidate regions by superimposing them on the original infrared region image. do. The lower part of FIG. 7 shows how a plurality of candidate regions are displayed superimposed on the infrared region image together with the determined abnormal region. For example, the information processing device 1 acquires the pixel value range of a plurality of pixels included in the confirmed abnormal region, and determines whether a plurality of pixels having a similar range to this pixel value range are gathered and the number of pixels is less than the threshold value. By extracting an image area that exceeds the above, it can be set as one or a candidate area.

The user can, for example, perform operations such as deletion and adjustment of the candidate area on one or more displayed candidate areas, as necessary. After completing these tasks as necessary, the information processing device 1 accepts one or more candidate regions as an annotation result region, that is, an avocado abnormal region. The information processing device 1 receives the result of the user's annotation work on the photographed image according to the above-described procedure, and generates data indicating the annotation result.

The annotation result data generated by the information processing device 1 can be an array (matrix) of the same size as the image captured by the camera 3. For example, if the captured image has a size of 100 pixels x 100 pixels, the annotation result data will be a 100 x 100 array, and the element of the array corresponding to each pixel of the image will be ``1'' indicating abnormality or normal. The information processing device stores any value of "0" indicating that the annotation result data is generated. Note that the value set by the annotation is not limited to binary values such as "0" and "1", but may be three values such as "0", "1", and "2", for example, It may be greater than or equal to the value. Furthermore, the structure of the data as a result of annotation is not limited to this, and may be data that collects the coordinates of the vertices of a polygon indicating one or more regions shown in the lower part of FIG. 7, for example.

Note that when a user performs annotation work, the user often continuously and repeatedly performs the work on a large number of images. In such a case, the information processing device 1 processes, for example, the first annotation task according to the steps shown in FIGS. 6 and 7, and omits some of the steps for the second and subsequent annotation tasks. You can. For example, when the first annotation work is performed using an infrared region image, the information processing device 1 displays a list of multiple captured images taken based on different wavelengths of light for the second and subsequent annotation work. The infrared region image may be displayed without accepting the user's selection in the second and subsequent annotation operations by omitting the selection procedure. Furthermore, the information processing device 1 extracts a region having the same characteristics as the region determined as an abnormal region in the first annotation work from the symmetrical images of the second and subsequent annotations, and extracts a plurality of candidate regions as in the lower part of FIG. may be presented to the user as

The information processing device 1 that has generated the annotation result data associates this data with the photographed image as learning data, and stores it in the learning data storage section 12b. FIG. 8 is a schematic diagram for explaining an example of learning data. In the illustrated example, the camera 3 captures images through the rotating filter 7, thereby producing "RGB images," "red region images," "green region images," "blue region images," "infrared region images," and " It is assumed that six types of photographed images can be obtained: ``ultraviolet region images''. Further, it is assumed that the information processing device 1 receives the user's annotation work based on the "infrared region image" and generates data as an annotation result, as shown in FIGS. 6 and 7, for example.

The information processing device 1 generates, as learning data, data in which an "RGB image" that is one of six types of captured images is associated with data as a result of annotation by the user, and stores the learning data. It can be stored in the section 12b. In this way, the learning data may include all of the plurality of images captured by the camera 3, or may include only some of the images. In addition, when only some captured images are included in the learning data, the captured images to be included in the learning data are not limited to "RGB images" as shown in the figure, but may include only "infrared region images", for example. It is also possible to use other images. Further, the images of the training data may include, for example, a composite image obtained by combining multiple captured images, or only the composite image and the data resulting from annotation may be associated with each other and used as the training data. .

Which photographed images are included in the learning data is determined by the purpose of use or configuration of the learning model in which machine learning is performed using the learning data. For example, when using machine learning to generate a learning model that detects anomalies based on captured images (RGB images) taken by a normal camera, the learning data may include RGB images and annotation result data. . Further, for example, when generating a learning model that is incorporated into a camera that performs infrared photography to detect a target object, the learning data may include an infrared region image and annotation result data. For example, when generating a learning model to be incorporated into a camera that can take pictures by switching between multiple filters in the same manner as the rotating filter 7, the learning data includes multiple images corresponding to the filters possessed by the camera and annotation result data. may be included. The information processing device 1 may accept, for example, settings from the user regarding which captured images are included in the learning data.

<Flowchart>
FIG. 9 is a flowchart showing the procedure of annotation processing performed by the information processing device 1 according to the present embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing device 1 according to the present embodiment acquires in advance a plurality of captured images captured by the camera 3 via the rotating filter 7, and stores and accumulates them in the storage unit 12. are doing. The annotation processing unit 11b of the processing unit 11 acquires one set of a plurality of captured images from among the captured images accumulated in the storage unit 12 (step S1). The annotation processing unit 11b generates a composite image by appropriately combining the plurality of photographed images acquired in step S1, based on, for example, the user's setting of whether or not composition processing is necessary (step S2).

The display processing unit 11c of the processing unit 11 displays a plurality of images including the captured image acquired in step S1 and the composite image generated in step S2 on the display unit 34 (step S3). The annotation processing section 11b receives selection of one or more images to be used for annotation work from among the plurality of images displayed on the display section 34 (step S4). The display processing unit 11c displays the one or more images selected in step S4 on the display unit 34 (step S5).

The annotation processing unit 11b receives a designation of an annotation target location (position, coordinates, etc.) such as an abnormal location for the image displayed on the display unit 34 based on the user's operation on the operation unit 35 (step S6). The annotation processing unit 11b calculates a candidate area that is a candidate for an annotation target area, such as an abnormal area, including the target location received in step S6, and displays the candidate area by superimposing it on the original image (step S7 ). The annotation processing unit 11b receives adjustments to the shape or size of the displayed candidate area based on the user's operation of the operation unit 35 (step S8). For example, in response to a user's confirmation operation, the annotation processing unit 11b determines the candidate region, which has been modified as necessary, as the annotation target region (step S9).

Next, the annotation processing unit 11b calculates one or more other candidate regions similar to the symmetrical region determined in step S9, and displays the other candidate regions by superimposing them on the original image (step S10). The annotation processing unit 11b receives adjustments to the shape, size, etc. of the displayed other candidate areas based on the user's operation of the operation unit 35 (step S11). For example, in response to a confirmation operation performed by the user, the annotation processing unit 11b confirms another candidate area, which has been modified as necessary, as the annotation target area (step S12).

The annotation processing unit 11b generates annotation result data including information regarding the target area determined in steps S9 and S12 (step S13). The annotation processing unit 11b appropriately selects one or more captured images from the plurality of captured images acquired in step S1, and combines the selected one or more captured images with the annotation result generated in step S13. The learning data associated with the data is generated and stored in the learning data storage section 12b (step S14), and the process ends.

FIG. 10 is a flowchart showing the procedure of annotation processing performed by the information processing device 1 according to the present embodiment. This flowchart is a procedure for performing the second and subsequent annotation operations by omitting part of the processing shown in the flowchart of FIG. 9 when the user repeatedly performs annotation operations. The information processing device 1 may adopt either the procedure shown in FIG. 10 or the procedure shown in FIG. 9 for the second and subsequent annotation operations, and determines which procedure to perform processing based on the user's selection. It may be determined by

The annotation processing unit 11b of the processing unit 11 of the information processing device 1 according to the present embodiment acquires one set of a plurality of captured images from among the captured images accumulated in the storage unit 12 (step S31). The annotation processing unit 11b generates a composite image by appropriately combining the plurality of photographed images obtained in step S31, based on, for example, the user's setting of whether or not composition processing is necessary (step S32).

The annotation processing unit 11b selects an image with the same conditions as the image selected by the user in the previous annotation process from among a plurality of images including the captured image acquired in step S1 and the composite image generated in step S2. , displays the selected image on the display section 34 (step S33). The annotation processing unit 11b also calculates one or more candidate regions having the same characteristics as the target region determined in the previous annotation process, and displays the one or more candidate regions by superimposing them on the original image (step S34 ).

The annotation processing unit 11b receives adjustments to the shape or size of the displayed candidate area based on the user's operation of the operation unit 35 (step S35). For example, in response to a user's confirmation operation, the annotation processing unit 11b determines one or more candidate regions, which have been modified as necessary, as the annotation target region (step S36).

The annotation processing unit 11b generates annotation result data including information regarding the target area determined in step S36 (step S37). The annotation processing unit 11b appropriately selects one or more captured images from the plurality of captured images acquired in step S31, and combines the selected one or more captured images with the annotation result generated in step S37. The learning data associated with the data is generated and stored in the learning data storage section 12b (step S38), and the process ends.

<Summary>
In the information processing system according to the present embodiment configured as described above, the information processing device 1 acquires a plurality of photographed images of the same object taken by the camera 3 based on different wavelengths of light, and the acquired plurality of photographed images Based on at least one of the captured images, annotation result data (related information) is provided to at least one of the plurality of captured images. Note that at least one photographed image used for the annotation work and at least one photographed image to which annotation result data is added may be the same photographed image or may be different photographed images. By using multiple captured images taken based on different wavelengths of light for annotation, the annotation by the user becomes easier compared to when annotation is performed based on one captured image taken based on a single wavelength. It is expected that this work will become easier and more accurate.

Further, in the information processing system according to the present embodiment, the information processing device 1 receives from the user a designation of an image area for at least one of the plurality of captured images, and selects a similar area of the accepted image area as a candidate area. The designated image area and the extracted candidate area are extracted from the photographed image and are set as target areas for annotation, and information regarding these target areas is added to the photographed image as related information. Further, the information processing device 1 may display an image area specified by the user and a candidate area similar thereto in a superimposed manner on the original photographed image. These allow the user to easily specify multiple areas included in the captured image as target areas for annotation.

Further, in the information processing system according to the present embodiment, the information processing device 1 receives a selection of a target location (position, coordinates, etc.) for a photographed image from a user, determines an image area including the accepted target location, and selects a target location for a photographed image. Display image areas in a superimposed manner. At this time, the information processing device 1 can determine the image area by, for example, extracting from the captured image a location that has features similar to the features of the received target location. The information processing device 1 accepts modifications to the displayed image region, such as shape or size, from the user as necessary, and subjects the image region that has been modified as necessary (or not modified if unnecessary) to be annotated. Accepted as area specification. Thereby, by selecting a target location in an image with an input device such as a mouse or a pen tablet, the user can easily specify an area including this location as an annotation target area.

Further, in the information processing system according to the present embodiment, the information processing device 1 repeatedly performs annotation processing such as acquisition of captured images and addition of related information. After acquiring a plurality of captured images for the current annotation, the information processing device 1 selects one or more candidate regions from the current captured image based on the results of the previous annotation (results of adding related information). The candidate area is extracted and superimposed on the captured image to display the candidate area. At this time, the information processing device 1 can extract, for example, an image region having the same characteristics as the image region selected by the user in the previous annotation from the currently captured image as a candidate region. The information processing device 1 accepts modifications to the displayed candidate region, such as shape or size, from the user as necessary, and selects the candidate region that has been modified as necessary (or not modified if unnecessary) as an annotation target. The information regarding this candidate area is added to the photographed image as related information. This can be expected to reduce the burden on the user in annotation work where the same process is repeated.

Further, in the information processing system according to the present embodiment, the information processing device 1 generates a composite image by combining at least two captured images, and adds related information based on the generated composite image. As a result, the user can perform annotation work based on the composite image generated by the information processing device 1 in addition to the plurality of images taken by the camera 3, making the annotation work easier and more accurate. We can expect that

Further, in the information processing system according to the present embodiment, the plurality of captured images captured by the camera 3 include visible light images captured based on visible light (for example, RGB images) and non-visible light images captured based on invisible light. visible light images (for example, infrared region images or ultraviolet region images). The information processing device 1 performs annotation processing based on the non-visible light image, adds related information of the annotation result to the visible light image, and generates learning data. By performing machine learning using such learning data, it is expected that, for example, a learning model that handles images taken with visible light can be generated with high accuracy.

In this embodiment, the camera 3 and the rotary filter 7 are separate devices as shown in FIG. You may prepare by doing so. Further, in the information processing system according to the present embodiment, the information processing device 1 acquires the image taken by the camera 3, and the user performs annotation work using the information processing device 1. However, the configuration is not limited to this. Instead, a configuration may be adopted in which the camera 3 is provided with a display section, an operation section, etc., and the user performs the annotation work using the camera 3. Alternatively, the camera 3 may transmit the photographed image to a remote server device or the like by wireless communication or the like, and the information processing device 1 may communicate with the server device to acquire the photographed image. Furthermore, the images shown in FIGS. 4 to 7 are examples, and the information processing system according to this embodiment may handle any captured images and may perform any annotation processing.

Further, in this embodiment, annotation work for generating learning data used for machine learning of a learning model that performs so-called segmentation has been described as an example, but the present invention is not limited to this. For example, learning data used for machine learning, such as a learning model that classifies objects captured in a captured image into multiple classes, or a learning model that determines the presence or absence of abnormalities in captured images, that is, learning data for captured images. The present technology is also applicable to annotation work that generates learning data to which a single label indicating the presence or absence of a class or abnormality is attached. In this case, the information processing device 1 displays a list of a plurality of captured images captured by the camera 3 and a composite image appropriately generated from these captured images on the display unit 34, receives a label input from the user, and selects the captured images. Training data is generated in which at least one of the images is associated with the input label. Compared to simply looking at RGB images and labeling them, the user can see and label multiple images taken based on different wavelengths of light, thereby making it easier to label the images. It is possible to easily determine the class of the object and the presence or absence of an abnormality.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope.

Items described in each embodiment can be combined with each other. Moreover, the independent claims and dependent claims recited in the claims may be combined with each other in any and all combinations, regardless of the form in which they are cited. Further, although the scope of claims uses a format in which claims refer to two or more other claims (multi-claim format), the invention is not limited to this format. It may be written using a format that describes multiple claims (multi-multi-claims) that cite at least one multiple claim.

1 Information processing device 3 Camera 7 Rotation filter 7a to 7f Filter 11 Processing unit 11a Photographed image acquisition unit 11b Annotation processing unit 11c Display processing unit 12 Storage unit 12a Program 12b Learning data storage unit 13 Communication unit 14 Display unit 15 Operation unit

In an information processing method according to an embodiment, an information processing apparatus captures a first captured image of an object based on light in a first wavelength band and a second captured image based on light in a second wavelength band different from the first wavelength band. A plurality of photographed images of the same object are acquired based on different wavelengths of light, including a second photographed image of the target object, each of which is photographed based on a different wavelength of light, and at least one of the plurality of acquired images is obtained. Based on the second photographed image, related information is added to the first photographed image, and data in which the first photographed image and the related information are associated is outputted, and the data includes the first photographed image. Does not include the photographed image 2 .

In an information processing method according to an embodiment, an information processing apparatus captures a first captured image of an object based on light in a first wavelength band and a second captured image based on light in a second wavelength band different from the first wavelength band. A plurality of photographed images of the same object are acquired based on different wavelengths of light, including a second photographed image of the target object, each of which is photographed based on a different wavelength of light, and at least one of the plurality of acquired images is obtained. A designation of an image area for the second photographed image is accepted, and information regarding the image area is given as related information to the first photographed image included in the plurality of acquired photographed images. and adding related information to the first photographed image are repeatedly performed, and from at least the second photographed image among the plurality of acquired images, one or A plurality of candidate regions are extracted, superimposed on the second photographed image, the extracted candidate regions are displayed, a modification of the displayed candidate region is accepted, and information regarding the candidate region for which modification has been accepted is transmitted to the related As information, data is attached to the first photographed image included in the plurality of acquired photographic images, and data in which the first photographed image and the related information are associated is output, and the data includes the second photographed image. Does not include captured images.

Claims (9)

The information processing device
Obtain multiple images of the same object, each using a different wavelength of light,
Adding related information to at least one of the plurality of captured images based on at least one of the acquired plurality of captured images;
Information processing method. The information processing device
Accepting designation of an image area for at least one photographed image among the plurality of acquired photographed images,
Extracting a similar region to the accepted image region from the photographed image,
providing information regarding the image area and the similar area as the related information;
The information processing method according to claim 1. The information processing device
displaying the image region and the similar region superimposed on the photographed image;
The information processing method according to claim 2. The information processing device
Accepting selection of a target location for at least one of the plurality of acquired images,
determining an image area including the accepted target location;
displaying the determined image area superimposed on the captured image;
Accepting corrections to the displayed image area,
accepting the image area for which modification has been accepted as the designation;
The information processing method according to claim 2 or 3. The information processing device
Repeatedly acquiring the photographed image and adding the related information,
extracting one or more candidate regions from at least one photographed image among the plurality of acquired images, based on the result of the previous addition of the related information;
displaying the extracted candidate area superimposed on the captured image;
Accepting corrections to the displayed candidate area,
adding information regarding the candidate area for which modification has been accepted as the related information;
The information processing method according to claim 2 or 3. The information processing device
Generating a composite image by combining at least two of the plurality of captured images,
Adding related information to at least one photographed image among the plurality of photographed images based on the generated composite image;
The information processing method according to claim 1. The plurality of photographed images include a visible light image photographed based on visible light and a non-visible light image photographed based on non-visible light,
the information processing device adds related information to the visible light image based on the non-visible light image;
The information processing method according to claim 1. to the computer,
Obtain multiple images of the same object, each using a different wavelength of light,
A computer program that causes a computer program to execute a process of adding related information to at least one of the plurality of captured images based on at least one of the plurality of captured images. an acquisition unit that acquires a plurality of captured images of the same object, each of which is captured based on a different wavelength of light;
An information processing device, comprising: a provision unit that adds related information to at least one of the plurality of captured images, based on at least one of the acquired plurality of captured images.

Images