JP2024025499A - Information processing device, complete convolutional network producing method and program - Google Patents

Abstract

To sort array data highly precisely at fast-speed.SOLUTION: A likelihood for array data to be included in a specific class is calculated. The likelihood for the array data to be included in the specific class is calculated by a different process that uses a complete convolutional network. Using information obtained during a process in which first processing means calculates a likelihood of array data for learning as teacher data, learning process on the complete convolutional network is executed.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, a method for producing a fully convolutional network, and a program, and particularly relates to segmentation processing for array data.

In recent years, techniques for classifying array data such as images and techniques for extracting part of the array data based on the classification results have been proposed. For example, by processing an image containing an object, useful information can be extracted from the image. In particular, object recognition technology that uses neural networks (eg, multilayer deep neural networks) to recognize categories of objects in images is being actively researched. Furthermore, techniques for segmenting images by estimating the categories of objects in images at the pixel level are also being actively researched.

For example, Non-Patent Document 1 proposes a Vision Transformer that performs object recognition using a Transformer architecture. In this method, processing is performed using the entire intermediate processing result in the Self-Attention layer or the like. On the other hand, Non-Patent Document 2 proposes a technique for performing image segmentation using a fully convolutional network consisting of only convolutional layers and pooling layers.

A. Dosovitskiy et al. "An Image Is Worth 16x16 Words: Transformers For Image Recognition at Scale", arXiv:2010.11929 J. Long et al. "Fully Convolutional Networks for Semantic Segmentation", Computer Vision and Pattern Recognition (CVPR) 2015

In the method of Non-Patent Document 1, the object recognition accuracy is improved by processing the entire image in each layer, but the processing takes time. On the other hand, according to the method of Non-Patent Document 2, since local filter processing is performed in the convolutional layer, high-speed segmentation processing can be performed, but the processing accuracy is lower than that of the method of Non-Patent Document 1. There is a tendency. In addition, in the method of Non-Patent Document 1, processing of each layer is performed while referring to the entire image, and in the method of Non-Patent Document 2, processing of each layer is performed while referring to a local area of the image. It was also not easy to incorporate the method shown in the method shown in Non-Patent Document 2.

The present invention aims to classify sequence data with high precision and high speed.

An information processing device according to an embodiment of the present invention has the following configuration. That is,
a first processing means for calculating the likelihood that the array data is included in a specific class;
a second processing means that performs processing different from the first processing means, the second processing means calculating the likelihood that the array data is included in a specific class using a fully convolutional network;
Learning means for performing learning processing for the fully convolutional network using information obtained in the process in which the first processing means calculates the likelihood for the learning array data as teacher data;
Equipped with

Sequence data can be classified with high accuracy and high speed.

FIG. 1 is a block diagram of an information processing device according to an embodiment. 5 is a flowchart showing the procedure of learning processing according to an embodiment. 7 is a diagram illustrating an example of processing by the second processing unit 102. FIG. 6 is a diagram illustrating an example of a learning method of the second processing unit 102. FIG. 6 is a diagram illustrating an example of a learning method of the second processing unit 102. FIG. 6 is a diagram illustrating an example of a learning method of the second processing unit 102. FIG. FIG. 1 is a block diagram of an information processing device according to an embodiment. FIG. 3 is a diagram illustrating a method for confirming a network structure using test data. FIG. 3 is a diagram illustrating an example of an additional learning method of the first processing unit 101. FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

An information processing device according to an embodiment of the present invention includes a first processing section, a second processing section, and a learning section. This information processing device can classify array data. In the following example, the information processing device classifies data around a plurality of positions in array data. Further, this information processing device can perform segmentation of array data according to the classification results. In the following example, the information processing apparatus performs processing to divide an image, particularly according to the class of an object shown in the image. Such an information processing device can be used, for example, as an object recognition system that recognizes objects in images.

The first processing unit can perform the same processing as in Non-Patent Document 1, for example. Further, the second processing unit can perform the same processing as in Non-Patent Document 2, for example. According to one embodiment of the present invention, the second processing section can classify array data with accuracy close to that of the first processing section that has been trained in advance. On the other hand, the second processing unit can perform the classification and segmentation processing of array data at high speed as in Non-Patent Document 2.

The array data includes, for example, data arranged along one or more coordinate axes. Examples of array data include audio data having sound pressure data arranged along a one-dimensional time axis, and image data having pixel data arranged along a two-dimensional coordinate axis. The array data may be voxel data having pixel data arranged along three-dimensional coordinate axes. An example in which the array data is image data will be described below. In the following example, the second processing unit uses a two-dimensional fully convolutional network to process the image data. On the other hand, the second processing unit can process the audio data using a fully convolutional network in the time axis direction. Further, the second processing unit can process the voxel data using a three-dimensional fully convolutional network.

FIG. 1 is a block diagram showing the configuration of an information processing device 1 according to an embodiment of the present invention. The first processing unit 101 calculates the likelihood that the array data is included in a specific class. For example, the first processing unit 101 can calculate the likelihood that the array data is included in the class for each of a plurality of classes. In this embodiment, the first processing unit 101 performs a process of generating object likelihood from an image.

In this example, array data of a predetermined size is input to the first processing unit 101. For example, fixed-sized learning array data (that is, learning images) is input to the first processing unit 101 . The first processing unit 101 then calculates the object likelihood for the input image. Object likelihood indicates an estimated value of the probability that an object appearing in an input image belongs to a predetermined category (ie, class) for each of one or more categories.

FIG. 4A is a diagram illustrating an example of processing by the first processing unit 101. An input image 10101 is input to the first processing unit 101. In order to improve classification accuracy, the first processing unit 101 can calculate object likelihood using Self-Attention processing, Transformer, or repetition of processing using all of the input features.

In this embodiment, the first processing unit 101 performs processing using Vision Transformer shown in Non-Patent Document 1. VisionTransformer first transforms the input image into features. The VisionTransformer then applies a multilayer perceptron called an encoder and a block containing Self-Attention to the feature L times. VisionTransformer finally transforms the obtained features into likelihoods for each category of objects.

In this embodiment, the parameters used for processing by the first processing unit 101 are adjusted so that the object likelihood can be estimated. For example, the Vision Transformer used by the first processing unit 101 is trained in advance to estimate object likelihood. Here, learning means adjusting parameters that Vision Transformer has. Although the learning method is not particularly limited, for example, the following method can be used. That is, the validity of the estimation is determined based on the result of applying the softmax function to the object likelihood and the label representing the category of the object in the image. At this time, it is possible to set a loss function that indicates the validity of the estimation. Then, by performing backpropagation so that the value of the loss function decreases, the parameters of the VisionTransformer can be adjusted.

In the example of FIG. 4A, an input image 10101 is converted into features 10103 by linearly converting pixel values. When using VisionTransformer, the data to be processed is divided into multiple features, and VisionTransformer handles the features obtained by the division. In this embodiment, an image is divided into partial images of the same size, and features are obtained by performing linear transformation on each partial image.

Features 10103 are input to a first encoder block 10104. The first encoder block 10104 includes a multilayer perceptron and self-attention processing, and outputs features of the same dimension as the input features. The first encoder block 10104 outputs the generated features to the second encoder block 10105. Each encoder block including the second encoder block 10105 similarly processes the features output from the previous encoder block and outputs the generated features. The features output from the L-th encoder block 10106 are input to a likelihood conversion process 10107 that converts the features into object likelihoods. VisionTransformer obtains object likelihood 10108 by performing multilayer perceptron processing on some dimensions of the features output from L-th encoder block 10106. The object likelihood 10108 is the object likelihood output by the first processing unit 101. Each dimension of object likelihood 10108 corresponds to a category of object. Each dimension of object likelihood 10108 represents an estimate of the probability that the object is included in the corresponding category. In the example of FIG. 4A, the object likelihood 10108 indicates the first category likelihood to the d-th category likelihood, which indicate estimated values of the probability that the object belongs to each of the first category and the d-th category. In this way, object likelihood can be represented by a vector.

Self-Attention processing included in the encoder blocks 10104 to 10106 of the Vision Transformer performs processing using all values of input features. In this way, in processing using VisionTransformer, processing is performed that effectively utilizes the information of the entire image. On the other hand, in convolution processing, processing is performed using only the features corresponding to the filter size among the input features, so local features are output. Therefore, processing using Vision Transformer improves the accuracy of estimating the category of an object compared to a convolutional neural network that performs convolution processing.

The second processing unit 102 uses a fully convolutional network to calculate the likelihood that the array data is included in a specific class. For example, the second processing unit 102 calculates the likelihood that the array data is included in the class for each of the plurality of classes. Here, the second processing unit 102 calculates the likelihood using a process different from that of the first processing unit 101. In this embodiment, the second processing unit 102 performs a process of generating an object likelihood map including one or more object likelihoods from an image. The object likelihood map has coordinates that correspond to image coordinates. Further, in the object likelihood map, each coordinate element represents the object likelihood. In this embodiment, the second processing unit 102 can calculate an object likelihood map that represents the object likelihood for each pixel or partial region of the input image.

FIG. 3A is a diagram illustrating an example of processing by the second processing unit 102. The image 10201 is input to the second processing unit 102. In this embodiment, the second processing unit 102 uses a fully convolutional network shown in Non-Patent Document 2. A fully convolutional network is a neural network in which all layers are convolutional or pooling layers. In processing using a fully convolutional network, a feature map is obtained by applying convolution processing or pooling processing to an input image multiple times (M times in FIG. 3A). The feature map is then converted into an object likelihood map and output. Note that in each layer, in addition to convolution processing or pooling processing, application of an activation function or bias may be performed.

As described above, array data of a predetermined size is input to the first processing unit 101. On the other hand, array data larger than this predetermined size can be input to the second processing unit 102. When the first array data of a predetermined size is input, the fully convolutional network used by the second processing unit 102 generates one object likelihood ( For example, it is adjusted to output one vector). For example, the fully convolutional network outputs one object likelihood when an image of the same size as the input image to the first processing unit 101 is input. In the example of FIG. 4B, the number of convolution layers and pooling layers after adjustment is M. For example, if the size of the input image to the first processing unit 101 is 99×99 pixels, the fully convolutional network may have 49 convolutional layers that perform convolution processing using 3×3 filters. When a 99x99 pixel image is input to such a neural network, one object likelihood is output. Note that, as will be described later, since padding processing is not performed in this convolutional layer, the vertical and horizontal sizes of the image are reduced by two pixels in one convolutional layer.

On the other hand, as shown in FIG. 3A, when second array data larger than a predetermined size is input to the fully convolutional network used by the second processing unit 102, the fully convolutional network uses an object including multiple object likelihoods. Output the likelihood map. For example, when an image larger in size than the input image to the first processing unit 101 is input, the second processing unit 102 can generate a likelihood map according to the size of the input image. In this case, the object likelihood map indicates the likelihood that each of the plurality of partial array data is included in a specific class. Here, each of the plurality of partial array data is part of the second array data. For example, the partial array data is a partial image of a predetermined size that is part of an image larger than the predetermined size.

Furthermore, in the present embodiment, the fully convolutional network is configured not to perform padding processing or hierarchical processing in which the processing contents vary depending on variations in the size of array data. In this embodiment, the size of an image (for example, image 10101 in FIG. 4B) that is input when the second processing unit 102 learns and the size that is input when estimating the object likelihood using the second processing unit 102 are determined. (for example, image 10201 in FIG. 3A). Therefore, if the processing performed by the second processing unit 102 includes hierarchical processing in which the processing content changes as the input image size changes, it becomes difficult to improve the accuracy of the segmentation processing even by learning. Therefore, for example, a fully convolutional network can be configured such that no padding is performed in the convolutional layer and the pooling layer. By removing the influence of padding at the edges of the image, the result of the estimation process does not change even if the size of the input image to the second processing unit 102 changes. Therefore, more accurate segmentation processing becomes possible.

The image 10201 is input to the first layer processing 10203 of the second processing unit 102, which is processing in a convolution layer or a pooling layer. In the first layer processing 10203, the input image is processed. Since convolution processing or pooling processing is performed in the first layer processing 10203, the processing result obtained in the first layer processing 10203 is a feature map having the same coordinate axes as the image 10201. The processing results obtained in the first layer processing 10203 are output to the second layer processing 10204. Similarly, each layer process including the second layer process 10204 and the Mth layer process 10205 is also a process in a convolution layer or a pooling layer. In each of these hierarchical processes, the output from the previous hierarchical process is similarly processed, and the processing results are output.

The feature map obtained by the Mth layer processing 10205 is input to the likelihood conversion processing 10206. In the likelihood conversion process 10206, an object likelihood map 10207 is generated by converting each feature shown in the feature map into an object likelihood. Then, the second processing unit 102 outputs this object likelihood map 10207. The likelihood conversion process 10206 is similar to the likelihood conversion process 10107 in the first processing unit 101. Since the second processing unit 102 handles feature maps, likelihood conversion processing 10206 is performed by convolution processing. For example, conversion from a feature map to an object likelihood can be performed by one-layer convolution processing.

Object likelihood map 10207 includes object likelihoods 10208 and 10209. Each object likelihood corresponds to a pixel or a partial region of the image 10201. Furthermore, the positional relationship of object likelihoods on the object likelihood map 10207 corresponds to the positional relationship on the image 10201. For example, object likelihood 10208 represents the object likelihood for a partial region located further to the left of image 10201 compared to object likelihood 10209.

FIG. 3C shows a likelihood map corresponding to an object of category A, which is represented by an object likelihood map obtained by inputting the image 10001 shown in FIG. 3B to the second processing unit 102. FIG. 3(D) shows a likelihood map corresponding to an object of category B, which is indicated by the same object likelihood map. The value of each likelihood map is high near the area of image 10001 where an object of the corresponding category exists.

In each of the hierarchical processes 10203 to 10205 included in the fully convolutional network, features adjacent in the coordinate axis direction are commonly used. On the other hand, when using Vision Transformer, all features in the coordinate axis direction are used in Self-Attention processing and multilayer perceptron processing. For this reason, the calculation process is completely different between processing for a certain input image and processing for an image obtained by translating this input image by one pixel, so intermediate features generated in the calculation process are shared. It is not possible. In this way, processing using a fully convolutional network in which processing is performed while sharing intermediate features generated during the calculation process is more computationally efficient than processing using Vision Transformer.

The learning unit 103 uses information obtained during the process in which the first processing unit 101 calculates the likelihood of the learning array data as training data to create a fully convolutional network used by the second processing unit 102. The learning process is performed. In this specification, learning of the fully convolutional network used by the second processing unit 102 may be referred to as learning of the second processing unit 102. For example, the learning unit 103 can transmit information used when the first processing unit 101 calculates the object likelihood based on the input image to the second processing unit 102. As described above, the first processing unit 101 performs processing using Vision Transformer. On the other hand, the VisionTransformer has a different structure from the fully convolutional network included in the second processing unit 102. Therefore, it is not possible to simply move the parameter used by the first processing unit 101 to the second parameter.

In this embodiment, the learning unit 103 performs learning of the second processing unit 102 using teacher data. As the teacher data, a set of the image 11 and the object likelihood output by the first processing unit 101 to which the image 11 is input is used. Through such learning, the second processing unit 102 is trained to have performance close to the object likelihood estimation performance by the first processing unit 101. Then, the second processing unit 102 can perform segmentation processing using such performance.

The learning unit 103 includes a likelihood acquisition unit 1031 and a likelihood learning unit 1032. The likelihood acquisition unit 1031 acquires the object likelihood output by the first processing unit 101 to which the image 11 is input. The likelihood acquisition unit 1031 then outputs the acquired object likelihood to the likelihood learning unit 1032.

The likelihood learning unit 1032 performs learning of the second processing unit 102. Specifically, the likelihood learning unit 1032 acquires the image 11 and the object likelihood corresponding to the image 11 output by the first processing unit 101 as teacher data. Then, the likelihood learning unit 1032 performs learning of the second processing unit 102 using this teacher data. The likelihood learning unit 1032 calculates the difference between the object likelihood output by the first processing unit 101 and the object likelihood shown in the object likelihood map output by the second processing unit 102 to which the image 11 is input. The parameters of the second processing unit 102 are updated so as to decrease. That is, the likelihood learning unit 1032 can update the parameters of the fully convolutional network used by the second processing unit 102. The likelihood learning unit 1032 can use, for example, backpropagation for learning, and can use, for example, the L1 norm of the difference in object likelihood as a loss function referred to for learning.

FIG. 2 is a flowchart illustrating a procedure of a learning processing method according to an embodiment, which is performed by the information processing device 1 according to the embodiment. Through this processing, it is possible to create a segmentation network that performs segmentation processing on images. According to such processing, it is possible to generate a neural network that performs classification processing and segmentation processing with high precision and high speed.

In S1001, data acquisition processing is performed. Specifically, the likelihood acquisition unit 1031 acquires the image 11. In this embodiment, the likelihood acquisition unit 1031 acquires a plurality of images 11 for which the first processing unit 101 estimates the object likelihood. In this embodiment, the acquired image 11 is a color image. However, the likelihood acquisition unit 1031 may acquire a grayscale image or a distance image as the image 11.

In S1002 and S1003, the learning unit 103 performs information transmission processing to transmit information from the first processing unit 101 to the second processing unit 102. In S1002, the first processing unit 101 performs likelihood acquisition processing. Specifically, the likelihood acquisition unit 1031 inputs the image 11 to the first processing unit 101. Next, the first processing unit 101 performs a process of calculating object likelihood for the image 11. Then, the likelihood acquisition unit 1031 acquires the object likelihood calculated from the first processing unit 101. As described above, the object likelihood indicates, for each of one or more categories, the estimated probability that an object appearing in the input image belongs to this category.

と表せる。また、尤度取得部１０３１は、取得した画像１１とオブジェクト尤度とを尤度学習部１０３２に出力する。 The likelihood acquisition process by the first processing unit 101 is performed as described with reference to FIG. 4(A). In this example, an image 10101 in FIG. 4A is the image 11 input to the first processing unit 101. Here, if the number of categories is d, the object likelihood u corresponding to the image 11 outputted by the first processing unit 101 and represented as the object likelihood 10108 in FIG.

It can be expressed as Furthermore, the likelihood acquisition unit 1031 outputs the acquired image 11 and object likelihood to the likelihood learning unit 1032.

In S1003, likelihood learning processing is performed by the likelihood learning unit 1032. For example, the likelihood learning unit 1032 uses the acquired image 11 and object likelihood as teacher data to perform learning of the fully convolutional network used by the second processing unit 102 as described above.

FIG. 4(B) is a diagram illustrating the likelihood learning process in this embodiment. Each process 10203 to 10206 in the second processing unit 102 shown in FIG. 4(B) is the same as that in FIG. 3(A). On the other hand, in this example, the image 11 that is also input to the first processing section 101 is input to the second processing section 102 . An image 10101 in FIG. 4B is the image 11 input to the second processing unit. As already explained, when an image of the same size as the input image to the first processing unit 101 is input, the second processing unit 102 outputs an object likelihood map 10217 consisting of one object likelihood 10218. do. The object likelihood v expressed as the object likelihood 10218 in FIG. 4B is a vector 10219 that is included in the object likelihood map 10217 and indicates the likelihood of the category of the object appearing in the image 11. The object likelihood v has the same dimensions as the object likelihood u.

を用いることができる。ロス関数はこれに限られず、例えばロス関数としてＬ２距離などが用いられてもよい。そして、このようなロス関数に基づいて第２の処理部１０２の最適化を行うことができる。例えば、Ａｄａｍ法に従って、第２の処理部１０２が用いるパラメータをロス関数の値が減少するように最適化することができる。 In this case, the learning unit 103 uses the likelihood calculated by the first processing unit 101 for the third array data for learning (for example, image 11) and the likelihood calculated by the second processing unit 102 for the third array data. The second processing unit 102 can be trained based on the difference between the likelihood and the likelihood. Specifically, the learning unit 103 can perform learning on the second processing unit 102 so that such a difference becomes small. As a loss function used to evaluate such a difference, for example,

can be used. The loss function is not limited to this, and for example, the L2 distance may be used as the loss function. Then, the second processing unit 102 can be optimized based on such a loss function. For example, according to the Adam method, the parameters used by the second processing unit 102 can be optimized so that the value of the loss function decreases.

The second processing unit 102 can be trained by repeatedly performing optimization according to S1001 to S1003 using each of a sufficient number of images 11. Thereafter, by inputting the image 21 to the second processing unit 102 after learning, the second processing unit 102 outputs a likelihood map corresponding to the image 21.

By the way, as shown in FIG. 4(B), the second processing unit 102 that has been trained in this way will process the first processing unit 102 when an image of the same size as the input image to the first processing unit 101 is input. The processing unit 101 outputs the same object likelihood. On the other hand, as shown in FIG. 3A, when an image larger in size than the input image to the first processing unit 101 is input, the second processing unit 102 performs processing according to the size of the input image. generate a likelihood map. This likelihood map indicates object likelihood, which is the likelihood that a partial image included in the input image is included in a specific class. That is, the category of the object included in the image 21 can be estimated based on this likelihood map. The second processing unit 102 trained as described above can perform this category estimation with an accuracy close to that of the object category estimation accuracy by the first processing unit 101.

Further, according to the object likelihood shown in the likelihood map output by the second processing unit 102, the image can be segmented for each category of the object in the image. Therefore, the second processing unit 102 trained in this way can be used to segment an image. In one embodiment, the second processing unit 102 performs segmentation of array data based on a likelihood map.

As described above, the information processing device according to the present embodiment has a second processing unit that has accuracy close to that of the first processing unit 101 that has been trained in advance and can efficiently calculate object likelihood. 102 learnings can be performed. In this way, the information processing apparatus according to this embodiment can produce a fully convolutional network whose parameters have been learned. Thereafter, the second processing unit 102 of the information processing apparatus according to the present embodiment can perform classification processing or segmentation processing of the array data using the fully convolutional network whose parameters have been learned. Further, the information processing apparatus according to the present embodiment may output parameters of a fully convolutional network obtained through learning. In this case, the other information processing device can perform classification processing or segmentation processing of the array data using the parameters of the fully convolutional network output from the information processing device learning according to this embodiment. With either method, classification processing and segmentation processing can be performed with high accuracy and high speed using a fully convolutional network.

In the embodiment described above, the learning unit 103 uses the object likelihood output by the first processing unit 101 as training data for the second processing unit 102 to learn. On the other hand, the learning unit 103 calculates the likelihood that the learning array data is included in the class based on other information obtained in the process in which the first processing unit 101 calculates the likelihood that the learning array data is included in the class. You can learn about.

For example, the learning unit 103 can use intermediate features obtained during the processing of the image 11 by the first processing unit 101 as training data for the second processing unit 102 to learn. In one embodiment, the characteristics of the learning array data obtained in the process of calculating the likelihood by the first processing unit 101 for the learning array data, and the learning The characteristics of the array data are used. The learning unit 103 performs learning processing on the fully convolutional network used by the second processing unit 102 so that the difference between these features becomes small.

For example, the first processing unit 101 can output the feature output by the L-th encoder block 10106 in the example of FIG. 4(A). This feature is called an intermediate feature. This intermediate feature is, for example, a p-dimensional vector. The learning unit 103 can acquire such intermediate features from the first processing unit 101. This intermediate feature can be used as training data.

Further, the second processing unit 102 can output the feature map obtained in the Mth layer processing 10205 in the example of FIG. 4(B). This feature map is called an intermediate feature map. In this example, this intermediate feature map is a 1×1×p-dimensional map, that is, it indicates a feature represented by a p-dimensional vector. The learning unit 103 can acquire such an intermediate feature map from the second processing unit 102.

In this case, the learning unit 103 reduces the difference between the intermediate feature output by the first processing unit 101 corresponding to the image 11 and the intermediate feature map output by the second processing unit 102 to which the image 11 is input. As such, the parameters of the second processing unit 102 can be updated. Parameters can be updated in the same way as in the above embodiments.

The object likelihood calculation and image segmentation performed by the second processing unit 102 learned according to this embodiment will be described with reference to FIG. 5. In this case, the second processing unit 102 can calculate the characteristics of the specific class of array data using a fully convolutional network. Then, the second processing unit 102 identifies the array data to be processed based on the correlation between the characteristics of the array data of a specific class and the characteristics of the array data to be processed calculated using the fully convolutional network. The likelihood of being included in the class can be calculated. In the example below, processing is performed based on the similarity between intermediate features output by the second processing unit 102 for an image of a specific category and intermediate features output by the second processing unit 102 for an image to be processed. The target image is segmented.

First, an image 10231 of an object in a certain category is input to the second processing unit 102 after learning. In this example, the size of the image 10231 is the same as the size of the image input to the first processing unit 101. Then, the second processing unit 102 outputs the intermediate feature map 10237 obtained by the Mth layer processing 10205. This intermediate feature map 10237 includes a feature 10239 that serves as a similarity criterion. In the example of FIG. 5, feature 10239 is a p-dimensional vector.

Furthermore, an image 10241 to be subjected to segmentation processing is input to the second processing unit 102 after learning. The size of this image 10241 may be larger than the size of image 10231. Then, the second processing unit 102 outputs the intermediate feature map 10247 obtained by the M-th layer processing 10205. In the example of FIG. 5, this intermediate feature map includes a plurality of p-dimensional vectors, each corresponding to a pixel or subregion of image 10241.

Next, the second processing unit 102 obtains a similarity map 10249 by performing inner product processing on the intermediate feature map 10247 and the feature 10239. This similarity map 10249 indicates the object likelihood in a partial region of the image 10241. The object likelihood in this case indicates an estimated value of the probability that the object appearing in the partial area of image 10241 belongs to the category of the object appearing in image 10231. Further, the second processing unit 102 can segment the image 10241 based on the object likelihood indicated by the similarity map 10249. In this way, the image 10241 can be segmented so that the area where the object shown in the image 10231 is present can be distinguished.

In this embodiment, the second processing unit 102 is trained to generate an intermediate feature map. With such a configuration, image segmentation can be performed based on similarity with arbitrary image features instead of estimating object likelihood for a predetermined category.

Furthermore, in the embodiments described above, the object likelihoods output by the first processing unit 101 are respectively used as training data for learning by the second processing unit 102. On the other hand, an object likelihood map in which a plurality of object likelihoods outputted by the first processing unit 101 are compiled may be used as training data for learning by the second processing unit 102. Such an embodiment will be described with reference to FIGS. 6(A) and 6(B).

In this case, the first processing unit 101 calculates the likelihood that each of the plurality of array data is included in a specific class. Here, each of the plurality of array data input to the first processing unit 101 has a predetermined size. Moreover, these plurality of array data are part of the third array data for learning which is larger than this predetermined size. For example, the first processing unit 101 can calculate object likelihoods corresponding to each of a plurality of images (for example, four images). The likelihood acquisition unit 1031 can acquire these object likelihoods output by the first processing unit 101.

For example, in the example of FIG. 6A, the size of the image 20301 that is the third array data for learning is larger than the image size that can be input to the first processing unit 101. The four images 20302, which are a plurality of array data, are sampled images smaller than the image 20301 from the image 20301. Then, the first processing unit 101 outputs object likelihoods 20311 to 20314 for each of the four images by performing the object likelihood calculation process four times. In the example of FIG. 6A, image 11 is shown as image 20301.

The number of images generated by sampling by the likelihood acquisition unit 1031 is the same as the number of object likelihoods output by the second processing unit 102 to which the image 20301 is input. Furthermore, the likelihood acquisition unit 1031 samples the image so that the image has a size suitable for input to the first processing unit 101. Furthermore, during sampling, the sampling position is shifted to match the positional relationship of the plurality of object likelihoods output by the second processing unit 102, as shown as four images 20302.

Furthermore, the likelihood acquisition unit 1031 generates a likelihood map including the likelihood calculated by the first processing unit 101. For example, the likelihood acquisition unit 1031 converts object likelihoods 20311 to 20314 into an object likelihood map 20304. Here, the likelihood map has the likelihood calculated by the first processing unit 101 for this sampling image at the sampling position of the sampling image. For example, in FIG. 6A, the arrangement of object likelihoods on the object likelihood map 20304 matches the positional relationship of the images sampled by the likelihood acquisition unit 1031. Specifically, the object likelihood 20311 corresponding to the upper left sampling image is placed at the upper left of the object likelihood map 20304. For this purpose, the likelihood acquisition unit 1031 determines a numerical value indicating the positional relationship of the image coordinates. In this example, object likelihood 20311 is expressed as u _0,0 , object likelihood 20312 is expressed as u _1,0 , object likelihood 20313 is expressed as u _0,1 , and object likelihood 20314 is expressed as u _1,1 . The likelihood acquisition unit 1031 then outputs the generated object likelihood map 20304 to the likelihood learning unit 1032.

ここで、ｖ_ｉ，ｊはオブジェクト尤度マップ２０３０６に示されるオブジェクト尤度であり、ｉ，ｊはｕ_ｉ，ｊと同様にマップ上の位置を表す。 The likelihood learning unit 1032 performs learning of the second processing unit 102. The likelihood learning unit 1032 acquires the image 20301 and the object likelihood map 20304 corresponding to the image 20301 from the likelihood acquisition unit 1031. Then, the likelihood learning unit 1032 performs learning of the second processing unit 102 using the image 20301 and the object likelihood map 20304 as teacher data. Specifically, the likelihood learning unit 1032 uses a likelihood map including the likelihood calculated by the first processing unit 101 and a likelihood calculated by the second processing unit 102 for the third array data for learning. The second processing unit 102 can perform learning based on the difference between the map and the map. Specifically, the likelihood learning unit 1032 uses the second The parameters of the processing unit 102 can be updated. As the loss function used to evaluate the difference, for example, the sum of the L1 norm of the difference of each object likelihood can be used. For example, the loss function can be expressed by the following formula.

Here, v _i,j is the object likelihood shown in the object likelihood map 20306, and i,j represents the position on the map similarly to u _i,j .

According to such an embodiment, a set of a plurality of object likelihoods is used as training data, so that learning efficiency is improved.

(Check network structure)
According to the above embodiment, the neural network used by the second processing unit 102 can be trained to perform classification processing and segmentation processing with high accuracy and high speed. On the other hand, it is possible to further confirm whether the neural network used by the second processing unit 102 has a structure capable of performing sufficiently accurate and high-speed classification processing and segmentation processing.

FIG. 7 is a block diagram showing the configuration of the information processing device 3 according to one embodiment. This information processing device 3 can confirm the structure of the neural network used by the second processing unit 102. The configurations of the first processing unit 101, the second processing unit 102, and the learning unit 103 are the same as those in FIG. 1, and the different points will be explained below.

In this embodiment, the second processing unit 102 can receive the network structure 12 as input. The network structure 12 is information indicating the structure of a neural network used by the second processing unit 102. The network structure 12 can indicate, for example, the number of hierarchical processes, the type of hierarchical processes, the order in which the hierarchical processes are performed, and the number of categories after likelihood conversion. The second processing unit 102 can generate an object likelihood map using a fully convolutional network having a structure according to the network structure 12. Further, the second processing unit 102 can output the received network structure 12 to the confirmation unit 304.

The confirmation unit 304 confirms whether the network structure 12 exhibits a structure that allows classification processing and segmentation processing to be performed with sufficiently high accuracy and high speed. Such processing can be performed, for example, before starting the processing according to FIG. 2. The confirmation unit 304 can confirm, for example, whether the fully convolutional network used by the second processing unit 102 includes padding processing or hierarchical processing in which the processing contents vary depending on variations in the size of array data.

As described above, when the processing performed by the second processing unit 102 includes hierarchical processing in which the processing content changes as the input image size changes, the accuracy of the segmentation processing becomes difficult to improve through learning. When a neural network includes such hierarchical processing, by modifying the structure of the neural network, it becomes possible to perform classification processing and segmentation processing with higher accuracy and speed.

Therefore, the confirmation unit 304 can confirm whether each hierarchical process includes a specific process whose processing content changes depending on a change in the input image size. An example of such specific processing is padding processing. In padding processing, special processing is performed only on the peripheral portions of the image. Therefore, the smaller the image, the more times the process using the values added by the padding process will be performed. That is, when padding processing is performed, the content of processing by the second processing unit 102 changes depending on changes in the input image size.

The confirmation unit 304 can also modify the configuration of the fully convolutional network so that the fully convolutional network does not perform padding processing or hierarchical processing in which the processing contents change due to changes in the size of array data. For example, if the network structure 12 indicates that the hierarchical processing includes padding processing, the confirmation unit 304 can modify the network structure 12 so as not to perform the padding processing. In this way, the confirmation unit 304 can automatically modify the network structure 12 and output the modified network structure 12 to the second processing unit 102.

Alternatively, the confirmation unit 304 can cause the second processing unit 102 to process the test data. In this way, the confirmation unit 304 can confirm whether the fully convolutional network used by the second processing unit 102 includes padding processing or hierarchical processing in which the processing contents vary depending on variations in the size of array data. Such an example will be explained with reference to FIGS. 8(A) to 8(C).

FIG. 8A shows an example in which the processing performed by the second processing unit 102a does not include hierarchical processing in which the processing contents change depending on changes in the input image size. In the example of FIG. 8A, test data 30201 is input to the second processing unit 102a that performs processing according to the network structure 12. The test data 30201 is a sufficiently large image in which only one pixel has a different pixel value and the other pixels have the same pixel value. If hierarchical processing in which the processing content changes due to changes in the input image size is not performed, the estimated values of the object likelihoods match in areas where the pixel values in the test data 30201 are the same. Therefore, the values are the same in most areas of the object likelihood map 30207 output for the test data 30201. On the other hand, near pixels that have different values in the test data 30201, calculations are performed using features extracted from these pixels, so the object likelihood value differs from other values as shown in FIG. 8(A). It is different from the area of

On the other hand, FIG. 8(B) shows an example in which the processing performed by the second processing unit 102b includes hierarchical processing in which the processing contents change depending on changes in the input image size. In the example of FIG. 8(B) as well, test data 30201 similar to that of FIG. 8(A) is input to the second processing unit 102b that performs processing according to the network structure 12. Then, the second processing unit 102b outputs an object likelihood map 30217 for the test data 30201. For example, if the processing performed by the second processing unit 102b includes padding processing, the values in the peripheral portion of the object likelihood map are different from the values in the inner portion, regardless of the content of the test data. FIG. 8C shows the object likelihood map 30217 in detail. Object likelihood map 30217 includes a portion 30218 affected by pixels having different pixel values included in test data 30201. The object likelihood map 30217 also includes a portion 30219 that is outside the portion 30218 and is not affected by pixels having different pixel values included in the test data 30201. Furthermore, the object likelihood map 30217 has a portion outside the portion 30219 that has a different value from the portion 30219 due to padding processing or the like.

In this way, the confirmation unit 304 checks whether the object likelihood map output by the second processing unit 102 to which the test data has been input has an area in the periphery of the map that has a value different from that in the interior. Can be judged. Then, the confirmation unit 304 determines that if such an area does not exist in the peripheral portion of the map, the processing performed by the second processing unit 102 does not include hierarchical processing in which the processing content changes when the input image size changes. It can be determined that In this case, the second processing unit 102 can determine that the network structure 12 exhibits a structure capable of performing highly accurate and high-speed classification processing and segmentation processing.

As another method, the confirmation unit 304 may detect a portion 30218 affected by pixels having different pixel values in the object likelihood map 30217 output by the second processing unit 102 input with test data. can. Next, the confirmation unit 304 can determine whether a portion of the object likelihood map 30217 excluding a portion 30218 affected by pixels having different pixel values has a uniform value. . For example, if the ratio of values within a predetermined range from the mode in this part exceeds a predetermined ratio, the confirmation unit 304 can determine that this part has a uniform value. The confirmation unit 304 determines that when this portion has a uniform value, the processing performed by the second processing unit 102 does not include hierarchical processing in which the processing content changes when the input image size changes. can be determined. The confirmation unit 304 may perform such processing on each of test data of different sizes.

By using the test data in this way, it is determined whether the processing performed by the second processing unit 102 includes hierarchical processing other than padding processing in which the processing contents change when the input image size changes. be able to. The confirmation unit 304 may further output the confirmation result of the network structure 12 via the display device 13 or the like. In this case, the user can know the confirmation result by the confirmation unit 304.

According to such a method, before learning the second processing unit 102, it is checked whether the second processing unit 102 has a structure that can perform classification processing and segmentation processing with high accuracy and high speed. be able to. Therefore, it is possible to generate a neural network that can more reliably perform classification processing and segmentation processing with high accuracy and high speed.

(Additional learning of the first processing unit)
Teacher data used for additional learning of the first processing unit 101 may be created by using the object likelihood map output by the second processing unit 102 after learning.

As described above, when the image 21 is input to the second processing unit 102 after learning, the second processing unit 102 outputs the object likelihood map corresponding to the image 21. Here, the learning unit 103 can acquire the object likelihood map output from the second processing unit 102. The learning unit 103 can then create teacher data used for learning by the first processing unit 101 from the acquired object likelihood map and image 21. Further, the learning unit 103 can perform additional learning of the first processing unit 101 using this teacher data.

Below, with reference to FIG. 9, the process performed by the learning unit 103 for additional learning of the first processing unit 101 will be described. Such processing can be performed after S1003. First, the second processing unit 102 inputs the array data for additional learning to the fully convolutional network after learning by the learning unit 103. In this way, the second processing unit 102 generates a likelihood map that indicates the likelihood that each piece of partial array data included in the array data for additional learning is included in a specific class. For example, an image 40001 is input to the second processing unit 102. This image 40001 corresponds to the image 21 in FIG. 1 and has a larger size than the image input to the first processing unit 101. Then, the second processing unit 102 outputs an object likelihood map 40007 corresponding to the image 40001. Likelihood maps 40008 and 40009 are likelihood maps for objects in specific categories extracted from object likelihood map 40007. Specifically, likelihood map 40008 is a likelihood map for category A objects. Further, the likelihood map 40009 is a likelihood map for objects of category B.

Then, the learning unit 103 extracts partial sequence data for additional learning from the sequence data for additional learning based on the likelihood map. Further, the learning unit 103 performs additional learning of the first processing unit 101 using the extracted partial sequence data. The process performed by the learning unit 103 is shown as block 40410. The learning unit 103 uses the image 40001 and the object likelihood map 40007 to generate teacher data that can be used for learning by the first processing unit 101. The learning unit 103 detects a region having a high likelihood in the object likelihood map. For example, the learning unit 103 can identify a region having a high likelihood by binarizing the object likelihood map 40007 using a predetermined threshold (for example, 0.3). Then, when the area of this region is larger than the threshold value, the learning unit 103 can cut out a portion corresponding to the identified region from the image 40001. The data thus cut out is used as training data.

In the example of FIG. 9, the learning unit 103 detects a region with a high likelihood in the likelihood map 40008, and cuts out the corresponding region in the image 40001, which is a broken line region. A combination of the image thus cut out and the category information (category A in this example) corresponding to the likelihood map 40008 is used as the teacher data 40413. Similarly, the learning unit 103 refers to the likelihood map 40009 and cuts out the broken line area from the image 40001. A combination of the image thus cut out and the category information (category B in this example) corresponding to the likelihood map 40009 is used as the teacher data 40414.

Then, the learning unit 103 performs learning of the first processing unit 101 using the obtained teacher data. The specific learning method of the first processing unit 101 is not particularly limited, and for example, the above-mentioned method can be used.

According to the above method, it is possible to generate an object likelihood map for any image, and further to generate training data based on this object likelihood map. By performing additional learning of the first processing unit 101 using the teacher data generated in this way, the first processing unit 101 can estimate the object likelihood with higher accuracy. Furthermore, by using the first processing section 101 that has been additionally trained in this way, the second processing section 102 is trained using the method already described, thereby further improving the performance of the neural network that performs segmentation processing at high speed. Can be done.

(Other examples)
Although examples of embodiments of the present invention have been described so far, the present invention can be realized as, for example, a system, an apparatus, a method, a program, a recording medium, or the like. For example, the present invention can be applied to a system composed of a plurality of devices (eg, a host computer, an interface device, an imaging device, a web application, etc.). On the other hand, the present invention may be applied to a device consisting of one device.

The information processing apparatus according to each of the embodiments described above can be realized using a computer. For example, the functions of each processing unit included in each information processing device shown in FIG. 1 and the like can be realized by a computer. Examples of the computer include a general-purpose personal computer and a server. However, at least some of the processing units may be realized by dedicated hardware. Further, each image processing device may be configured by a plurality of information processing devices connected via a network, for example. For example, the functions of each image processing device may be provided as a cloud service.

FIG. 10 is a diagram illustrating an example of a hardware configuration of an information processing apparatus implemented using a computer, according to an embodiment. In FIG. 10, a processor 1010 is, for example, a CPU, and controls the operation of the entire computer. The memory 1020 is, for example, a RAM, and temporarily stores programs, data, and the like. The computer-readable storage medium 1030 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. for a long period of time. In this embodiment, programs that are stored in the storage medium 1030 and implement the functions of each part are read into the memory 1020. The functions of each part are realized by the processor 1010 operating according to the program on the memory 1020.

In FIG. 10, an input interface 1040 is an interface for acquiring information from an external device. Further, the output interface 1050 is an interface for outputting information to an external device. A bus 1060 connects the above-mentioned units and enables data exchange.

The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure of this specification includes the following information processing apparatus, method for producing a fully convolutional network with learned parameters, and a program.

(Item 1)
a first processing means for calculating the likelihood that the array data is included in a specific class;
a second processing means that performs processing different from the first processing means, the second processing means calculating the likelihood that the array data is included in a specific class using a fully convolutional network;
Learning means for performing learning processing for the fully convolutional network using information obtained in the process in which the first processing means calculates the likelihood for the learning array data as teacher data;
An information processing device comprising:

(Item 2)
Array data of a predetermined size is input to the first processing means,
2. The information processing apparatus according to item 1, wherein array data of the predetermined size or more is input to the second processing means.

(Item 3)
The fully convolutional network is characterized in that when the first array data of the predetermined size is input, it outputs one vector indicating the likelihood that the first array data is included in the specific class. , the information processing device according to item 2.

(Item 4)
When the second array data larger than the predetermined size is input, the fully convolutional network outputs a likelihood map indicating the likelihood that each of the plurality of partial array data is included in the specific class; 4. The information processing device according to item 2 or 3, wherein each of the plurality of partial array data is a part of the second array data.

(Item 5)
The learning means calculates the difference between the likelihood calculated by the first processing means for the third array data for learning and the likelihood calculated by the second processing means for the third array data. The information processing device according to any one of items 1 to 4, characterized in that the learning process of the fully convolutional network is performed so that the total convolutional network is reduced.

(Item 6)
The first processing means calculates the likelihood that each of the plurality of array data is included in a specific class,
Each of the plurality of array data is a part of third array data for learning that is larger than the predetermined size,
The learning means is configured to reduce the difference between a likelihood map including the likelihood calculated by the first processing means and a likelihood map calculated by the second processing means for the third array data. The information processing device according to any one of items 2 to 4, characterized in that the learning process of the fully convolutional network is performed.

(Item 7)
the third array data is an image;
The plurality of array data are sampled images smaller than the image from different positions of the image,
The likelihood map including the likelihood calculated by the first processing means has a structure in which the likelihood calculated by the first processing means for the sampling image is placed at the sampling position of the sampling image. The information processing device according to item 6, characterized by:

(Item 8)
The information processing device according to item 4, wherein the second processing means performs segmentation of the array data based on the likelihood map.

(Item 9)
The array data is an image, the partial array data is a partial image included in the image, and the second processing means performs segmentation of the image based on the likelihood that the partial image is included in a specific class. The information processing device according to item 8, characterized in that the information processing device performs the following.

(Item 10)
The first processing means calculates the likelihood that the array data is included in the class for each of the plurality of classes;
9. The information processing apparatus according to any one of items 1 to 9, wherein the second processing means calculates, for each of a plurality of classes, the likelihood that the array data is included in the class.

(Item 11)
Any one of items 1 to 10, wherein the first processing means calculates the likelihood using Self-Attention processing, Transformer, or repetition of processing using all input features. The information processing device described in item 1.

(Item 12)
The learning means uses the features of the learning array data obtained in the process in which the first processing means calculates the likelihood for the learning array data, and the second processing means According to any one of items 1 to 4, the learning process of the fully convolutional network is performed so that the difference between the characteristics of the learning array data calculated using the convolutional network is reduced. information processing equipment.

(Item 13)
The second processing means is based on the correlation between the characteristics of the array data of a specific class calculated using the fully convolutional network and the characteristics of the array data to be processed calculated using the fully convolutional network. , the information processing apparatus according to item 12, wherein the information processing apparatus calculates the likelihood that the array data to be processed is included in the specific class.

(Item 14)
According to any one of items 1 to 13, the fully convolutional network is configured so as not to perform padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. The information processing device described.

(Item 15)
Any one of items 1 to 13, further comprising confirmation means for confirming whether the fully convolutional network includes padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. The information processing device described in item 1.

(Item 16)
The confirmation means causes the second processing means to perform processing on the test data, thereby determining whether the fully convolutional network includes padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. The information processing device according to item 15, characterized in that the information processing device confirms whether or not the information is present.

(Item 17)
The confirmation means is characterized in that the configuration of the fully convolutional network is modified so that the fully convolutional network does not perform padding processing or hierarchical processing in which the processing contents vary due to variations in the size of the array data. , the information processing device according to item 15 or 16.

(Item 18)
The second processing means specifies each of the partial sequence data included in the sequence data for additional learning by inputting the sequence data for additional learning into the fully convolutional network after learning by the learning means. Generate a likelihood map showing the likelihood of being included in the class of
The learning means extracts partial sequence data for additional learning from the sequence data for additional learning based on the likelihood map, and performs additional learning of the first processing means using the extracted partial sequence data. The information processing device according to any one of items 1 to 17, characterized in that the information processing device performs the following.

(Item 19)
A method for an information processing device to produce a fully convolutional network with learned parameters, the method comprising:
The information processing device includes:
a first processing means for calculating the likelihood that the array data is included in a specific class;
a second processing means that performs processing different from the first processing means, the second processing means calculating the likelihood that the array data is included in a specific class using a fully convolutional network; Prepare,
The method includes:
using the first processing means to calculate the likelihood of the learning array data;
performing a learning process of the fully convolutional network using information obtained in the process of calculating the likelihood by the first processing means as training data;
A method, comprising:

(Item 20)
A program for causing a computer to function as the information processing device according to any one of items 1 to 18.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

101: First processing unit, 102: Second processing unit, 103: Learning unit, 304: Confirmation unit

Claims (20)

a first processing means for calculating the likelihood that the array data is included in a specific class;
a second processing means that performs processing different from the first processing means, the second processing means calculating the likelihood that the array data is included in a specific class using a fully convolutional network;
Learning means for performing learning processing for the fully convolutional network using information obtained in the process in which the first processing means calculates the likelihood for the learning array data as teacher data;
An information processing device comprising: Array data of a predetermined size is input to the first processing means,
2. The information processing apparatus according to claim 1, wherein array data of the predetermined size or more is input to the second processing means. The fully convolutional network is characterized in that when the first array data of the predetermined size is input, it outputs one vector indicating the likelihood that the first array data is included in the specific class. , The information processing device according to claim 2. When the second array data larger than the predetermined size is input, the fully convolutional network outputs a likelihood map indicating the likelihood that each of the plurality of partial array data is included in the specific class; 4. The information processing apparatus according to claim 3, wherein each of the plurality of partial array data is a part of the second array data. The learning means calculates the difference between the likelihood calculated by the first processing means for the third array data for learning and the likelihood calculated by the second processing means for the third array data. 5. The information processing apparatus according to claim 4, wherein learning processing of the fully convolutional network is performed so that the total number of convolutional networks decreases. The first processing means calculates the likelihood that each of the plurality of array data is included in a specific class,
Each of the plurality of array data is a part of third array data for learning that is larger than the predetermined size,
The learning means is configured to reduce the difference between a likelihood map including the likelihood calculated by the first processing means and a likelihood map calculated by the second processing means for the third array data. 5. The information processing apparatus according to claim 4, wherein learning processing of the fully convolutional network is performed. the third array data is an image;
The plurality of array data are sampled images smaller than the image from different positions of the image,
The likelihood map including the likelihood calculated by the first processing means has a structure in which the likelihood calculated by the first processing means for the sampling image is placed at the sampling position of the sampling image. The information processing device according to claim 6, characterized in that: 5. The information processing apparatus according to claim 4, wherein the second processing means performs segmentation of the array data based on the likelihood map. The array data is an image, the partial array data is a partial image included in the image, and the second processing means performs segmentation of the image based on the likelihood that the partial image is included in a specific class. The information processing device according to claim 8, wherein the information processing device performs the following. The first processing means calculates the likelihood that the array data is included in the class for each of the plurality of classes;
5. The information processing apparatus according to claim 4, wherein the second processing means calculates, for each of a plurality of classes, the likelihood that the array data is included in the class. The information according to claim 4, wherein the first processing means calculates the likelihood using Self-Attention processing, Transformer, or repetition of processing using all input features. Processing equipment. The learning means uses the features of the learning array data obtained in the process in which the first processing means calculates the likelihood for the learning array data, and the second processing means 2. The information processing apparatus according to claim 1, wherein the learning processing of the complete convolutional network is performed so that a difference between the characteristics of the learning array data calculated using the convolutional network and the characteristics of the learning array data is reduced. The second processing means is based on the correlation between the characteristics of the array data of a specific class calculated using the fully convolutional network and the characteristics of the array data to be processed calculated using the fully convolutional network. 13. The information processing apparatus according to claim 12, wherein the information processing apparatus calculates the likelihood that the array data to be processed is included in the specific class. 2. The information processing apparatus according to claim 1, wherein the fully convolutional network is configured not to perform padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. The information according to claim 1, further comprising confirmation means for confirming whether the fully convolutional network includes padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. Processing equipment. The confirmation means causes the second processing means to perform processing on the test data, thereby determining whether the fully convolutional network includes padding processing or hierarchical processing in which processing contents vary depending on variations in the size of the array data. 16. The information processing apparatus according to claim 15, wherein the information processing apparatus checks whether the The confirmation means is characterized in that the configuration of the fully convolutional network is modified so that the fully convolutional network does not perform padding processing or hierarchical processing in which the processing contents vary due to variations in the size of the array data. , the information processing device according to claim 15. The second processing means specifies each of the partial sequence data included in the sequence data for additional learning by inputting the sequence data for additional learning into the fully convolutional network after learning by the learning means. Generate a likelihood map showing the likelihood of being included in the class of
The learning means extracts partial sequence data for additional learning from the sequence data for additional learning based on the likelihood map, and performs additional learning of the first processing means using the extracted partial sequence data. The information processing apparatus according to claim 1, wherein the information processing apparatus performs the following. A method for an information processing device to produce a fully convolutional network with learned parameters, the method comprising:
The information processing device includes:
a first processing means for calculating the likelihood that the array data is included in a specific class;
a second processing means that performs processing different from the first processing means, the second processing means calculating the likelihood that the array data is included in a specific class using a fully convolutional network; Prepare,
The method includes:
using the first processing means to calculate the likelihood of the learning array data;
performing a learning process of the fully convolutional network using information obtained in the process of calculating the likelihood by the first processing means as training data;
A method, comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 18.

Images