WO2025041210A1 - Inference device, inference method, and inference program - Google Patents

Abstract

In an inference device (10), an acquisition unit (15a) acquires, for a plurality of tasks for a prescribed medium, a feature quantity of the medium and a query parameter representing a feature quantity of each task. A prediction generation unit (15d) predicts a task feature quantity of each task for the medium from the query parameters acquired by the acquisition unit (15a), using a first relationship between the query parameters representing the feature quantities of tasks that cross over different tasks and a second relationship between the feature quantity of the medium and the query parameter representing the feature quantity of each task.

Description

Inference device, inference method, and inference program

The present invention relates to an inference device, an inference method, and an inference program.

Conventionally, for media such as images, audio, and natural language, inference results for multiple tasks, such as object detection, region segmentation, and pose estimation for images, may be required. As an inference method that can handle multiple tasks, for example, a method called DETR (Detection TRansformer) is known that solves object detection using a structure called a Transformer and parameters that serve as the basis for prediction called Object Query (see Non-Patent Document 1). The method that uses a Transformer and Object Query is known to be effective for various tasks such as pose estimation and region segmentation in addition to object detection.

X. Zhu, “Deformable DETR: Deformable Transformers for End-to-End Object Detection”, 2021

However, with conventional technology, it is difficult to solve multiple tasks simultaneously. For example, although a method using a Transformer and Object Query is effective for a variety of tasks, it is necessary to create a task-specific model specialized for each task and perform inference. In other words, there is an issue that a cost corresponding to the number of tasks is incurred because a model is created for each task and inference is performed.

The present invention was made in consideration of the above, and aims to make it possible to efficiently execute inference on multiple tasks.

In order to solve the above-mentioned problems and achieve the object, the inference device according to the present invention is characterized by having an acquisition unit that acquires, for a plurality of tasks for a given medium, a feature of the medium and parameters representing the feature of each task, and a prediction unit that predicts the task feature of each task in the medium from the parameters acquired by the acquisition unit, using a first relationship between the parameters representing the feature of each task across different tasks, and a second relationship between the feature of the media and the parameters representing the feature of each task.

The present invention makes it possible to efficiently execute inference on multiple tasks.

FIG. 1 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 2 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 3 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 4 is a schematic diagram illustrating a schematic configuration of the inference device of this embodiment. FIG. 5 is a diagram for explaining the processing of the inference device. FIG. 6 is a diagram for explaining the processing of the inference device. FIG. 7 is a flowchart showing the procedure of the inference process. FIG. 8 is a flowchart showing the procedure of the inference process. FIG. 9 is a flowchart showing the procedure of the inference process. FIG. 10 is a flowchart showing the procedure of the inference process. FIG. 11 is a flowchart showing the procedure of the inference process. FIG. 12 is a diagram for explaining the second embodiment. FIG. 13 is a flowchart showing an inference processing procedure according to the second embodiment. FIG. 14 is a diagram for explaining the embodiment. FIG. 15 is a diagram illustrating an example of a computer that executes an inference program.

Below, one embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the drawings, the same parts are denoted by the same reference numerals.

[Outline of the inference device]
FIG. 1 is a diagram for explaining an outline of an inference device of this embodiment. In this embodiment, the media is, for example, any one of an image, a voice, and a natural language. FIG. 1 illustrates a method of using a Transformer and an Object Query to predict a single task for media. In this method, for example, a model for an image is composed of a CNN (Convolutional Neural Network) part called a backbone created for image classification, and a part called a task head that predicts each task. Then, the image feature amount, which is the output of the backbone, and the Object Query, which is a parameter related to the task, are input to the Transformer to obtain the output of each task.

Here, the Transformer is composed of a Transformer Encoder and a Transformer Decoder, as shown in Figure 1. The Transformer Encoder is designed to be common to each model of multiple tasks, and refines the backbone image features through nonlinear transformation using Attention, a mechanism for learning noteworthy parts of the input by extracting features based on the similarity between the input features.

The Transformer Decoder also receives the image features and Object Query output from the Transformer Encoder, and repeats the processing of multiple layers (L layers). This allows each Object Query to grasp the state of its own prediction and recognize its role within the image, gradually predicting the desired task.

This Transformer Decoder is mainly composed of Self-Attention and Cross-Attention. Self-Attention extracts the relationship between predictions corresponding to each of multiple Object Queries. Cross-Attention extracts the relationship between image features and predictions corresponding to each of multiple Object Queries. Here, Attention is a mechanism that performs nonlinear transformation by extracting features based on the similarity between input features.

Self-Attention processing is often almost the same for each task model, while Cross-Attention processing often differs depending on the task and method. Therefore, while methods using Transformers are generally effective for a variety of tasks, it is necessary to create a model specialized for one task.

The inference device according to this embodiment provides a specific improvement over the conventional method using Transformer and Object Query, which required creating task-specific models for each task and performing inference, resulting in costs according to the number of tasks, and represents an advancement in the technical field related to multitask AI.

Here, Fig. 2 and Fig. 3 show an example of the processing in the inference device of this embodiment. Specifically, as shown in Fig. 2, in the inference device, the Transformer Decoder and Object Query are expanded to multitask in order to solve multiple tasks simultaneously.

In other words, as shown in FIG. 3(b), the inference device prepares the same number of types of Object Queries as the number of tasks for each task, inputs them together into the Self-Attention of the Transformer Decoder, and performs a process to extract the relationships between the predictions of the Object Queries. This makes it possible to extract the relationships between the predictions of multiple tasks.

In addition, Cross-Attention (CA) branches as many times as the number of tasks, and performs processing to extract the relationship between image features and the prediction of the Object Query for each type of Object Query corresponding to each task. This makes it possible to incorporate Cross-Attention technology that has developed independently for each task. In this way, the inference device uses a multi-task Object Query to perform Self-Attention processing across tasks in the Transformer Decoder, and Cross-Attention processing unique to the branched task. This makes it possible to perform multi-task inference more efficiently than the conventional processing for a single task as illustrated in Figure 3(a).

[Configuration of inference device]
Fig. 4 is a schematic diagram illustrating the general configuration of the inference device of this embodiment. Fig. 5 and Fig. 6 are diagrams for explaining the processing of the inference device. First, as illustrated in Fig. 4, the inference device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

The input unit 11 is realized using input devices such as a keyboard and a mouse, and inputs various instruction information such as starting processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of the inference processing described below.

The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device that manages various types of information.

The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the inference device 10 and data used during execution of the processing program, or stores the data temporarily each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

In this embodiment, the storage unit 14 stores an image 14a, image feature generation parameters 14b, image feature refinement parameters 14c, prediction generation parameters 14d, query parameters 14e, etc., which are used in the inference process described below. This information is obtained from a management device that manages various information via the input unit 11 or the communication control unit 13 before the inference process described below or during the inference process, and is stored in the storage unit 14.

Image 14a stores the image to be processed and input to the inference process described below.

In image feature generation parameters 14b, image feature generation parameters used in the inference process described below are stored. The image feature generation parameters are parameters of a neural network model, and are obtained by supervised learning of a general neural network using learning data for the task.

In image feature refinement parameters 14c, image feature refinement parameters used in the inference process described below are stored. The image feature refinement parameters are parameters of a neural network model, and like the image feature generation parameters, are obtained by supervised learning of a general neural network using the learning data for the task.

Prediction generation parameters 14d store prediction generation parameters used in the inference process described below. The prediction generation parameters are parameters of a neural network model, and like the image feature generation parameters, are obtained by supervised learning of a general neural network using learning data for the task.

Query parameters 14e store query parameters used in the inference process described below. The query parameters are parameters that represent the feature quantities of a task, such as an Object Query, and the same number of query parameters as the number of tasks N are stored corresponding to each of the multiple tasks. In the example shown in FIG. 5, N types of query parameters are stored: task 1 query parameters, task 2 query parameters, ..., task N query parameters.

The query parameters for each task are randomly initialized and obtained through typical supervised learning of neural networks. The query parameters are vectors of the same dimensions as the vectors processed in the inference process described below.

The control unit 15 is realized using a CPU (Central Processing Unit), NP (Network Processor), FPGA (Field Programmable Gate Array), etc., and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, an image feature generation unit 15b, an image feature refinement unit 15c, a prediction generation unit 15d, an inter-prediction relationship extraction unit 15e, and an image-prediction relationship extraction unit 15f, as exemplified in FIG. 4, to execute inference processing. Note that each of these functional units, or a portion of them, may be implemented in different hardware. The control unit 15 may also include other functional units.

The acquisition unit 15a acquires, via the input unit 11 or the communication control unit 13, from the memory unit 14, the image to be processed, parameters for generating image features, parameters for refining image features, parameters for predictive generation, parameters for queries, etc., which are used in the inference process described below.

In addition, instead of the acquisition unit 15a, each of the above information may be acquired from the storage unit 14 when a functional unit described below performs processing.

As shown in FIG. 5, the image feature generating unit 15b uses the image feature generating parameters acquired from the storage unit 14 and the image to be processed to generate image features for the image to be processed based on the image feature generating parameters. The image feature generating unit 15b corresponds to, for example, a Transformer Encoder.

Here, the image to be processed is acquired as a three-dimensional vector of length x width x RGB. Furthermore, when processing multiple images, the image feature generation unit 15b only needs to acquire the image feature generation parameters the first time.

5, the image feature refinement unit 15c further refines the image feature generated by the image feature generation unit 15b based on the image feature refinement parameters acquired from the storage unit 14. Specifically, the image feature refinement unit 15c corresponds to a Transformer Encoder, and refines the feature of the media by performing a nonlinear transformation based on Attention, which is a mechanism for further extracting features using the similarity between the features of the media.

Here, when the image feature refinement unit 15c processes the image features of multiple images, it is sufficient to obtain the image feature refinement parameters only the first time.

The aforementioned acquisition unit 15a may acquire refined image features and transfer them to the prediction generation unit 15d described below. In this case, the acquisition unit 15a acquires media features and task parameters representing the features of each task for multiple tasks on a specific media, such as an image.

The prediction generation unit 15d predicts the task features of each task in the media from the query parameters acquired by the acquisition unit 15a, using a first relationship between the query parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the query parameters representing the features of each task.

Specifically, as illustrated in FIG. 5, the prediction generation unit 15d predicts the desired task using the prediction generation parameters and query parameters acquired from the storage unit 14, and the image features refined by the image feature refinement unit 15c.

Here, when the prediction generation unit 15d processes image features of multiple images, it is sufficient to obtain the prediction generation parameters and query parameters only the first time.

Furthermore, as shown in FIG. 5, there are query parameters corresponding to each of a plurality of tasks. For example, when two tasks, object detection and pose estimation, are the targets of inference processing, the query parameters for object detection and the query parameters for pose estimation are input to the prediction generation unit 15d, and prediction results for the two tasks, object detection and pose estimation, are output from the image features refined by the image feature refinement unit 15c.

The prediction generation unit 15d predicts task features by repeatedly converting parameters using the similarity between the query parameters as the first relationship, and converting parameters using the similarity between the media features and the query parameters as the second relationship.

In other words, the prediction generation unit 15d performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts features using the similarity between the query parameters of each task, and performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts features using the similarity between the features of the media and the query parameters of each task. The prediction generation unit 15d repeats this process to predict task features of each task in the media.

Specifically, the prediction generation unit 15d is a decoder of a transformer, which is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.

In other words, the prediction generation unit 15d is a decoder of a transformer, in which Self-Attention extracts the relationship between the query parameters of each task across different tasks, and Cross-Attention extracts the relationship between the media features and the query parameters of each task for each task, thereby predicting the task features of each task in the media.

Here, FIG. 6 is a diagram for explaining the processing of the prediction generation unit. As illustrated in FIG. 6, the prediction generation unit 15d includes an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f. The inter-prediction relationship extraction unit 15e corresponds to, for example, a self-attention of a transformer decoder, and performs the same processing across tasks regardless of the type and number of tasks. On the other hand, the image-inter-prediction relationship extraction unit 15f corresponds to, for example, a cross-attention of a transformer decoder, and performs different processing depending on the type of task, so that the same number of image-inter-prediction relationship extraction units 15f as the number of tasks perform processing in parallel.

The prediction generation unit 15d has a predetermined number L of layers, each of which is a pair of an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f. The query parameters change gradually with each layer, and finally a prediction is output, which is the inference result for each task.

Specifically, first, in the first layer, inter-prediction relationship extraction unit 15e extracts relationships between each query parameter based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (hereinafter, the updated query parameters may be referred to as "query parameters (prediction)"). Next, image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (prediction) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.

Next, in the second layer, the prediction relationship extraction unit 15e extracts relationships between each query parameter (prediction) based on the updated prediction generation parameters, and updates each query parameter based on the extracted relationships. Next, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (prediction) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.

In this way, the inter-prediction relationship extraction unit 15e and the image-prediction relationship extraction unit 15f repeat the same process up to the L layer, thereby outputting a prediction that is the inference result for each task. In this way, the inference device 10 can efficiently execute inference for multiple tasks for media. Furthermore, learning of various parameters can also be performed collectively for multiple tasks, making it possible to execute the learning efficiently.

[Inference processing]
Next, the inference process performed by the inference device 10 according to this embodiment will be described with reference to Fig. 7 to Fig. 11. Fig. 7 to Fig. 11 are flowcharts showing the procedure of the inference process. First, in Fig. 7, the flowchart starts at the timing when the user performs an operation input to instruct the start of the process.

First, the acquisition unit 15a acquires from the storage unit 14 the image to be processed, parameters for generating image features, parameters for refining image features, parameters for predictive generation, parameters for queries, etc., to be used in the inference process.

Then, the image feature generating unit 15b uses the image feature generating parameters and the image to be processed to generate image features for the image to be processed based on the image feature generating parameters (step S1).

The image feature refinement unit 15c further refines the image features generated by the image feature generation unit 15b based on the image feature refinement parameters (step S2).

Next, the prediction generation unit 15d uses the prediction generation parameters, the refined image features, and the query parameters for each task to generate a prediction of the features of each task in the image (step S3), and outputs the generated prediction (step S4). This completes the series of inference processes.

Next, FIG. 8 illustrates the processing procedure of the image feature generation unit in step S1 shown in FIG. 7. First, the image feature generation unit 15b acquires image feature generation parameters and an input image to be processed from the acquisition unit 15a or the storage unit 14 (steps S101 to S102).

Next, the image feature generating unit 15b generates image features for the input image to be processed based on the image feature generation parameters (step S103). The image feature generating unit 15b also outputs the generated image features to the downstream image feature refining unit 15c (step S104), and the series of processes ends.

Next, FIG. 9 illustrates the processing procedure of the image feature generation unit in step S2 shown in FIG. 7. First, the image feature refinement unit 15c acquires parameters for image feature refinement from the acquisition unit 15a or the storage unit 14 (step S201), and acquires the image feature amount generated from the image feature generation unit 15b (step S202).

Next, the image feature refinement unit 15c refines the image feature based on the image feature refinement parameters (step S203). The image feature generation unit 15b outputs the refined image feature to the downstream prediction generation unit 15d (step S204), and the series of processes ends.

Next, FIG. 10 illustrates the processing procedure of the prediction generation unit in steps S3 to S4 shown in FIG. 7. First, the prediction generation unit 15d acquires prediction generation parameters and query parameters from the acquisition unit 15a or the storage unit 14 (steps S301 to S302), and acquires refined image features from the image feature refinement unit 15c (step S303).

Then, the prediction generation unit 15d generates a prediction of the task features based on the prediction generation parameters for the image features and the query parameters (step S304). The prediction generation unit 15d then outputs the generated task prediction (step S305), and ends the series of processes.

FIG. 11 also illustrates a detailed processing procedure for the process of step S304 shown in FIG. 10. After obtaining refined image features from the image feature refinement unit 15c in the process of step S303, the prediction generation unit 15d performs the first layer processing shown in FIG. 6. Specifically, the prediction relationship extraction unit 15e extracts relationships between query parameters for each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (step S401). Next, the image-prediction relationship extraction unit 15f extracts relationships between the image features and the updated query parameters for each task, and further updates the query parameters for each task based on the extracted relationships (step S402).

Next, the prediction generation unit 15d performs second-layer processing. That is, the prediction relationship extraction unit 15e extracts relationships between query parameters for the updated query parameters of each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships. In addition, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters of each task, and further updates the query parameters of each task based on the extracted relationships.

In this way, the prediction generation unit 15d repeats the same process up to the Lth layer (step S403), and sets the updated query parameters in the Lth layer as a prediction for the generated task (step S404). After that, the prediction generation unit 15d outputs the prediction for the generated task (step S305), and ends the series of processes.

Note that the media in the above processing is not limited to images. As mentioned above, similar processing can also be applied to speech and natural language as media. In that case, general feature extraction can be performed when processing each media, and these can be used as input to the model.

Second Embodiment
12 is a diagram for explaining the processing of the inference device of the second embodiment. Note that, below, only the points that are different from the inference processing of the inference device 10 of the first embodiment described above will be explained, and a description of the points in common will be omitted.

The inference device 10 according to the second embodiment, like the inference device 10 according to the above embodiment, provides a specific improvement over the conventional method of using a Transformer and Object Query, which required creating task-specific models for each task and performing inference, resulting in costs according to the number of tasks, and represents an advancement in the technical field related to multitask AI.

Specifically, in the inference device 10 of the above embodiment, as shown in FIG. 6, in the prediction generation unit 15d, the inter-prediction relationship extraction unit 15e performs processing, and then the image-prediction relationship extraction unit 15f performs processing, but the processing order is not limited to this. In other words, as shown in FIG. 12, the inter-prediction relationship extraction unit 15f may perform processing, and then the inter-prediction relationship extraction unit 15e may perform processing.

In this case, as with the inference device 10 of the above embodiment, inference of multiple tasks for media can be efficiently executed. Furthermore, learning of various parameters can also be performed for multiple tasks collectively, so that it can be executed efficiently.

FIG. 13 is a flowchart showing the inference processing procedure of the second embodiment. As shown in FIG. 13, the processing of steps S411 to S413 differs from the flowchart in FIG. 11.

Specifically, after obtaining refined image features from image feature refinement unit 15c in the process of step S303, prediction generation unit 15d performs the first layer process shown in FIG. 12. Specifically, image-prediction relationship extraction unit 15f extracts relationships between image features and query parameters of each task, and updates each query parameter based on the extracted relationships (step S411). Next, prediction relationship extraction unit 15e extracts relationships between query parameters based on prediction generation parameters for the updated query parameters of each task, and updates each query parameter based on the extracted relationships (step S412).

Next, the prediction generation unit 15d performs second-layer processing. That is, the image-prediction relationship extraction unit 15f extracts the relationship between the image feature amount and the updated query parameters of each task, and updates the query parameters of each task based on the extracted relationship. In addition, the prediction relationship extraction unit 15e extracts the relationship between query parameters based on the prediction generation parameters for the updated query parameters of each task, and updates each query parameter based on the extracted relationship.

In this way, the prediction generation unit 15d repeats the same process up to the Lth layer (step S413), and sets the updated query parameters in the Lth layer as a prediction for the generated task (step S404). After that, the prediction generation unit 15d outputs the prediction for the generated task (step S305), and ends the series of processes.

[effect]
As described above, in the inference device 10 of this embodiment, the acquisition unit 15a acquires, for a plurality of tasks for a predetermined medium, the feature amounts of the media and the query parameters representing the feature amounts of each task. The prediction generation unit 15d predicts the task feature amounts of each task in the media from the query parameters acquired by the acquisition unit 15a, using a first relationship between the query parameters representing the feature amounts of each task across different tasks, and a second relationship between the feature amounts of the media and the query parameters representing the feature amounts of each task.

Specifically, the prediction generation unit 15d predicts task features by repeatedly converting parameters using the similarity between the query parameters as the first relationship, and converting parameters using the similarity between the media features and the query parameters as the second relationship.

For example, the prediction generation unit 15d is a decoder of a transformer, which is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.

This makes it possible to efficiently execute inference for multiple tasks. In addition, learning of various parameters for multiple tasks can also be performed simultaneously, making execution more efficient.

The media can be images, audio, or natural language. In this way, the inference device 10 can efficiently execute inference processing for multiple tasks in a variety of media.

[Example]
FIG. 14 is a diagram for explaining an example. In this example, the tasks were object detection and pose estimation, and the inference time and learning time of a model using a conventional Transformer were compared with the inference device 10 of the above embodiment. As a model using a conventional Transformer, Deformable DETR was used for object detection, and PETR (Position Embedding TRansformation) was used for pose estimation. In addition, R-50 was used as the backbone. In contrast, a comparison was made between the case where R-50 was used as the backbone of the model of the above embodiment, and the case where Swin-T was used.

As shown in FIG. 14, when the backbone is the same R-50, it was confirmed that the inference device 10 of the above embodiment reduces both the learning time and inference time to 70% of the conventional time, and achieves an accuracy of 90% or more. Furthermore, when using a backbone Swin-T that is larger than R-50, it was confirmed that the inference device 10 achieves excellent accuracy with 80% of the conventional learning time and inference time.

[program]
A program in which the process executed by the inference device 10 according to the above embodiment is written in a language executable by a computer can also be created. As an embodiment, the inference device 10 can be implemented by installing an inference program that executes the above inference process as package software or online software on a desired computer. For example, the inference program can be executed by an information processing device, causing the information processing device to function as the inference device 10. The information processing device referred to here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDA (Personal Digital Assistant). The functions of the inference device 10 may be implemented on a cloud server.

FIG. 15 is a diagram showing an example of a computer that executes an inference program. The computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

Here, the hard disk drive 1031 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. Each piece of information described in the above embodiment is stored, for example, in the hard disk drive 1031 or memory 1010.

The inference program is stored in the hard disk drive 1031, for example, as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the inference device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

In addition, data used for information processing by the inference program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the procedures described above.

The program module 1093 and program data 1094 related to the inference program are not limited to being stored in the hard disk drive 1031, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the inference program may be stored in another computer connected via a network, such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

The following notes are further provided with respect to the above embodiment.

(Additional Note 1)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
acquiring, for a plurality of tasks for a predetermined medium, a feature of the medium and a parameter representing the feature of each task;
A task feature of each task in the medium is predicted from the acquired parameters using a first relationship between parameters representing the feature of each task across different tasks and a second relationship between the feature of the medium and the parameters representing the feature of each task.
Inference device.

(Additional Note 2)
A non-transitory storage medium storing a program executable by a computer to perform inference processing by the inference device described in appended claim 1.

The above describes an embodiment of the invention made by the inventor, but the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention according to this embodiment. In other words, other embodiments, examples, operational techniques, etc. made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

REFERENCE LIST 10 Inference device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Image 14b Image feature generation parameters 14c Image feature refinement parameters 14d Prediction generation parameters 14e Query parameters 15 Control unit 15a Acquisition unit 15b Image feature generation unit 15c Image feature refinement unit 15d Prediction generation unit (prediction unit)
15e Inter-prediction relationship extraction unit 15f Image-inter-prediction relationship extraction unit

Claims (6)

an acquisition unit that acquires, for a plurality of tasks for a predetermined medium, a feature amount of the medium and a parameter that represents the feature amount of each task;
a prediction unit that predicts a task feature of each task in the medium from the parameters acquired by the acquisition unit, using a first relationship between parameters that represent feature amounts of each task across different tasks and a second relationship between the feature amounts of the medium and parameters that represent feature amounts of each task;
An inference device comprising: The inference device according to claim 1, characterized in that the prediction unit predicts the task features by repeating the process of transforming the parameters using the similarity between the parameters as the first relationship, and transforming the parameters using the similarity between the media features and the parameters as the second relationship. The inference device according to claim 1, characterized in that the prediction unit is a decoder of a transformer that is a pre-trained neural network, and the first relationship is extracted by a self-attention mechanism, and the second relationship is extracted by a cross-attention mechanism. The inference device according to claim 1, characterized in that the media is one of an image, an audio, and a natural language. An inference method executed by an inference device, comprising:
an acquisition step of acquiring, for a plurality of tasks for a predetermined medium, a feature amount of the medium and a parameter representing a feature amount of each task;
a prediction step of predicting a task feature of each task in the medium from the parameters acquired in the acquisition step, using a first relationship between parameters representing feature amounts of each task across different tasks and a second relationship between the feature amounts of the medium and the parameters representing feature amounts of each task;
23. An inference method comprising: An inference program for causing a computer to function as an inference device according to any one of claims 1 to 4.

Images