WO2025041210A1 - Dispositif d'inférence, procédé d'inférence et programme d'inférence - Google Patents

Dispositif d'inférence, procédé d'inférence et programme d'inférence Download PDF

Info

Publication number
WO2025041210A1
WO2025041210A1 PCT/JP2023/029898 JP2023029898W WO2025041210A1 WO 2025041210 A1 WO2025041210 A1 WO 2025041210A1 JP 2023029898 W JP2023029898 W JP 2023029898W WO 2025041210 A1 WO2025041210 A1 WO 2025041210A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
parameters
inference
prediction
relationship
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2023/029898
Other languages
English (en)
Japanese (ja)
Inventor
大河 山根
聡志 鈴木
智大 田中
亮 増村
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NTT Inc
Original Assignee
Nippon Telegraph and Telephone Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nippon Telegraph and Telephone Corp filed Critical Nippon Telegraph and Telephone Corp
Priority to JP2025541172A priority Critical patent/JPWO2025041210A1/ja
Priority to PCT/JP2023/029898 priority patent/WO2025041210A1/fr
Publication of WO2025041210A1 publication Critical patent/WO2025041210A1/fr
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present invention relates to an inference device, an inference method, and an inference program.
  • inference results for multiple tasks such as object detection, region segmentation, and pose estimation for images
  • a method called DETR Detection TRansformer
  • Object Query a structure called a Transformer and parameters that serve as the basis for prediction called Object Query
  • DETR Detection TRansformer
  • the method that uses a Transformer and Object Query is known to be effective for various tasks such as pose estimation and region segmentation in addition to object detection.
  • the present invention was made in consideration of the above, and aims to make it possible to efficiently execute inference on multiple tasks.
  • the inference device is characterized by having an acquisition unit that acquires, for a plurality of tasks for a given medium, a feature of the medium and parameters representing the feature of each task, and a prediction unit that predicts the task feature of each task in the medium from the parameters acquired by the acquisition unit, using a first relationship between the parameters representing the feature of each task across different tasks, and a second relationship between the feature of the media and the parameters representing the feature of each task.
  • the present invention makes it possible to efficiently execute inference on multiple tasks.
  • FIG. 1 is a diagram for explaining an overview of the inference device of this embodiment.
  • FIG. 2 is a diagram for explaining an overview of the inference device of this embodiment.
  • FIG. 3 is a diagram for explaining an overview of the inference device of this embodiment.
  • FIG. 4 is a schematic diagram illustrating a schematic configuration of the inference device of this embodiment.
  • FIG. 5 is a diagram for explaining the processing of the inference device.
  • FIG. 6 is a diagram for explaining the processing of the inference device.
  • FIG. 7 is a flowchart showing the procedure of the inference process.
  • FIG. 8 is a flowchart showing the procedure of the inference process.
  • FIG. 9 is a flowchart showing the procedure of the inference process.
  • FIG. 10 is a flowchart showing the procedure of the inference process.
  • FIG. 10 is a flowchart showing the procedure of the inference process.
  • FIG. 11 is a flowchart showing the procedure of the inference process.
  • FIG. 12 is a diagram for explaining the second embodiment.
  • FIG. 13 is a flowchart showing an inference processing procedure according to the second embodiment.
  • FIG. 14 is a diagram for explaining the embodiment.
  • FIG. 15 is a diagram illustrating an example of a computer that executes an inference program.
  • FIG. 1 is a diagram for explaining an outline of an inference device of this embodiment.
  • the media is, for example, any one of an image, a voice, and a natural language.
  • FIG. 1 illustrates a method of using a Transformer and an Object Query to predict a single task for media.
  • a model for an image is composed of a CNN (Convolutional Neural Network) part called a backbone created for image classification, and a part called a task head that predicts each task. Then, the image feature amount, which is the output of the backbone, and the Object Query, which is a parameter related to the task, are input to the Transformer to obtain the output of each task.
  • CNN Convolutional Neural Network
  • the Transformer is composed of a Transformer Encoder and a Transformer Decoder, as shown in Figure 1.
  • the Transformer Encoder is designed to be common to each model of multiple tasks, and refines the backbone image features through nonlinear transformation using Attention, a mechanism for learning noteworthy parts of the input by extracting features based on the similarity between the input features.
  • the Transformer Decoder also receives the image features and Object Query output from the Transformer Encoder, and repeats the processing of multiple layers (L layers). This allows each Object Query to grasp the state of its own prediction and recognize its role within the image, gradually predicting the desired task.
  • This Transformer Decoder is mainly composed of Self-Attention and Cross-Attention.
  • Self-Attention extracts the relationship between predictions corresponding to each of multiple Object Queries.
  • Cross-Attention extracts the relationship between image features and predictions corresponding to each of multiple Object Queries.
  • Attention is a mechanism that performs nonlinear transformation by extracting features based on the similarity between input features.
  • Self-Attention processing is often almost the same for each task model, while Cross-Attention processing often differs depending on the task and method. Therefore, while methods using Transformers are generally effective for a variety of tasks, it is necessary to create a model specialized for one task.
  • the inference device provides a specific improvement over the conventional method using Transformer and Object Query, which required creating task-specific models for each task and performing inference, resulting in costs according to the number of tasks, and represents an advancement in the technical field related to multitask AI.
  • FIG. 2 and Fig. 3 show an example of the processing in the inference device of this embodiment. Specifically, as shown in Fig. 2, in the inference device, the Transformer Decoder and Object Query are expanded to multitask in order to solve multiple tasks simultaneously.
  • the inference device prepares the same number of types of Object Queries as the number of tasks for each task, inputs them together into the Self-Attention of the Transformer Decoder, and performs a process to extract the relationships between the predictions of the Object Queries. This makes it possible to extract the relationships between the predictions of multiple tasks.
  • Cross-Attention branches as many times as the number of tasks, and performs processing to extract the relationship between image features and the prediction of the Object Query for each type of Object Query corresponding to each task.
  • This makes it possible to incorporate Cross-Attention technology that has developed independently for each task.
  • the inference device uses a multi-task Object Query to perform Self-Attention processing across tasks in the Transformer Decoder, and Cross-Attention processing unique to the branched task. This makes it possible to perform multi-task inference more efficiently than the conventional processing for a single task as illustrated in Figure 3(a).
  • FIG. 4 is a schematic diagram illustrating the general configuration of the inference device of this embodiment.
  • Fig. 5 and Fig. 6 are diagrams for explaining the processing of the inference device.
  • the inference device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.
  • the input unit 11 is realized using input devices such as a keyboard and a mouse, and inputs various instruction information such as starting processing to the control unit 15 in response to input operations by an operator.
  • the output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of the inference processing described below.
  • the communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet.
  • the communication control unit 13 controls communication between the control unit 15 and a management device that manages various types of information.
  • Image 14a stores the image to be processed and input to the inference process described below.
  • image feature generation parameters 14b image feature generation parameters used in the inference process described below are stored.
  • the image feature generation parameters are parameters of a neural network model, and are obtained by supervised learning of a general neural network using learning data for the task.
  • image feature refinement parameters 14c image feature refinement parameters used in the inference process described below are stored.
  • the image feature refinement parameters are parameters of a neural network model, and like the image feature generation parameters, are obtained by supervised learning of a general neural network using the learning data for the task.
  • Prediction generation parameters 14d store prediction generation parameters used in the inference process described below.
  • the prediction generation parameters are parameters of a neural network model, and like the image feature generation parameters, are obtained by supervised learning of a general neural network using learning data for the task.
  • Query parameters 14e store query parameters used in the inference process described below.
  • the query parameters are parameters that represent the feature quantities of a task, such as an Object Query, and the same number of query parameters as the number of tasks N are stored corresponding to each of the multiple tasks.
  • N types of query parameters are stored: task 1 query parameters, task 2 query parameters, ..., task N query parameters.
  • the image feature generating unit 15b uses the image feature generating parameters acquired from the storage unit 14 and the image to be processed to generate image features for the image to be processed based on the image feature generating parameters.
  • the image feature generating unit 15b corresponds to, for example, a Transformer Encoder.
  • the image feature refinement unit 15c further refines the image feature generated by the image feature generation unit 15b based on the image feature refinement parameters acquired from the storage unit 14.
  • the image feature refinement unit 15c corresponds to a Transformer Encoder, and refines the feature of the media by performing a nonlinear transformation based on Attention, which is a mechanism for further extracting features using the similarity between the features of the media.
  • the image feature refinement unit 15c processes the image features of multiple images, it is sufficient to obtain the image feature refinement parameters only the first time.
  • the aforementioned acquisition unit 15a may acquire refined image features and transfer them to the prediction generation unit 15d described below.
  • the acquisition unit 15a acquires media features and task parameters representing the features of each task for multiple tasks on a specific media, such as an image.
  • the prediction generation unit 15d predicts the task features of each task in the media from the query parameters acquired by the acquisition unit 15a, using a first relationship between the query parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the query parameters representing the features of each task.
  • the prediction generation unit 15d predicts the desired task using the prediction generation parameters and query parameters acquired from the storage unit 14, and the image features refined by the image feature refinement unit 15c.
  • the prediction generation unit 15d processes image features of multiple images, it is sufficient to obtain the prediction generation parameters and query parameters only the first time.
  • query parameters corresponding to each of a plurality of tasks there are query parameters corresponding to each of a plurality of tasks.
  • the query parameters for object detection and the query parameters for pose estimation are input to the prediction generation unit 15d, and prediction results for the two tasks, object detection and pose estimation, are output from the image features refined by the image feature refinement unit 15c.
  • the prediction generation unit 15d predicts task features by repeatedly converting parameters using the similarity between the query parameters as the first relationship, and converting parameters using the similarity between the media features and the query parameters as the second relationship.
  • the prediction generation unit 15d performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts features using the similarity between the query parameters of each task, and performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts features using the similarity between the features of the media and the query parameters of each task.
  • the prediction generation unit 15d repeats this process to predict task features of each task in the media.
  • the prediction generation unit 15d is a decoder of a transformer, which is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.
  • the prediction generation unit 15d is a decoder of a transformer, in which Self-Attention extracts the relationship between the query parameters of each task across different tasks, and Cross-Attention extracts the relationship between the media features and the query parameters of each task for each task, thereby predicting the task features of each task in the media.
  • FIG. 6 is a diagram for explaining the processing of the prediction generation unit.
  • the prediction generation unit 15d includes an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f.
  • the inter-prediction relationship extraction unit 15e corresponds to, for example, a self-attention of a transformer decoder, and performs the same processing across tasks regardless of the type and number of tasks.
  • the image-inter-prediction relationship extraction unit 15f corresponds to, for example, a cross-attention of a transformer decoder, and performs different processing depending on the type of task, so that the same number of image-inter-prediction relationship extraction units 15f as the number of tasks perform processing in parallel.
  • the prediction generation unit 15d has a predetermined number L of layers, each of which is a pair of an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f.
  • the query parameters change gradually with each layer, and finally a prediction is output, which is the inference result for each task.
  • inter-prediction relationship extraction unit 15e extracts relationships between each query parameter based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (hereinafter, the updated query parameters may be referred to as "query parameters (prediction)").
  • image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (prediction) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.
  • the prediction relationship extraction unit 15e extracts relationships between each query parameter (prediction) based on the updated prediction generation parameters, and updates each query parameter based on the extracted relationships.
  • the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (prediction) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.
  • the inter-prediction relationship extraction unit 15e and the image-prediction relationship extraction unit 15f repeat the same process up to the L layer, thereby outputting a prediction that is the inference result for each task.
  • the inference device 10 can efficiently execute inference for multiple tasks for media. Furthermore, learning of various parameters can also be performed collectively for multiple tasks, making it possible to execute the learning efficiently.
  • Fig. 7 to Fig. 11 are flowcharts showing the procedure of the inference process. First, in Fig. 7, the flowchart starts at the timing when the user performs an operation input to instruct the start of the process.
  • the acquisition unit 15a acquires from the storage unit 14 the image to be processed, parameters for generating image features, parameters for refining image features, parameters for predictive generation, parameters for queries, etc., to be used in the inference process.
  • the image feature generating unit 15b uses the image feature generating parameters and the image to be processed to generate image features for the image to be processed based on the image feature generating parameters (step S1).
  • the image feature refinement unit 15c further refines the image features generated by the image feature generation unit 15b based on the image feature refinement parameters (step S2).
  • the prediction generation unit 15d uses the prediction generation parameters, the refined image features, and the query parameters for each task to generate a prediction of the features of each task in the image (step S3), and outputs the generated prediction (step S4). This completes the series of inference processes.
  • FIG. 8 illustrates the processing procedure of the image feature generation unit in step S1 shown in FIG. 7.
  • the image feature generation unit 15b acquires image feature generation parameters and an input image to be processed from the acquisition unit 15a or the storage unit 14 (steps S101 to S102).
  • the image feature generating unit 15b generates image features for the input image to be processed based on the image feature generation parameters (step S103).
  • the image feature generating unit 15b also outputs the generated image features to the downstream image feature refining unit 15c (step S104), and the series of processes ends.
  • FIG. 9 illustrates the processing procedure of the image feature generation unit in step S2 shown in FIG. 7.
  • the image feature refinement unit 15c acquires parameters for image feature refinement from the acquisition unit 15a or the storage unit 14 (step S201), and acquires the image feature amount generated from the image feature generation unit 15b (step S202).
  • the image feature refinement unit 15c refines the image feature based on the image feature refinement parameters (step S203).
  • the image feature generation unit 15b outputs the refined image feature to the downstream prediction generation unit 15d (step S204), and the series of processes ends.
  • FIG. 10 illustrates the processing procedure of the prediction generation unit in steps S3 to S4 shown in FIG. 7.
  • the prediction generation unit 15d acquires prediction generation parameters and query parameters from the acquisition unit 15a or the storage unit 14 (steps S301 to S302), and acquires refined image features from the image feature refinement unit 15c (step S303).
  • FIG. 11 also illustrates a detailed processing procedure for the process of step S304 shown in FIG. 10.
  • the prediction generation unit 15d performs the first layer processing shown in FIG. 6. Specifically, the prediction relationship extraction unit 15e extracts relationships between query parameters for each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (step S401).
  • the image-prediction relationship extraction unit 15f extracts relationships between the image features and the updated query parameters for each task, and further updates the query parameters for each task based on the extracted relationships (step S402).
  • the prediction generation unit 15d repeats the same process up to the Lth layer (step S413), and sets the updated query parameters in the Lth layer as a prediction for the generated task (step S404). After that, the prediction generation unit 15d outputs the prediction for the generated task (step S305), and ends the series of processes.
  • the prediction generation unit 15d is a decoder of a transformer, which is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.
  • the inference program is stored in the hard disk drive 1031, for example, as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the inference device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

Dans un dispositif d'inférence (10), une unité d'acquisition (15a) acquiert, pour une pluralité de tâches pour un support prescrit, une quantité de caractéristiques du support et un paramètre d'interrogation représentant une quantité de caractéristiques de chaque tâche. Une unité de génération de prédiction (15d) prédit une quantité de fonctionnalités de tâche de chaque tâche pour le support à partir des paramètres d'interrogation acquis par l'unité d'acquisition (15a), au moyen d'une première relation entre les paramètres d'interrogation représentant les quantités de caractéristiques de tâches qui croisent différentes tâches et une seconde relation entre la quantité de caractéristiques du support et le paramètre d'interrogation représentant la quantité de caractéristiques de chaque tâche.
PCT/JP2023/029898 2023-08-18 2023-08-18 Dispositif d'inférence, procédé d'inférence et programme d'inférence Pending WO2025041210A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2025541172A JPWO2025041210A1 (fr) 2023-08-18 2023-08-18
PCT/JP2023/029898 WO2025041210A1 (fr) 2023-08-18 2023-08-18 Dispositif d'inférence, procédé d'inférence et programme d'inférence

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/029898 WO2025041210A1 (fr) 2023-08-18 2023-08-18 Dispositif d'inférence, procédé d'inférence et programme d'inférence

Publications (1)

Publication Number Publication Date
WO2025041210A1 true WO2025041210A1 (fr) 2025-02-27

Family

ID=94731785

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/029898 Pending WO2025041210A1 (fr) 2023-08-18 2023-08-18 Dispositif d'inférence, procédé d'inférence et programme d'inférence

Country Status (2)

Country Link
JP (1) JPWO2025041210A1 (fr)
WO (1) WO2025041210A1 (fr)

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
RANFTL RENE; BOCHKOVSKIY ALEXEY; KOLTUN VLADLEN: "Vision Transformers for Dense Prediction", 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, 10 October 2021 (2021-10-10), pages 12159 - 12168, XP034093831, DOI: 10.1109/ICCV48922.2021.01196 *
vol. 13, 1 January 1900, SPRINGER, article CARION NICOLAS; MASSA FRANCISCO; SYNNAEVE GABRIEL; USUNIER NICOLAS; KIRILLOV ALEXANDER; ZAGORUYKO SERGEY: "End-to-End Object Detection with Transformers", pages: 213 - 229, XP047569461, DOI: 10.1007/978-3-030-58452-8_13 *
XU YANGYANG; LI XIANGTAI; YUAN HAOBO; YANG YIBO; ZHANG LEFEI: "Multi-Task Learning With Multi-Query Transformer for Dense Prediction", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE, USA, vol. 34, no. 2, 6 July 2023 (2023-07-06), USA, pages 1228 - 1240, XP011959532, ISSN: 1051-8215, DOI: 10.1109/TCSVT.2023.3292995 *

Also Published As

Publication number Publication date
JPWO2025041210A1 (fr) 2025-02-27

Similar Documents

Publication Publication Date Title
JP7286013B2 (ja) ビデオコンテンツ認識方法、装置、プログラム及びコンピュータデバイス
US12548306B2 (en) Method, device and storage medium for training model based on multi-modal data joint learning
US20230229898A1 (en) Data processing method and related device
CN114970522B (zh) 语言模型的预训练方法、装置、设备、存储介质
CN113094125B (zh) 业务流程处理方法、装置、服务器及存储介质
JP7269972B2 (ja) 事前トレーニングのセマンティックモデルに基づくモデル改良方法及び装置
CN116543075B (zh) 图像生成方法、装置、电子设备及存储介质
CN115809325B (zh) 文档处理模型训练方法、文档处理方法、装置及设备
WO2025261111A1 (fr) Modèle génératif multimodal de texte et d'image entrelacés basé sur un synchroniseur de caractéristiques dynamiques
CN118520912A (zh) 稀疏注意力网络的稀疏处理方法、装置及电子设备
JP2023160198A (ja) 継続学習装置、継続学習方法および継続学習プログラム
CN118364061A (zh) 基于工具增强的大语言模型的多模态处理方法及装置
JP2023533404A (ja) 駆動可能３ｄキャラクター生成方法、装置、電子機器、及び記憶媒体
CN116362240A (zh) 文本处理的方法、装置、设备和介质
CN116976467A (zh) 基于大模型的多任务处理方法、系统及计算设备
Mediakov et al. Specifics of Designing and Construction of the System for Deep Neural Networks Generation
CN113919490A (zh) 推理引擎的适配方法、装置及电子设备
US20240403636A1 (en) Self-attention based neural networks for processing network inputs from multiple modalities
US12288102B2 (en) System and method for queuing node load in data preprocessing
WO2020095655A1 (fr) Dispositif et procédé de sélection
CN113312555B (zh) 转化率预测方法、装置、电子设备和存储介质
CN117273069A (zh) 一种基于神经网络模型的推理方法、装置、设备及介质
CN115756692A (zh) 基于样式属性自动组合展示页面的方法及其相关设备
US11640428B2 (en) Collation device, collation method, and collation program
CN118394483A (zh) 用于大模型的任务执行方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23949672

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025541172

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025541172

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE