WO2021250811A1 - データ処理装置、データ処理方法及び記録媒体 - Google Patents

データ処理装置、データ処理方法及び記録媒体 Download PDF

Info

Publication number
WO2021250811A1
WO2021250811A1 PCT/JP2020/022844 JP2020022844W WO2021250811A1 WO 2021250811 A1 WO2021250811 A1 WO 2021250811A1 JP 2020022844 W JP2020022844 W JP 2020022844W WO 2021250811 A1 WO2021250811 A1 WO 2021250811A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature vector
vector
data processing
feature
map information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/022844
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
亮 高本
宏 福井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to JP2022530425A priority Critical patent/JP7524946B2/ja
Priority to PCT/JP2020/022844 priority patent/WO2021250811A1/ja
Priority to US18/009,080 priority patent/US20230214447A1/en
Publication of WO2021250811A1 publication Critical patent/WO2021250811A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present disclosure relates to a technical field of a data processing apparatus, a data processing method, and a recording medium capable of generating a feature vector.
  • Non-Patent Document 1 a method of performing a synthesis process for synthesizing a plurality of feature vectors indicating the feature quantities of a plurality of types of data, and performing a desired arithmetic process using the feature vectors generated by the synthesis process (so-called so-called). How to perform multimodal processing) is described.
  • Non-Patent Document 1 performs a synthesis process in which a plurality of feature vectors are simply added (for example, added along the channel direction). That is, the method described in Non-Patent Document 1 always synthesizes a plurality of feature vectors by the same method without considering the contents of the plurality of feature vectors. Therefore, the method described in Non-Patent Document 1 has a technical problem that it is not always possible to appropriately synthesize a plurality of feature vectors.
  • the same technical problem may occur not only in the scene where a plurality of feature vectors are synthesized but also in any scene where another feature vector is generated by using a plurality of feature vectors.
  • a first aspect of the data processing apparatus of the present disclosure is a data processing apparatus that generates a third feature vector from a first feature vector and a second feature vector, wherein the first and second feature vectors are generated. Based on the above, the map information showing the distribution of the vector component having a relatively high importance among the plurality of vector components constituting the fourth feature vector obtained by synthesizing the first and second feature vectors is obtained. A calculation means for calculating and a generation means for generating the third feature vector by using the fourth feature vector and the map information are provided.
  • a second aspect of the data processing apparatus of the present disclosure is a data processing apparatus that generates a third feature vector from a first feature vector and a second feature vector, wherein the first and second feature vectors are generated.
  • the first aspect of the data processing method of the present disclosure is a data processing method for generating a third feature vector from a first feature vector and a second feature vector, wherein the first and second feature vectors are generated.
  • the map information showing the distribution of the vector component having a relatively high importance among the plurality of vector components constituting the fourth feature vector obtained by synthesizing the first and second feature vectors is obtained. It includes a calculation step of calculation and a generation step of generating the third feature vector by using the fourth feature vector and the map information.
  • a second aspect of the data processing method of the present disclosure is a data processing method for generating a third feature vector from a first feature vector and a second feature vector, wherein the first and second feature vectors are generated.
  • the first aspect of the recording medium of the present disclosure is a recording medium in which a computer program for causing a computer to execute a data processing method is recorded, and the data processing method has a first feature vector and a second feature.
  • the first It includes a generation step of generating the feature vector of 3.
  • a second aspect of the recording medium of the present disclosure is a recording medium on which a computer program for causing a computer to execute a data processing method is recorded, wherein the data processing method has a first feature vector and a second feature.
  • FIG. 1 is a block diagram showing a configuration of a data processing device of the present embodiment.
  • FIG. 2 is a block diagram showing an example of a feature vector generation unit and a map calculation unit that form at least a part of the attention mechanism.
  • FIG. 3 is a flowchart showing the flow of vector calculation processing performed by the data processing apparatus of the present embodiment.
  • FIG. 4 is a block diagram showing a configuration of a data processing device of the first modification.
  • FIG. 5 is a block diagram showing the configuration of the data processing device of the second modification.
  • FIG. 6 is a block diagram showing an example of a feature vector generation unit and a map calculation unit that form at least a part of the attention mechanism in the second modification.
  • FIG. 7 is a block diagram showing the configuration of the data processing device of the third modification.
  • FIG. 8 is a block diagram showing a configuration of a data processing device of the fourth modification.
  • FIG. 1 is a block diagram showing a configuration of the data processing device 1 of the present embodiment.
  • the data processing device 1 includes an arithmetic unit 2 and a storage device 3. Further, the data processing device 1 may include an input device 4 and an output device 5. However, the data processing device 1 does not have to include at least one of the input device 4 and the output device 5.
  • the arithmetic unit 2, the storage device 3, the input device 4, and the output device 5 are connected via the data bus 6.
  • the arithmetic unit 2 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an FPGA (Field Programmable Gate Array).
  • the arithmetic unit 2 reads a computer program.
  • the arithmetic unit 2 may read the computer program stored in the storage device 3.
  • the arithmetic unit 2 may read a computer program stored in a recording medium that is readable by a computer and is not temporary by using a recording medium reading device (not shown).
  • the arithmetic unit 2 may acquire a computer program from a device (not shown) arranged outside the data processing device 1 via a communication device (not shown) (that is, it may be downloaded or read). ..
  • the arithmetic unit 2 executes the read computer program.
  • a logical functional block for executing the operation to be performed by the data processing device 1 is realized in the arithmetic unit 2. That is, the arithmetic unit 2 can function as a controller for realizing a logical functional block for executing an operation to be performed by the data processing unit 1.
  • FIG. 1 shows an example of a logical functional block for performing vector arithmetic processing.
  • a feature vector generation unit 21, a feature vector generation unit 22, a feature vector generation unit 23, a map calculation unit 24, and an arithmetic unit are included. 25 and is realized.
  • the feature vector generation unit 21, the feature vector generation unit 22, the feature vector generation unit 23, the map calculation unit 24, and the calculation unit 25 are typically a learnable learning model (for example, a learning model based on a neural network). It is a functional block realized by.
  • the learning model that defines the operation contents of the feature vector generation unit 21, the feature vector generation unit 22, the feature vector generation unit 23, the map calculation unit 24, and the calculation unit 25 uses the learning data associated with the correct label. It may be constructed (in other words, updated) according to the learning operation used. However, at least one of the feature vector generation unit 21, the feature vector generation unit 22, the feature vector generation unit 23, the map calculation unit 24, and the calculation unit 25 does not have to be a functional block realized by the learning model.
  • the feature vector generation unit 21 generates a feature vector z1 indicating the feature amount of the data D1 from the data D1.
  • the feature vector generation unit 21 outputs the generated feature vector z1 to each of the feature vector generation unit 23 and the map calculation unit 24.
  • the data D1 is arbitrary data that can be handled by the data processing device 1.
  • the data D1 may include image data, audio data, text data, or other formats of data.
  • the feature vector generation unit 22 generates a feature vector z2 indicating the feature amount of the data D2 from the data D2 different from the data D1.
  • the feature vector generation unit 22 outputs the generated feature vector z2 to the feature vector generation unit 23 and the map calculation unit 24, respectively.
  • the data D2 is arbitrary data that can be handled by the data processing device 1.
  • the data D2 may include image data, audio data, text data, or other formats of data.
  • the feature vector generation unit 23 generates the feature vector z3 from the feature vectors z1 and z2. Specifically, the feature vector generation unit 23 first generates the feature vector z4 by synthesizing the feature vectors z1 and z2. For example, the feature vector generation unit 23 may generate the feature vector z4 by synthesizing the feature vectors z1 and z2 in the channel direction (that is, performing a so-called concatenate operation). In this case, for example, even if the N1 + N2 dimensional feature vector z4 is generated from the N1 (N1 is an integer of 1 or more) dimensional feature vector z1 and the N2 (N2 is an integer of 1 or more) dimensional feature vector z2. good.
  • the map calculation unit 24 calculates the map information AP based on the feature vectors z1 and z2.
  • the map information AP shows the distribution of the vector components having a relatively high importance among the plurality of vector components constituting the feature vector z4.
  • the map information AP indicates the distribution of the vector components to be paid attention to among the plurality of vector components constituting the feature vector z4.
  • the map information AP may be considered to indicate the weights of the plurality of vector components constituting the feature vector z4.
  • the map information AP may show the distribution of vector components that are relatively important for the generation of the feature vector z3 among the plurality of vector components constituting the feature vector z4.
  • the map information AP may indicate the distribution of the vector components that should be paid attention to when the feature vector z3 is generated among the plurality of vector components constituting the feature vector z4.
  • the vector component that is relatively important for the generation of the feature vector z3 contributes to the accuracy of the arithmetic processing performed by the arithmetic unit 25 described later. It may mean a vector component having a relatively large degree. That is, the vector component having a relatively high importance for the generation of the feature vector z3 may mean a vector component having a greater contribution to the increase in the accuracy of the arithmetic processing performed by the arithmetic unit 25 than the other vector components. ..
  • the map calculation unit 24 may consider that the map information AP is calculated by using the attention mechanism that calculates the map information AP as a weight. In other words, the map calculation unit 24 may be regarded as constituting at least a part of the attention mechanism that calculates the map information AP as a weight.
  • the map information AP When the map information AP is generated by using the attention mechanism, the map information AP may be referred to as an attention level map (attention map). However, the map calculation unit 24 may calculate the map information AP without using the attention mechanism.
  • FIG. 2 shows an example of the map calculation unit 24 that constitutes at least a part of the attention mechanism.
  • the map calculation unit 24 may include a feature vector generation unit 241 and a map calculation unit 242.
  • the feature vector generation unit 241 generates the feature vector z5 by synthesizing the feature vectors z1 and z2.
  • the feature vector z5 generated by the feature vector generation unit 241 may be the same vector as the feature vector z4 generated by the feature vector generation unit 23 described above. In this case, even if the feature vector generation unit 241 generates the feature vector z5 by synthesizing the feature vectors z1 and z2 in the channel direction (that is, performing a so-called concatenate operation), similarly to the feature vector generation unit 23. good.
  • the map calculation unit 242 calculates the map information AP based on the feature vector z5. Specifically, the map calculation unit 242 may calculate the map information AP by using the feature vector z5 as a key and a query in the attention mechanism. Alternatively, the map calculation unit 242 performs a first process (for example, a first 1 ⁇ 1 convolution process) on the feature vector z5 to generate a key in the attention mechanism, and the map calculation unit 242 generates a key for the feature vector z5.
  • the query in the attention mechanism may be generated by performing the second process (for example, the second 1 ⁇ 1 convolution process). In any case, in the example shown in FIG.
  • the map calculation unit 24 constitutes at least a part of the self-attention mechanism using the key and the query derived from the same input. That is, it can be said that the map calculation unit 24 uses the self-attention mechanism to generate the map information AP.
  • the map calculation unit 242 may calculate the map information AP by using an arbitrary method for calculating the weight from the key and the query in the attention mechanism. For example, the map calculation unit 242 may calculate the matrix sum of the key and the query as the map information AP. For example, the map calculation unit 242 may calculate the matrix product of the key and the query as the map information AP.
  • the generated map information AP is a matrix (or vector) containing weights indicating the distribution of vector components to be paid attention to as elements.
  • the map calculation unit 24 may calculate the normalized map information AP. That is, the map calculation unit 24 may normalize the calculated map information AP. For example, the map calculation unit 24 may use a sigmoid function to normalize the map information AP. As a result, the map information AP is normalized so that each element of the map information AP has a value between 0 and 1. Alternatively, for example, the map calculation unit 24 may use the softmax function to normalize the map information AP. As a result, the map information AP is normalized so that the sum of the elements of each row and each column of the map information AP is 1.
  • the feature vector generation unit 23 When the map calculation unit 24 constitutes at least a part of the attention mechanism, the feature vector generation unit 23 also calculates the feature vector z3 by using the attention mechanism that performs an operation using the map information AP as a weight. It may be considered that it is. In other words, the feature vector generation unit 23 may be regarded as constituting at least a part of the attention mechanism that performs the operation using the map information AP as the weight. However, the feature vector generation unit 23 may generate the feature vector z3 without using the attention mechanism.
  • FIG. 2 shows an example of the feature vector generation unit 23 that constitutes at least a part of the attention mechanism.
  • the feature vector generation unit 23 may include a feature vector generation unit 231, a multiplication unit 232, and an addition unit 233.
  • the feature vector generation unit 231 generates the feature vector z4 by synthesizing the feature vectors z1 and z2.
  • the feature vector generation unit 231 generates the feature vector z4 by synthesizing the feature vectors z1 and z2, and the generated feature vector z4 is subjected to a third process (for example, a third 1 ⁇ 1 convolution process). May be done.
  • a third process for example, a third 1 ⁇ 1 convolution process
  • the feature vector generation unit 231 outputs the generated feature vector z4 (or the feature vector z4 subjected to the third processing) to the multiplication unit 232 and the addition unit 233, respectively.
  • the multiplication unit 232 calculates the matrix product of the feature vector z4 output from the feature vector generation unit 231 and the map information AP (that is, generates the feature vector z4 ⁇ AP).
  • the feature vector z4 output from the feature vector generation unit 231 may be regarded as corresponding to the value in the attention mechanism.
  • the value is also a vector derived from the same input as the key and the query.
  • the feature vector generation unit 23 does not have to include the addition unit 233.
  • the feature vector z4 ⁇ AP output by the multiplication unit 232 may be used as the feature vector z3.
  • the feature vector generation unit 23 acquires the feature vector z5 from the map calculation unit 24 in addition to or instead of generating the feature vector z4 by using the feature vector generation unit 231 and obtains the acquired feature vector z5. It may be used as the feature vector z4.
  • the feature vector generation unit 23 acquires the feature vector z5 from the map calculation unit 24 as the feature vector z4, the feature vector generation unit 23 does not have to include the feature vector generation unit 231.
  • the calculation unit 25 performs a desired calculation process using the feature vector z3 generated by the feature vector generation unit 23.
  • the feature vector z3 is the person's.
  • the feature amount related to the face and the feature amount related to the words of the person may be shown.
  • the calculation unit 25 may perform a calculation process for estimating the emotion of the person reflected in the image based on the feature vector z3.
  • the calculation unit 25 may perform a calculation process for adding a caption (that is, subtitles) indicating a person's words to the image based on the feature vector z3.
  • the storage device 3 can store desired data.
  • the storage device 3 may temporarily store the computer program executed by the arithmetic unit 2.
  • the storage device 3 may temporarily store data temporarily used by the arithmetic unit 2 while the arithmetic unit 2 is executing a computer program.
  • the storage device 3 may store data stored for a long period of time by the data processing device 1.
  • the storage device 3 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. good. That is, the storage device 3 may include a recording medium that is not temporary.
  • the input device 4 is a device that receives input of information to the data processing device 1 from the outside of the data processing device 1.
  • the output device 5 is a device that outputs information to the outside of the data processing device 1.
  • the output device 5 may output information regarding the vector calculation processing performed by the data processing device 1.
  • FIG. 3 is a flowchart showing the flow of vector calculation processing performed by the data processing device 1 of the present embodiment.
  • the feature vector generation unit 21 acquires the data D1 (step S11). Further, the feature vector generation unit 22 acquires the data D2 (step S11). After that, the feature vector generation unit 21 generates the feature vector z1 from the data D1 acquired in step S11 (step S12). Further, the feature vector generation unit 22 generates the feature vector z2 from the data D2 acquired in step S11 (step S12).
  • the map calculation unit 24 calculates the map information AP based on the feature vectors z1 and z2 generated in step S12 (step S13). Since the method of calculating the map information AP has already been described, the description thereof will be omitted here.
  • the feature vector generation unit 23 After that, the feature vector generation unit 23 generates the feature vector z3 based on the feature vectors z1 and z2 generated in step S12 and the map information AP calculated in step S13 (step S14). Since the method of generating the feature vector z3 based on the map information AP has already been described, the description thereof will be omitted here.
  • the calculation unit 25 performs a desired calculation process using the feature vector z3 generated in step S14 (step S15).
  • the data processing device 1 can appropriately generate the feature vector z3 using the feature vectors z1 and z2. Specifically, the data processing apparatus 1 can generate the feature vector z3 by using the map information AP in addition to the feature vectors z1 and z2. As a result, the feature vector z3 has a vector component having a relatively high importance and a vector having a relatively low importance as compared with the feature vector z4 generated by simply synthesizing the feature vectors z1 and z2. It becomes a feature vector that is emphasized rather than the component.
  • the feature vector obtained by synthesizing the feature vectors z1 and z2, and the vector component having a relatively high importance is emphasized rather than the vector component having a relatively low importance.
  • the feature vector can be generated as the feature vector z3. Therefore, the accuracy of the arithmetic processing performed by the arithmetic unit 25 using the feature vector z3 is higher than the accuracy of the arithmetic processing performed by the arithmetic unit 25 using the feature vector z4.
  • the contents of the feature vectors z1 and z2 change.
  • the contents of the map information AP calculated based on the feature vectors z1 and z2 also change. That is, it can be said that the map calculation unit 24 changes the map information AP according to the contents of the data D1 and the data D2. Therefore, it can be said that the feature vector generation unit 23 that generates the feature vector z3 based on the map information AP has changed the method of generating the feature vector z3 according to the contents of the data D1 and the data D2.
  • the feature generated by the generation method changed according to the contents of the data D1 and the data D2 as compared with the feature vector z4 generated by the fixed generation method regardless of the contents of the data D1 and the data D2. It can be said that the vector z3 more appropriately indicates the feature amounts of the data D1 and the data D2 (particularly, the feature amounts desired to be extracted for the processing finally realized by using the data D1 and D2). As a result, the accuracy of the arithmetic processing performed by the arithmetic unit 25 using the feature vector z3 is higher than the accuracy of the arithmetic processing performed by the arithmetic unit 25 using the feature vector z4.
  • FIG. 4 is a block diagram showing a configuration of the data processing device 1a of the first modification.
  • the feature vector generation units 21 and 22 generate feature vectors z1 and z2 from the same data D1a, respectively, as compared with the data processing device 1 described above. It is different in that.
  • Other features of the data processing device 1a may be the same as other features of the data processing device 1.
  • the feature vector generation unit 21 generates a feature vector z1 indicating the first feature amount of the data D1a from the data D1a.
  • the feature vector generation unit 22 generates a feature vector z2 indicating a second feature amount of the data D1a different from the first feature amount from the data D1a.
  • the feature vector generation unit 21 when image data showing an image in which a person is reflected is used as data D1a, the feature vector generation unit 21 generates a feature vector z1 indicating a feature amount related to the direction of the line of sight of the person, and the feature vector generation unit 21. 22 may generate a feature vector z2 indicating a feature amount relating to the orientation of the person's face.
  • the data processing device 1a of the first modification can enjoy the same effect as the effect that the above-mentioned data processing device 1 can enjoy.
  • FIG. 5 is a block diagram showing the configuration of the data processing device 1b of the second modification.
  • the data processing device 1b of the second modification is characterized in place of the feature vector generation unit 21, the feature vector generation unit 23, and the map calculation unit 24, as compared with the data processing device 1 described above. It differs in that it includes a vector generation unit 21b, a feature vector generation unit 23b, and a map calculation unit 24b. Other features of the data processing device 1b may be the same as other features of the data processing device 1.
  • the feature vector generation unit 21b is different from the feature vector generation unit 21 in that it includes an intermediate vector generation unit 211b and a feature vector generation unit 212b.
  • the other features of the feature vector generation unit 21b may be the same as the other features of the feature vector generation unit 21.
  • the intermediate vector generation unit 211b generates an intermediate vector z1b_int used for generating the feature vector z1 from the data D1.
  • the intermediate vector z1b_int may also be regarded as a vector indicating the feature amount of the data D1 as in the feature vector z1.
  • the feature vector generation unit 212b generates the feature vector z1 from the intermediate vector z1b_int.
  • the feature vector generation unit 212b generates the feature vector z1 by using the map information AP calculated by the map calculation unit 24b in addition to the intermediate vector z1b_int.
  • the feature vector z1 of the second modification generated by using the map information AP is referred to as “feature vector z1b” to distinguish it from the feature vector z1 generated without using the map information AP.
  • the feature vector generation unit 23b generates the feature vector z3 from the feature vector z1b and the feature vector z2 generated by using the map information AP, and the feature vector z1 and the feature generated without using the map information AP. It is different from the above-mentioned feature vector generation unit 23 that generates the feature vector z3 from the vector z2. Further, the feature vector generation unit 23b uses the map information AP when generating the feature vector z3 in that the map information AP does not have to be used when the feature vector z3 is generated. Is different.
  • the feature vector generation unit 23b may generate the feature vector z3 by, for example, synthesizing the feature vectors z1b and z2.
  • the feature vector generation unit 23 may generate the feature vector z3 by synthesizing the feature vectors z1b and z2 in the channel direction (that is, performing a so-called concatenate operation).
  • the other features of the feature vector generation unit 23b may be the same as the other features of the feature vector generation unit 23.
  • the map calculation unit 24b is different from the above-mentioned map calculation unit 24 that calculates the map information AP based on the feature vectors z1 and z2 in that the map information AP is calculated based on the intermediate vector z1b_int and the feature vector z2. That is, the map calculation unit 24b is different from the map calculation unit 24 described above in that the map information AP is calculated using the intermediate vector z1b_int generated in the process of generating the feature vector z1. Other features of the map calculation unit 24b may be the same as other features of the map calculation unit 24.
  • the map calculation unit 24b may include a map calculation unit 242 in the same manner as the map calculation unit 24 described above.
  • the intermediate vector z1b_int and the feature vector z2 are input to the map calculation unit 242.
  • the map calculation unit 242 may calculate the map information AP by using the intermediate vector z1b_int as a key in the attention mechanism and the feature vector z2 as a query in the attention mechanism.
  • the map calculation unit 242 performs a fourth process (for example, a fourth 1 ⁇ 1 convolution process) on the intermediate vector z1b_int to generate a key in the attention mechanism, and the map calculation unit 242 generates a key for the feature vector z2.
  • the query in the attention mechanism may be generated by performing the process of 5 (for example, the fifth 1 ⁇ 1 convolution process).
  • the map calculation unit 24 constitutes at least a part of the source / target attention mechanism using the key and the query resulting from different inputs. That is, it can be said that the map calculation unit 24 uses the source / target attention mechanism to generate the map information AP.
  • the map calculation unit 242 may calculate the map information AP by using an arbitrary method for calculating the weight from the key and the query in the attention mechanism.
  • the map calculation unit 24b may form at least a part of the attention mechanism. Further, when the map calculation unit 24b constitutes at least a part of the attention mechanism, the feature vector generation unit 21b (particularly, the feature vector generation unit 212b) that generates the feature vector z1b using the map information AP is used for map information. It may be considered that the feature vector z1b is calculated by using a caution mechanism that performs an operation using AP as a weight. In other words, the feature vector generation unit 21b (particularly, the feature vector generation unit 212b) may be regarded as constituting at least a part of the attention mechanism that performs the operation using the map information AP as a weight.
  • FIG. 6 shows an example of the feature vector generation unit 21b (particularly, the feature vector generation unit 212b) that constitutes at least a part of the attention mechanism.
  • the feature vector generation unit 212b may include a multiplication unit 2121b and an addition unit 2122b.
  • the multiplication unit 2121b generates the feature vector z1b_int ⁇ AP by multiplying the intermediate vector z1b_int generated by the intermediate vector generation unit 211b by the map information AP.
  • the multiplication unit 2121b performs a sixth process (for example, a sixth 1 ⁇ 1 convolution process) on the intermediate vector z1b_int, and multiplies the intermediate vector z1b_int on which the sixth process is performed by the map information AP.
  • a sixth process for example, a sixth 1 ⁇ 1 convolution process
  • the feature vector z1b_int ⁇ AP may be generated.
  • the intermediate vector z1b_int input to the multiplication unit 2121b may be regarded as corresponding to the value in the attention mechanism.
  • the source / target attention mechanism is used in the example shown in FIG.
  • the value is a vector derived from the same input as the key (so-called source), and the query is an input different from the source (so-called so-called source). It becomes a vector caused by the target).
  • the feature vector generation unit 212b does not have to include the addition unit 2122b.
  • the feature vector z1b_int ⁇ AP output by the multiplication unit 2121b may be used as the feature vector z1b.
  • the map information AP is used in the process of generating the feature vector z1b.
  • the map information AP is generated from the intermediate vector z1b_int and the feature vector z2 generated in the process of generating the feature vector z1b.
  • the map information AP substantially shows the distribution of the vector components having a relatively high importance among the plurality of vector components constituting the intermediate vector z1b_int. That is, the map information AP shows the distribution of the vector components that are relatively important for the generation of the feature vector z1b among the plurality of vector components constituting the intermediate vector z1b_int.
  • the map information AP is a distribution of vector components that are relatively important for the generation of the feature vector z3 among the plurality of vector components constituting the intermediate vector z1b_int. It can be said that it shows. Therefore, the feature vector z3 generated from the feature vector z1b generated by using the map information AP is also the feature of the data D1 and the data D2 like the above-mentioned feature vector z3 generated by using the map information AP. It can be said that the amount is shown more appropriately. Therefore, the data processing device 1b of the second modification can enjoy the same effect as the effect that can be enjoyed by the above-mentioned data processing device 1.
  • FIG. 7 is a block diagram showing the configuration of the data processing device 1c of the third modification.
  • the data processing device 1c of the third modification is different from the data processing device 1b of the second modification described above in that the feature vector generation unit 23b may not be provided. .. Further, the data processing device 1c is different from the data processing device 1b in that the calculation unit 25 performs the calculation processing using the feature vector z1b, and the calculation unit 25 performs the calculation processing using the feature vector z3. Other features of the data processing device 1c may be the same as other features of the data processing device 1b.
  • the feature vector z1b is generated using the map information AP calculated based on the intermediate vector z1b_int and the feature vector z2. Therefore, the feature vector z1b itself shows not only the feature amount of the data D1 but also the feature amount of the data D2. Therefore, the data processing device 1c of the third modification can appropriately enjoy the same effect as the effect that can be enjoyed by the data processing device 1b of the second modification described above.
  • the calculation unit 25 has the calculation unit 25 as described in the second modification. It is preferable to perform arithmetic processing using the feature vector z3 generated from the feature vector z1b and the feature vector z2 instead of the feature vector z1b.
  • FIG. 8 is a block diagram showing the configuration of the data processing device 1d of the fourth modification.
  • the data processing device 1d of the fourth modification includes at least one of a feature vector generation unit 21, a feature vector generation unit 22, and a calculation unit 25, as compared with the data processing device 1 described above. It differs in that it does not have to be. In the example shown in FIG. 8, the data processing device 1d does not include all of the feature vector generation unit 21, the feature vector generation unit 22, and the calculation unit 25. Other features of the data processing device 1d may be the same as other features of the data processing device 1.
  • each of the feature vector generation unit 23 and the map calculation unit 24 may acquire the feature vector z1 from the outside of the data processing device 1d.
  • each of the feature vector generation unit 23 and the map calculation unit 24 may acquire the feature vector z2 from the outside of the data processing device 1d.
  • the feature vector generation unit 23 may output the generated feature vector z3 to the outside of the data processing device 1d.
  • the data processing apparatus 1 generates the feature vector z3 from the two feature vectors (specifically, the feature vectors z1 and z2). However, the data processing apparatus 1 may generate the feature vector z3 from three or more feature vectors.
  • the data processing device 1 includes three or more feature vector generation units that generate three or more feature vectors, a map calculation unit that calculates a map information AP using three or more feature vectors, and three. It may be provided with a feature vector generation unit that generates another feature vector by using one or more feature vectors and the map information AP.
  • a data processing device that generates a third feature vector (z3) from a first feature vector (z1) and a second feature vector (z2). The importance of the plurality of vector components constituting the fourth feature vector (z4) obtained by synthesizing the first and second feature vectors based on the first and second feature vectors is relative.
  • [Appendix 3] A data processing device that generates a third feature vector (z1b) from a first feature vector (z1b_int) and a second feature vector (z2).
  • map information (AP) showing the distribution of vector components having a relatively high importance among the plurality of vector components constituting the first feature vector (z1b_int) is provided.
  • the generation means (212b) is By multiplying the first feature vector (z1b_int) by the map information (AP) as a weight, a fourth feature vector (z1b_int * AP) is generated.
  • the data processing apparatus which generates the third feature vector (z1b) by adding the fourth feature vector (z1b_int * AP) to the first feature vector (z1b_int).
  • the generation means (212b) generates a fifth feature vector (z3) by synthesizing the third feature vector (z1b) and the second feature vector (z2).
  • the data processing device according to any one of Supplementary note 1 to 5, wherein the calculation means calculates the map information by using a caution mechanism for calculating the map information as a weight.
  • Appendix 7 The data processing apparatus according to any one of Supplementary note 1 to 6, wherein the generation means generates the third feature vector by using a caution mechanism using the map information as a weight.
  • Appendix 8 A first vector generation means for generating the first feature vector indicating the feature amount of the first data from the first data, and Any one of Appendix 1 to 7, comprising a second vector generation means for generating the second feature vector indicating the feature amount of the second data from the second data different from the first data.
  • the data processing apparatus according to any one of 7.
  • the data processing device according to any one of Supplementary note 1 to 9, wherein the calculation means calculates the map information using a neural network.
  • the generation means generates the third feature vector using a neural network.
  • the importance of the plurality of vector components constituting the fourth feature vector (z4) obtained by synthesizing the first and second feature vectors based on the first and second feature vectors is relative.
  • a data processing method including a generation step of generating the third feature vector (z3 z4 * (1 + AP) or z4 * AP) using the fourth feature vector and the map information.
  • the data processing method is A data processing method for generating a third feature vector (z3) from a first feature vector (z1) and a second feature vector (z2).
  • the importance of the plurality of vector components constituting the fourth feature vector (z4) obtained by synthesizing the first and second feature vectors based on the first and second feature vectors is relative.
  • a recording medium including a generation step of generating the third feature vector (z3 z4 * (1 + AP) or z4 * AP) using the fourth feature vector and the map information.
  • AP map information
  • the data processing method is A data processing method for generating a third feature vector (z1b) from a first feature vector (z1b_int) and a second feature vector (z2). Based on the first and second feature vectors, map information (AP) showing the distribution of vector components having a relatively high importance among the plurality of vector components constituting the first feature vector (z1b_int) is provided.
  • Calculation process (24b) to calculate and A recording medium including a generation step (212b) for generating the third feature vector (z1b) using the first feature vector and the map information.
  • Data processing device 2 Arithmetic logic unit 21, 22, 23 Feature vector generator 24 Map calculation unit 25 Arithmetic logic unit z1, z2, z3, z4 Feature vector AP map information

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Analysis (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Algebra (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
PCT/JP2020/022844 2020-06-10 2020-06-10 データ処理装置、データ処理方法及び記録媒体 Ceased WO2021250811A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2022530425A JP7524946B2 (ja) 2020-06-10 2020-06-10 データ処理装置、データ処理方法及び記録媒体
PCT/JP2020/022844 WO2021250811A1 (ja) 2020-06-10 2020-06-10 データ処理装置、データ処理方法及び記録媒体
US18/009,080 US20230214447A1 (en) 2020-06-10 2020-06-10 Data processing apparatus, data processing method, and recording medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/022844 WO2021250811A1 (ja) 2020-06-10 2020-06-10 データ処理装置、データ処理方法及び記録媒体

Publications (1)

Publication Number Publication Date
WO2021250811A1 true WO2021250811A1 (ja) 2021-12-16

Family

ID=78845586

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/022844 Ceased WO2021250811A1 (ja) 2020-06-10 2020-06-10 データ処理装置、データ処理方法及び記録媒体

Country Status (3)

Country Link
US (1) US20230214447A1 (https=)
JP (1) JP7524946B2 (https=)
WO (1) WO2021250811A1 (https=)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019091172A (ja) * 2017-11-13 2019-06-13 日本電信電話株式会社 句構造学習装置、句構造解析装置、方法、及びプログラム
CN111161306A (zh) * 2019-12-31 2020-05-15 北京工业大学 一种基于运动注意力的视频目标分割方法

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019091172A (ja) * 2017-11-13 2019-06-13 日本電信電話株式会社 句構造学習装置、句構造解析装置、方法、及びプログラム
CN111161306A (zh) * 2019-12-31 2020-05-15 北京工业大学 一种基于运动注意力的视频目标分割方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
FUKUI HIROSHI; HIRAKAWA TSUBASA; YAMASHITA TAKAYOSHI; FUJIYOSHI HIRONOBU: "Attention Branch Network: Learning of Attention Mechanism for Visual Explanation", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 15 June 2019 (2019-06-15), pages 10697 - 10706, XP033687020, DOI: 10.1109/CVPR.2019.01096 *
WANG KUNPENG, ZHENG MINGYAO, WEI HONGYAN, QI GUANQIU, LI YUANYUAN: "Multi-Modality Medical Image Fusion Using Convolutional Neural Network and Contrast Pyramid", SENSORS, vol. 20, no. 8, pages 2169, XP055885538, DOI: 10.3390/s20082169 *

Also Published As

Publication number Publication date
JP7524946B2 (ja) 2024-07-30
JPWO2021250811A1 (https=) 2021-12-16
US20230214447A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
US11545135B2 (en) Acoustic model learning device, voice synthesis device, and program
US11373098B2 (en) Processing apparatus, learning apparatus, processing method, and nonvolatile recording medium
JP2015230570A (ja) 学習モデル作成装置、判定システムおよび学習モデル作成方法
CN112700791B (zh) 电子设备及其控制方法
US10546045B2 (en) Efficient extended-precision processing
JP5638503B2 (ja) テキスト要約装置、方法及びプログラム
JP2021184217A (ja) モデル生成プログラム、モデル生成装置、及びモデル生成方法
JP6585846B2 (ja) 秘密計算システム、秘密計算装置、秘密計算方法、およびプログラム
US20240111998A1 (en) Inferring device, inferring method, and training device
US12341885B2 (en) Secret batch approximation system, secure computation device, secret batch approximation method, and program
WO2021250811A1 (ja) データ処理装置、データ処理方法及び記録媒体
AU2018338249B2 (en) Secure reading apparatus, secure writing apparatus, method thereof, and program
KR102409958B1 (ko) 전자 장치, 그 제어 방법 및 서버
KR102559036B1 (ko) 음수값을 갖는 활성화 함수에서 제로 스키핑을 제공하는 방법 및 장치
JP7546822B1 (ja) 機械学習装置、機械学習方法及び機械学習プログラム
JP2020504386A (ja) 情報処理装置、情報処理方法及びプログラム
US20210149339A1 (en) Method and apparatus for synthesizing high resolution computer generated hologram based on mesh
JP2018013608A (ja) 秘匿計算システム、第一秘匿計算装置、第二秘匿計算装置、秘匿回路生成方法、秘匿回路評価方法、プログラム
JP6495781B2 (ja) 音声パラメータ生成装置、音声パラメータ生成方法、プログラム
CN113924610A (zh) 秘密共轭梯度法计算系统、秘密计算装置、共轭梯度法计算装置、秘密共轭梯度法计算方法、共轭梯度法计算方法、以及程序
US20250363366A1 (en) Structure-secret neural network model generation apparatus
CN112447165A (zh) 信息处理、模型训练和模型构建方法、电子设备、智能音箱
JP7617437B2 (ja) 変換学習装置、変換学習方法、変換学習プログラム及び変換装置
WO2020054402A1 (ja) ニューラルネットワーク処理装置、コンピュータプログラム、ニューラルネットワーク製造方法、ニューラルネットワークデータの製造方法、ニューラルネットワーク利用装置、及びニューラルネットワーク小規模化方法
KR20250083652A (ko) 기여도 계산방법 및 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20939649

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022530425

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20939649

Country of ref document: EP

Kind code of ref document: A1