WO2024239783A1 - 一种数据处理方法、装置、设备以及可读存储介质 - Google Patents

一种数据处理方法、装置、设备以及可读存储介质 Download PDF

Info

Publication number
WO2024239783A1
WO2024239783A1 PCT/CN2024/083332 CN2024083332W WO2024239783A1 WO 2024239783 A1 WO2024239783 A1 WO 2024239783A1 CN 2024083332 W CN2024083332 W CN 2024083332W WO 2024239783 A1 WO2024239783 A1 WO 2024239783A1
Authority
WO
WIPO (PCT)
Prior art keywords
modal
task group
modality
training task
type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/083332
Other languages
English (en)
French (fr)
Inventor
刘烨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Publication of WO2024239783A1 publication Critical patent/WO2024239783A1/zh
Priority to US19/233,363 priority Critical patent/US20250299072A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/219—Managing data history or versioning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G06N5/022—Knowledge engineering; Knowledge acquisition
    • G06N5/025—Extracting rules from data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2457—Query processing with adaptation to user needs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and readable storage medium.
  • the task of understanding media data can provide media data with many rich and diverse media tags (such as dance tags, singing tags, competition tags, game tags, etc.).
  • the media tags of media data can facilitate various media processing of media data. For example, it can facilitate media retrieval, classification, archiving, media recommendation, media secondary editing and other processing of media data.
  • media tags can be used to easily retrieve relevant media segments, text tags of media data, title information, etc.
  • Media retrieval is of great significance for media recommendation and media processing.
  • the requirements for media retrieval capabilities are also getting higher and higher.
  • the demand for retrieval results is not limited to media data with a modality type of text, but also requires the retrieval of media data with a modality type of non-text type (such as video type, audio type, image type, etc.). That is, in the media data retrieval business, the requirements for multi-modal retrieval performance are getting higher and higher.
  • a unified model will be used for understanding and reasoning. Specifically, the features of each modality will be extracted for each modality, and then the multimodal information of the features of all modalities will be fused to finally obtain multimodal information.
  • the related technologies are training the model, in order to improve the accuracy of multimodal information, the individual information of the single modality will be eliminated, and the common information between different modalities will be extracted, which will seriously affect the accuracy of the model's understanding of the single modal data, and thus seriously affect the accuracy of the processing results of the single modal data in subsequent tasks (such as retrieval tasks).
  • the embodiments of the present application provide a data processing method, apparatus, device, and readable storage medium, which can improve the versatility of the modal characterization model and the accuracy of the modal characterization results in the modal characterization business.
  • an embodiment of the present application provides a data processing method, including:
  • the training task group set includes a training task group S i , where i is a positive integer; the modality type to which the training task group S i belongs is a single modality type or a cross-modality type; when the modality type to which the training task group S i belongs is the single modality type, the training task group S i includes one sample media data; when the modality type to which the training task group S i belongs is the cross-modality type, the training task group S i includes sample media data of at least two different modality types;
  • attention interaction processing is performed on the training task group Si to obtain the attention representation vector corresponding to the training task group Si ; the attention interaction processing is used to fully interact different elements in the training task group Si ;
  • a target routing layer matching the modal type to which the training task group S i belongs is determined, and feature prediction processing is performed on the attention representation vector corresponding to the training task group S i based on the target routing layer to obtain a predicted modal representation vector corresponding to the training task group S i ;
  • the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group of the modality type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group of the cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group of the single-modality type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group of the cross-modal type.
  • An embodiment of the present application provides a data processing device, including:
  • a set acquisition module used to acquire a training task group set;
  • the training task group set includes a training task group S i , where i is a positive integer;
  • the modality type to which the training task group S i belongs is a single modality type or a cross-modality type; when the modality type to which the training task group S i belongs is the single modality type, the training task group S i includes one sample media data; when the modality type to which the training task group S i belongs is the cross-modality type, the training task group S i includes sample media data of at least two different modality types;
  • the attention processing module is used to perform attention interaction processing on the training task group S i in the modal representation model to obtain the attention representation vector corresponding to the training task group S i ; the attention interaction processing is used to fully interact different elements in the training task group S i ;
  • a routing layer determination module is used to determine a target routing layer that matches the modality type to which the training task group Si belongs, among the single-modality routing layers and the cross-modality routing layers included in the modality representation model;
  • a prediction module is used to perform feature prediction processing on the attention representation vector corresponding to the training task group S i based on the target routing layer to obtain a predicted modal representation vector corresponding to the training task group S i ;
  • a first optimization module is used to optimize the single-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type when determining the predicted modal representation vector corresponding to each training task group in the training task group set;
  • the second optimization module is used to optimize the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
  • the specific implementation method of the set acquisition module acquiring the training task group set includes:
  • N is a positive integer
  • a modality type to which a sample media data among the N media data belongs is any one of a first modality type and a second modality type; and the first modality type and the second modality type are different;
  • a task group construction process is performed on the N sample media data to obtain a training task group set.
  • the first sample media data refers to any sample media data of the N sample media data whose modality type is the first modality type;
  • the second sample media data refers to any sample media data among the N sample media data whose modality type is the second modality type;
  • the first training task group and the second training task group are both determined as training task groups whose modality type is a single modality type, and the set consisting of the training task group whose modality type is the cross-modality type and the training task group whose modality type is the single modality type is determined as a training task group set.
  • the number of the first sample media data and the number of the second sample media data are both one or more; the one or more first sample media data include the first sample media data M j , where j is a positive integer; the training task group of the modality type is a cross-modality type, and includes the training task group corresponding to the first sample media data M j ;
  • the specific implementation method of the set acquisition module performing cross-modal combination processing on the first sample media data and the second sample media data based on the media source channels respectively corresponding to the first sample media data and the second sample media data to obtain a training task group whose modality type is a cross-modal type includes:
  • the first sample media data, the associated sample media data, the first identifier and the second identifier are combined to obtain a training task group corresponding to the first sample media data M j .
  • the attention processing module performs attention interaction processing on the training task group Si in the modality representation model to obtain the specific implementation method of the attention representation vector corresponding to the training task group Si , including:
  • feature extraction processing is performed on the sample media data contained in the training task group Si through the feature extraction network layer to obtain the media features corresponding to the training task group Si ;
  • multi-head self-attention processing is performed on the media features corresponding to the training task group Si to obtain the attention representation vector corresponding to the training task group Si .
  • the multi-head self-attention network layer in the modality representation model includes Q self-attention sub-network layers; Q is a positive integer; the Q self-attention sub-network layers include a self-attention sub-network layer V k , k is a positive integer;
  • the attention processing module performs multi-head self-attention processing on the media features corresponding to the training task group Si through the multi-head self-attention network layer in the modality representation model to obtain the specific implementation method of the attention representation vector corresponding to the training task group Si , including:
  • the linear transformation matrix is feature integrated to obtain the attention representation sub-vector corresponding to the self-attention sub-network layer Vk ;
  • the Q attention representation sub-vectors are fused to obtain the attention representation vector corresponding to the training task group Si .
  • the single modality type includes a first modality type and a second modality type
  • the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type
  • the routing layer determination module determines the specific implementation method of the target routing layer that matches the modality type to which the training task group Si belongs in the single-modality routing layer and the cross-modality routing layer included in the modality representation model, including:
  • the cross-modality routing layer included in the modality representation model is determined as a target routing layer that matches the modality type to which the training task group S i belongs;
  • the first modality sub-routing layer corresponding to the first modality type is determined as the target routing layer that matches the modality type to which the training task group S i belongs;
  • the second modality sub-routing layer corresponding to the second modality type is determined as the target routing layer that matches the modality type to which the training task group Si belongs.
  • the single modality type includes a first modality type and a second modality type
  • the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type
  • the specific implementation method of the first optimization module optimizing the single-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type includes:
  • the first modality sub-routing layer corresponding to the first modality type is optimized based on the first loss value
  • the second modality sub-routing layer corresponding to the second modality type is optimized based on the second loss value.
  • the first modality type is a text modality type
  • the first modality training task group is a text word sequence
  • the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence
  • the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence
  • the first optimization module performs error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the real modal representation vector corresponding to the first modal training task group to obtain a specific implementation method of the first loss value corresponding to the first modal training task group, including:
  • the masked text word refers to a text word that has been masked
  • the first feature similarity is determined as a first loss value corresponding to the first modality training task group.
  • the specific implementation method of the second optimization module optimizing the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type includes:
  • each training task group whose modality type is a cross-modal type as a cross-modal training task group and determine the third feature similarity between the training task group S j and the remaining cross-modal training task groups in at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the cross-modal training task groups in at least two cross-modal training task groups except the training task group S j ;
  • the cross-modal routing layer is optimized.
  • the data processing device may further include:
  • a task group acquisition module used to acquire a target task group for describing the media data to be classified; the target task group includes at least two media data of different modal types;
  • An interaction processing module is used to perform attention interaction processing on the target task group in the modality representation model to obtain an attention representation vector corresponding to the target task group;
  • the category recognition module is used to perform category recognition processing on the predicted modal representation vector corresponding to the target task group to obtain the media category to which the media data to be classified belongs; the media category to which the media data to be classified belongs is used to archive the media data to be classified.
  • An embodiment of the present application provides a computer device, including: a processor and a memory;
  • the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.
  • an embodiment of the present application provides a computer-readable storage medium, which stores a computer program.
  • the computer program includes program instructions.
  • the program instructions are executed by a processor, the method in the embodiment of the present application is executed.
  • a computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in one aspect of the embodiments of the present application.
  • a general modal representation model which can serve both data whose modal type is a single modal type and data whose modal type is a cross-modal type.
  • the present application can use a set of training task groups including single modal type and cross-modal type to train the modal representation model, wherein, in the modal representation model, a single modal routing layer and a cross-modal routing layer are included, the single modal routing layer can perform feature prediction processing on data whose modal type is a single modal type (attention representation vector), and the cross-modal routing layer can perform feature prediction processing on data whose modal type is a cross-modal type (attention representation vector), and during training optimization, the single modal routing layer can be optimized based on the predicted modal representation vector corresponding to the training task group whose modal type is the single modal type, and the cross-modal routing layer can be optimized by the predicted modal representation vector corresponding to the training task group whose modal type is the cross-modal type.
  • the single-modal routing layer and the cross-modal routing layer in the modal characterization model can be trained and optimized at the same time, and during the training and optimization, the training and optimization are performed separately based on data of different modal types, without losing any modal type of data, and the performance of the single-modal routing layer and the cross-modal routing layer can be maintained at the same time, thereby improving the versatility and flexibility of the modal characterization model for single-modal type and cross-modal type.
  • the present application can set a routing distribution mechanism in the modal characterization model, divide the routing layer into a single-modal routing layer and a cross-modal routing layer, and different routing layers can process data of different modal types. During training, the data of different modal types are also trained and optimized separately.
  • Each routing layer will only retain the relevant information of the current modal type, so there is no need to discard any modal type of data, and the cross-modal type and single-modal type of characterization vectors can be output at the same time, which can well improve the accuracy of the characterization results of data of various modal types.
  • the present application can improve the versatility of the modal characterization model and the accuracy of the modal characterization results in the modal characterization business.
  • FIG1 is a network architecture diagram of a data processing system provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of a transformer architecture provided in an embodiment of the present application.
  • FIG4 is a schematic diagram of an architecture of a modal routing distribution provided in an embodiment of the present application.
  • FIG6 is a schematic diagram of a process for optimizing a single-mode routing layer provided in an embodiment of the present application.
  • FIG7 is a schematic diagram of a process for optimizing a cross-modal routing layer provided in an embodiment of the present application.
  • FIG8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application.
  • FIG. 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application.
  • Artificial Intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
  • Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies.
  • the basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, mechatronics and other technologies.
  • Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.
  • artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
  • the solution provided in the embodiment of the present application belongs to the machine learning (ML) and natural language processing (NLP) technologies under the field of artificial intelligence.
  • ML machine learning
  • NLP natural language processing
  • Machine Learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.
  • Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence.
  • Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
  • Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.
  • Figure 1 is a network architecture diagram of a data processing system provided in an embodiment of the present application.
  • the network architecture may include a business server 1000 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, and the number of terminal devices will not be limited here.
  • multiple terminal devices may include terminal devices 100a, terminal devices 100b, terminal devices 100c, ..., terminal devices 100n; as shown in Figure 1, terminal devices 100a, terminal devices 100b, terminal devices 100c, ..., terminal devices 100n can be connected to the business server 1000 respectively through the network, so that each terminal device can exchange data with the business server 1000 through the network connection.
  • any terminal device in the terminal device cluster 100 may refer to an intelligent device running an operating system, and the embodiment of the present application does not specifically limit the operating system of the terminal device.
  • an application such as a video application, a browser application, a game application, an educational application, etc. is running in a terminal device (such as the terminal device 100a), and examples of the applications will not be given one by one here.
  • the object can run a video application in a terminal device, and the video application can provide media data (such as text data, video data, audio data, image data, etc.) to the object in the form of feeds (feed can refer to the source of the message, and feed can also be translated as source material, feed, information provision, feed, summary, source, news subscription. It is a data format through which the application transmits the latest information to the object, usually arranged in a timeline manner), and the object can browse these media data in the video application.
  • a retrieval function can also be provided to the object.
  • the object can retrieve the media data (such as text data, image data, audio data, video data, etc.) associated with the input information by inputting certain information.
  • the video application can output another text data associated with the text data, and can also output image data, video data, audio data, etc. associated with the text data; or, after the object inputs certain image data, the video application can output another image data associated with the image data, and can also output video data, text data, audio data, etc. associated with the image data.
  • image data, text data, audio data and video data they can be regarded as media data of different modal types.
  • image data it can be understood as media data with a modal type of image modal type
  • video data it can be understood as media data with a modal type of video modal type
  • video applications can provide a retrieval function for media data of multiple modal types.
  • the present application provides a multimodal joint training framework based on a modal routing distribution mechanism.
  • cross-modal refers to a modal combination of two or more modalities
  • single-modal modal representation feature determination tasks are completed at the same time.
  • modal representation features of various modalities which may include cross-modal modal representation features and single-modal modal representation features
  • media data of different modal types can be retrieved, thereby reducing problems such as large model storage, time-consuming retrieval and repeated calculations, so that single-modal and cross-modal tasks can promote each other, respectively maintain good performance, and improve versatility and accuracy.
  • a modal representation model can be deployed in the business server 1000, and the business server 1000 can collect the information input by the object in the video application (such as input text data). Based on the information input by the object in the video application, the business server 1000 can perform reasoning analysis on the information input by the object through the modal representation model to obtain a modal representation vector of the input information. Based on the modal representation vector of the input information, the business server 1000 can determine the media data associated with the input information (such as text data, image data, video data, audio data, etc.).
  • the media data associated with the input information such as text data, image data, video data, audio data, etc.
  • the modal representation model deployed in the business server 1000 can include different modal routing layers, for example, it can specifically include a text modal routing layer for routing text data, an image modal routing layer for routing image data, an audio modal routing layer for routing audio data, and a cross-modal routing layer for routing multimodal media data. Then, for input information of different modal types, the corresponding modal routing layer can perform corresponding routing processing (the routing layer here can refer to the fully connected layer, and the routing processing can refer to the linear transformation processing through the fully connected layer. Since the linear transformation processing through the fully connected layer actually refers to the feature integration prediction processing through the fully connected layer, the routing processing here can also be understood as the feature prediction processing) to obtain the modal representation vector of the input information. For example, when the input information is text data, it can be routed by the text modal routing layer, and when the input information is image data, it can be routed by the image modal routing layer.
  • the present application sets up a routing distribution mechanism, and sets up different modal routing layers for different modal types.
  • Input information of different modal types can be distributed to the corresponding modal routing layer for routing processing.
  • the way of setting up a routing distribution mechanism can improve the targetedness of the routing layer.
  • Input data of one modal type can be targetedly routed by the corresponding modal routing layer, and different modal routing layers can be adapted to input data of various modal types. This can greatly improve the versatility of the modal representation model for different modal types.
  • the modal representation model in order to improve the accuracy of the output results of the modal representation model, the modal representation model can be trained and optimized.
  • training samples of different modal types can be used for targeted training.
  • text sample data can be used for training
  • image sample data can be used for training.
  • cross-modality sample data (such as sample data containing both image sample data and text sample data) can be used for training.
  • this application can be referred to as a single modality routing layer, and for the routing layer for multiple modality types, this application can be referred to as a cross-modality routing layer, and the cross-modality routing layer can process multi-modality data.
  • the single-modality routing layer it can be trained by training data (training task group) whose modality type is a single modality type (such as text modality type, image modality type, audio modality type, etc.), and for the cross-modality routing layer, it can be trained by training data (training task group) whose modality type is a cross-modality type.
  • the present application can configure a set of training task groups. For example, a sample media data and its corresponding modal type can be combined into a training task group, or two sample media data of different modal types and their corresponding modal types can be combined into a training task group. Based on this, by combining the sample media data and the modal type, a set of training task groups including single modal types and cross-modal types can be obtained. For example, when the sample media data is sample text data 1, its modal type is text modal type, then after combining them, a training task group of ⁇ sample text data 1, text modal type> can be obtained.
  • the training task group only contains one text data, and the modal type of the training task group is text modal type (single modal type); for another example, when the sample media data is sample image data 1, its modal type is image modal type, then after combining them, a training task group of ⁇ sample image data 1, image modal type> can be obtained.
  • the training task group contains only one text data, and the modality type to which the training task group belongs is an image modality type (single modality type); for another example, when the sample media data is sample text data 1 and sample image data 1, its modality type includes text modality type and image modality type, then after combining them, a training task group can be obtained as ⁇ sample text data 1-sample image data 1, text modality type-image modality type>, which contains both a text data and an image data, and the modality type to which the training task group belongs is a cross-modality type.
  • the training task group in the present application may include a single-modality type training task group and a cross-modality type training task group.
  • a single-modality type training task group it will only include sample media data of one modality type, which can be a text modality type, an image modality type, an audio modality type, etc.; for a cross-modality type training task group, it needs to include sample media data of two or more different modality types.
  • the modal representation model can be called to perform attention interaction processing on it, where the attention interaction processing here can refer to multi-head self-attention processing based on the multi-head self-attention mechanism. Since the multi-head self-attention processing can make each element in the input data fully interact with other elements, the multi-head self-attention processing is called attention interaction processing in this application, and the result obtained by the attention interaction processing is called the attention representation vector corresponding to the training task group. Subsequently, for each attention representation vector, based on the modality type to which the training task group belongs, a target routing layer that matches it can be determined in the single-modal routing layer and the cross-modal routing layer contained in the modality representation model.
  • the attention representation vector of the training task group can be routed (feature prediction processing), and finally the modality representation vector corresponding to the training task group can be obtained (the modality representation vector corresponding to the training task group refers to the feature vector output by the target routing layer for characterizing the modality type.
  • the modality representation model needs to be trained in the training phase, the modality representation vector output by the target routing layer can be understood as a predicted value, and the output modality representation vector can be called a predicted modality representation vector.
  • the target routing layer needs to be trained and optimized based on the predicted modality representation vector, so that the modality representation vector output by the modality representation model becomes more and more accurate).
  • the text modality routing layer can be determined as the target routing layer of the training task group in the unimodal routing layer contained in the modality representation model, and the predicted modality representation vector corresponding to the training task group can be output based on the text modality routing layer, and based on the predicted modality representation vector corresponding to the training task group, the text modality routing layer can be trained and optimized accordingly.
  • the embodiments of the present application provide a multimodal joint training framework based on a modal routing distribution mechanism.
  • On the basis of training only one modal representation model it can simultaneously complete the tasks of outputting modal representation vectors of cross-modality and unimodality.
  • Based on the modal representation features of each modality media data of different modal types can be retrieved. This can reduce problems such as large model storage, time-consuming retrieval, and repeated calculations, so that unimodal and cross-modal tasks can promote each other, maintain good performance respectively, and improve versatility and accuracy.
  • the method provided in the embodiment of the present application can be executed by a computer device, and the computer device includes but is not limited to the terminal device or business server mentioned in Figure 1.
  • Figure 2 is a flow chart of a data processing method provided by an embodiment of the present application.
  • the method can be executed by a computer device, and the computer device can be a terminal device (for example, any terminal device in the terminal device cluster shown in Figure 1 above, such as terminal device 100a), and the computer device can also be a server (such as the business server 1000 in the embodiment corresponding to Figure 1 above), and the computer device can also be a terminal device and a server.
  • this embodiment is explained by taking the method being executed by a terminal device as an example.
  • the data processing method can at least include the following steps S101-step S104:
  • Step S101 obtain a training task group set;
  • the training task group set includes a training task group S i , where i is a positive integer;
  • the modality type to which the training task group S i belongs is a single modality type or a cross-modality type;
  • the training task group S i contains one sample media data;
  • the training task group S i contains sample media data of at least two different modality types.
  • the training task group contains only one sample media data, and the modality type to which the sample media data belongs is a text modality type (i.e., the sample media data is sample text data), then the modality type to which the training task group belongs may be a single modality type, and may specifically be a text modality type; for another example, if the training task group contains only one sample media data, and the modality type to which the sample media data belongs is an image modality type (i.e., the sample media data is sample image data), then the modality to which the training task group belongs may be a single modality type, and may specifically be an image modality type.
  • the modality type to which the training task group belongs may be a single modality type, and the specific modality type is determined based on the modality type of the sample media data contained.
  • the modal type to which the training task group belongs is a cross-modal type (or multi-modal type).
  • a training task group contains both sample text data (sample media data with a modal type of text modal type) and sample video data (sample media data with a modal type of video modal type), then it can be determined that the modal type to which the training task group belongs is a cross-modal type.
  • the sample media data in the present application may refer to media data serving as training samples, which may specifically include sample text data, sample image data, sample audio data or sample video data.
  • a training task group is composed of sample media data of one modality type, it can be determined that the modality type to which the training task group belongs is a single modality type, and the modality type to which the sample media data belongs can be determined as the modality type to which the training task group belongs; and when a training task group is composed of sample media data of multiple (two or more) modality types, it can be determined that the modality type to which the training task group belongs is a cross-modality type or a multi-modality type.
  • a specific implementation method for obtaining the training task group set may be: N (N is a positive integer) sample media data may be obtained; Among them, the modal type to which a sample media data among N media data belongs is any one of the first modal type and the second modal type; and the first modal type is different from the second modal type; based on the modal type to which each sample media data among the N sample media data belongs, the N sample media data can be processed by task group construction, thereby obtaining a training task group set.
  • a plurality of sample media data of different modality types are pre-acquired as samples, and then a task group construction process is performed based on the modality type to which each sample media data belongs, thereby obtaining a training task group containing different sample media data, and then obtaining a training task group set.
  • a specific implementation method for constructing a task group for the N sample media data to obtain a training task group set may be as follows: the first sample media data of the N sample media data may be combined with a first identifier for representing the first modal type, thereby obtaining a first training task group whose modal type is the first modal type; wherein the first sample media data refers to any sample media data of the N sample media data whose modal type is the first modal type; it should be understood that the N sample media data include media data of the first modal type and a first identifier for representing the first modal type.
  • the second training task group of the two modal types wherein the second sample media data refers to any sample media data of the N sample media data whose modal type is the second modal type; it should be understood that the N sample media data include media data of the first modal type and media data of the second modal type, and any media data whose modal type is the second modal type can be referred to as the second sample media data. Then, for any second sample media data, it can be combined with the identifier of the second modal type, thereby obtaining a training task group (this training task group can be referred to as the second training task group).
  • the N sample media data may include sample text data 1 (modality type is a text modality type), sample text data 2 (modality type is a text modality type), sample image data 1 (modality type is an image modality type), sample video data 1 (modality type is a video modality type), and sample video data 2 (modality type is a video modality type).
  • the identifier used to characterize the text modality type can be "text”
  • the identifier used to characterize the image modality type can be "vision”
  • the identifier used to characterize the video modality type can be "video”.
  • sample text data 1 and the identifier "text" used to characterize the text modality type can be combined to obtain a training task group of ⁇ sample text data 1, text>, or the sample text data 1 and the identifier "text” used to characterize the text modality type can be combined to obtain a training task group of ⁇ sample text data 2, text>.
  • the training task groups ⁇ sample text data 1, text> and ⁇ sample text data 2, text> only contain sample media data of the text modality type
  • the modality type to which these two training task groups belong can be a single modality type, and the modality type to which they belong can specifically be a text modality type; similarly, the sample image data 1 can be combined with the identifier "vision" used to characterize the image modality type, to obtain a training task group of ⁇ sample image data 1, vision>.
  • the modality type to which this training task group belongs can be a single modality type, and the modality type to which it belongs can be specifically an image modality type; similarly, the sample video data 1 and the sample video data 2 can be respectively combined with the identifier "video" used to characterize the video modality type, thereby obtaining the training task groups ⁇ sample video data 1, video> and ⁇ sample video data 2, video>. Both of these training task groups only contain sample media data of the video modality type, then the modality type to which this training task group belongs can be a single modality type, and the modality type to which it belongs can be specifically a video modality type.
  • sample media data of different modal types with the same media source channel may be combined.
  • the media source channel may refer to a method or location for obtaining the sample media data
  • the one or more first sample media data include the first sample media data M j (j is a positive integer)
  • the modality type is a cross-modal training task group
  • the training task group corresponding to the first sample media data M j is taken as an example, based on the media source channels corresponding to the first sample media data and the second sample media data respectively, the first sample media data and the second sample media data are cross-modally combined and processed to obtain the training task group whose modality type is a cross-modal training task group.
  • the specific implementation method may be as follows: the media source channel corresponding to the first sample media data M j may be determined as the target media source channel; then, the second sample media data whose media source channel is the target media source channel in one or more second sample media data may be determined as the associated sample media data corresponding to the first sample media data M j ; finally, the first sample media data, the associated sample media data, the first identifier and the second identifier may be combined to obtain the training task group corresponding to the first sample media data M j .
  • sample media data include sample text data a, sample text data b, sample image data a, sample video data h1, and sample video data h2, and the sample text data a and the sample image data a both come from the sample video data h1 (the sample text data a is the video description information of the sample video data h1, and the sample image data a is a certain picture frame of the sample video data h1)
  • the sample text data a and the sample image data a can be determined to be mutually related (the sample text data a is the associated media data of the sample image data a, and similarly, the sample image data a is also the associated media data of the sample text data a), the sample text data a and the sample video data h1 are also mutually related, and the sample image data a and the sample video data h1 are also mutually related.
  • sample text data a and the sample image data a and their corresponding identifiers can be identified.
  • the sample text data a can be combined with the sample video data h1 and its corresponding identifier (the identifier "text" for representing the text modality type and the identifier "video” for representing the video modality type) to obtain a training task group of ⁇ sample text data a-sample video data h1, text-video>.
  • the sample image data a can be combined with the sample video data h1 and its corresponding identifier (the identifier "vision” for representing the image modality type and the identifier "video” for representing the video modality type) to obtain a training task group of ⁇ sample image data a-sample video data h1, vision-video>. It can be seen that a training task group whose modality type is a cross-modal type should include two or more sample media data and identifiers of the modality type to which each sample media data belongs.
  • the identifier for characterizing the modal type it can be clearly indicated whether the modal type contained in the training task group is a single modal type or a cross-modal type, and further, it can be reflected whether the modal type to which the training task group belongs is a single modal type or a cross-modal type.
  • the training task group S i in the present application may refer to any training task group in the training task group set, and for the training task group whose modal type in the training task group set is a single modal type, it is mainly composed of a modal
  • the set of training task groups is composed of sample media data of a single modality type (such as a text modality type, an image modality type, a video modality type or an audio modality type), and the number of the sample media data is one
  • the training task group may only include one sample media data
  • the training task group may only include one sample media data; and for a training task group in the set of training task groups whose modality type is a cross-modality type, it needs to be composed of sample media data of at least two different modality types, and the number of sample media data under each modality type needs to be at least one, then for a certain training task group in the set of training task groups
  • the modal representation model may refer to a model based on the transformer architecture.
  • feature extraction modeling processing can be performed on each training task group.
  • the elements of each feature level in the training task group can fully interact, so that each element can pay more attention to the elements that are more closely related to itself, so the feature extraction modeling processing can also be called attention interaction processing.
  • the entire network structure of the transformer architecture is completely composed of the attention mechanism (Attention mechanism). More precisely, the transformer architecture consists of and only consists of self-Attenion and feedforward neural network (Feed Forward Neural Network, FFN).
  • the trainable neural network based on the transformer can be built in the form of stacked transformers.
  • the use of the Attention mechanism can solve the problem of forgetting the computational context of traditional RNN (or LSTM, GRU, etc.).
  • Figure 3 is a schematic diagram of the architecture of a transformer architecture provided in an embodiment of the present application.
  • the transformer architecture includes an encoder and a decoder.
  • the encoder may include a multi-head attention layer and a routing layer.
  • the multi-head attention layer can be used to process the input features.
  • the routing layer in the encoder can be used to route the content processed by the multi-head attention layer.
  • the routing layer here can refer to an FFN, specifically a multi-layer fully connected layer.
  • the routing layer can be used to fully connect the content processed by the multi-head attention layer (feature integration processing, also referred to as routing processing).
  • feature integration processing also referred to as routing processing.
  • the features encoded by the encoder i.e., the content output by the routing layer
  • the decoder can decode the features encoded by the encoder.
  • the content decoded and output by the decoder can be used as the modal representation vector corresponding to the input features.
  • the modal representation model may include a multi-head attention layer, which may be determined as a multi-head self-attention network layer in the present application, and the attention interaction processing of the training task group S i in the modal representation model here can be regarded as the attention interaction processing of the training task group S i in the multi-head self-attention network layer.
  • the specific implementation method may be: in the modal representation model, the sample media data contained in the training task group S i may be subjected to feature extraction processing through the feature extraction network layer, thereby obtaining the media features corresponding to the training task group S i ; further, the media features corresponding to the training task group S i may be subjected to multi-head self-attention processing through the multi-head self-attention network layer in the modal representation model, thereby obtaining the attention representation vector corresponding to the training task group S i .
  • the feature extraction network layer here can refer to the embedding layer, through which the features of the sample media data in the training task group can be extracted to obtain the media features.
  • the media features here can be determined based on the modality type to which the sample media data belongs.
  • the media features of the sample media data can refer to the word features corresponding to each text word; when the sample media data is of image modality type, the media features of the sample media data can refer to pixel features (an image can be divided into multiple pixel networks, and the media features can be composed of pixel features corresponding to each pixel grid); when the sample media data is of audio modality type, the media features of the sample media data can refer to phoneme features; when the sample media data is of video modality type, the media features of the sample media data can be composed of frame features corresponding to video frames.
  • the media features of the training task group can be composed of the media features of sample media data of different modality types, that is, it can be composed of two or more features of word features, pixel features, phoneme features, and frame features.
  • the multi-head self-attention network layer may include Q (Q is a positive integer) self-attention sub-network layers, and the output result of the multi-head self-attention network layer may be obtained by merging the output results of these self-attention sub-network layers.
  • Q is a positive integer
  • the self-attention sub-network layer including the self-attention sub-network layer V k (k is a positive integer) as an example
  • multi-head self-attention processing is performed on the media features corresponding to the training task group S i
  • the specific implementation method of obtaining the attention representation vector corresponding to the training task group S i can be as follows: the attention parameter matrix contained in the self-attention sub-network layer V k can be obtained, and then the attention parameter matrix contained in the self-attention sub-network layer V k can be operated and processed with the media features corresponding to the training task group S i , thereby obtaining the linear transformation matrix corresponding to the attention parameter matrix; further, the linear transformation matrix can be
  • the included attention parameter matrix may include W Q , W K and W V matrices, and the media features corresponding to the training task group S i may be multiplied with W Q , W K and W V respectively, so as to obtain the linear transformation matrix corresponding to the W Q matrix, the linear transformation matrix corresponding to the W K matrix and the linear transformation matrix corresponding to the W V matrix.
  • the linear transformation matrix W Q and the linear transformation matrix W K may be multiplied, and the result of the multiplication operation of the two may be input to a fully connected layer (such as a Softmax layer) for full connection processing (that is, feature integration processing), and the result obtained by the full connection processing may be multiplied with the linear transformation matrix W V again, and the result obtained by the multiplication operation with the linear transformation matrix W V again may be used as the output result of a self-attention sub-network layer (that is, the attention representation sub-vector).
  • a self-attention sub-network layer contains different W Q , W K and W V matrices. Using the same principle, different attention representation sub-vectors can be obtained. Finally, these different attention representation sub-vectors can be fused to obtain the final attention representation vector.
  • E attn can be used to represent the final attention representation vector; It can be used to represent the attention representation sub-vector of the self-attention sub-network layer Hn; Concat() can be used to represent the splicing process.
  • the various attention representation sub-vectors can be fused (spliced), and the attention representation vector can be obtained after fusion (splicing).
  • Step S103 in the single-modal routing layer and the cross-modal routing layer included in the modal representation model, determine the target routing layer that matches the modality type to which the training task group S i belongs, and perform feature prediction processing on the attention representation vector corresponding to the training task group S i based on the target routing layer to obtain the predicted modal representation vector corresponding to the training task group S i .
  • the encoder and decoder included in the modal representation model will include a multi-head attention layer and a routing layer.
  • the routing layer can perform feature integration processing (i.e., routing processing, which is referred to as feature prediction processing in this application) on the attention representation vector output by the multi-head attention layer to obtain the final modal representation vector.
  • the modal representation vector refers to the features obtained after reasoning and understanding the training task group.
  • the present application can set the routing layer in the modal representation model to different modal routing layers, and connect each modal routing layer to different modal types.
  • the text modality routing layer, image modality routing layer, audio modality routing layer, and video modality routing layer can all be understood as a single modality routing layer. That is to say, in the modality representation model, a single modality routing layer and a cross-modality routing layer can be included at the same time.
  • the modality routing layer that matches the modality type to which the training task group S i belongs can be determined as the target routing layer in the single modality routing layer and the cross-modality routing layer included in the modality representation model, and then the attention representation vector corresponding to the training task group S i is feature predicted based on the target routing layer to obtain the modality representation vector (predicted modality representation vector) corresponding to the training task group S i .
  • a specific implementation method for determining a target routing layer that matches the modality type to which the training task group S i belongs in the single modality routing layer and the cross-modality routing layer included in the modality representation model may be as follows: if the modality type to which the training task group S i belongs is a cross-modality type, the cross-modality routing layer included in the modality representation model may be determined as a target routing layer that matches the modality type to which the training task group S i belongs; if the modality type to which the training task group S i belongs is the first modality type, the first modality sub-routing layer corresponding to the first modality type may be determined as a target routing layer that matches the modality type to which the
  • the first modality type and the second modality type may refer to different modality types.
  • the first modality type may refer to one or more of a text modality type, an image modality type, an audio modality type, and a video modality type
  • the second modality type may refer to the remaining modality types among the text modality type, the image modality type, the audio modality type, and the video modality type except the first modality type.
  • the text modality routing layer can be determined as the target routing layer corresponding to the training task group S i ;
  • the modality type to which the training task group S i belongs is an image modality type, the image modality routing layer can be determined as the target routing layer corresponding to the training task group S i;
  • the audio modality routing layer can be determined as the target routing layer corresponding to the training task group S i ;
  • the video modality routing layer can be determined as the target routing layer corresponding to the training task group S i.
  • i corresponds to the target routing layer;
  • the modality type to which the training task group S i belongs is a cross-modal type, the cross-modal routing layer can be determined as the target routing layer corresponding to the training task group S i .
  • the single-modal routing layer can specifically be a text modal routing layer, an image modal routing layer, an audio modal routing layer, or a video modal routing layer, which can be used to perform routing processing on data of the corresponding modal type.
  • the input feature is a media feature corresponding to the text modality type
  • the attention representation vector after the attention representation vector is output through the multi-head attention layer, it can be distributed to the text modality routing layer, and the text modality routing layer performs routing processing on it
  • the input feature when the input feature is a media feature corresponding to the image modality type, after the attention representation vector is output through the multi-head attention layer, it can be distributed to the image modality routing layer, and the image modality routing layer performs routing processing on it
  • the input feature is a media feature corresponding to the audio modality type
  • after the attention representation vector after the attention representation vector is output through the multi-head attention layer, it can be distributed to the audio modality routing layer, and the audio modality routing layer performs routing processing on it
  • the input feature when the input feature is a media feature corresponding to the video modality type, after the attention representation vector is output through the multi-head attention layer, it can be distributed to the video modality routing layer, and the video modality routing layer performs routing processing on it;
  • Step S104 when the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
  • the corresponding modal routing layer can also be trained and optimized based on the predicted modal representation vectors corresponding to the training task groups of the corresponding modal types.
  • the single-modal routing layer can be optimized based on the predicted modal representation vectors corresponding to the training task groups whose modal types are single-modal types
  • the cross-modal routing layer can be optimized based on the predicted modal representation vectors corresponding to the training task groups whose modal types are cross-modal types.
  • the optimized single-modal routing layer can be used to perform feature prediction processing on the task groups whose modal types are single-modal types; the optimized cross-modal routing layer can be used to perform feature prediction processing on the task groups whose modal types are cross-modal types.
  • the text modal routing layer can be trained and optimized, and based on the predicted modal representation vectors corresponding to the training task groups whose modal types are image modal types, the image modal routing layer can be trained and optimized.
  • the layers are trained and optimized. Based on the predicted modal representation vector corresponding to the training task group whose modality type is the audio modality type, the audio modality routing layer can be trained and optimized.
  • the video modality routing layer can be trained and optimized.
  • the text modality routing layer after training and optimization can perform feature prediction processing on text data.
  • the image modality routing layer after training and optimization can perform feature prediction processing on image data.
  • the audio modality routing layer after training and optimization can perform feature prediction processing on audio data.
  • the video modality routing layer after training and optimization can perform feature prediction processing on video data.
  • the target task group of the cross-modal type can be processed based on the optimized cross-modal routing layer.
  • a target task group for describing the media data to be classified can be obtained; wherein the target task group includes at least two media data of different modal types; then, the target task group can be subjected to attention interaction processing in the modal representation model, thereby obtaining the attention representation vector corresponding to the target task group; further, the attention representation vector corresponding to the target task group can be subjected to feature prediction processing in the optimized cross-modal routing layer, thereby obtaining the predicted modal representation vector corresponding to the target task group; then, the predicted modal representation vector corresponding to the target task group can be subjected to category recognition processing, thereby obtaining the media category to which the media data to be classified belongs.
  • the media data to be classified can be archived according to the media category to which it belongs.
  • the scenario of archiving media data is taken as an example here to illustrate the scenario in which the embodiment of the present application can be applied.
  • the media data to be classified is a certain video data
  • the text data and image data from the video data can be used as a task group, and then the cross-modal routing layer after the training optimization is used to infer and understand it to obtain an accurate modal representation vector of the video data.
  • the media category to which the video data belongs (such as dance category, competition category, gymnastics category, etc.) can be determined. Through the media category to which the video data belongs, the video data can be accurately archived.
  • the application scenario is not limited to this.
  • it can also be applied to the retrieval scenario of multimodal content, the scenario of media data recommendation, etc.
  • Figure 5 is a scene schematic diagram of media data retrieval provided in the embodiment of the present application.
  • the video data currently exposed to the object a is video data 300a (the creator of the video data 300a is an object named "La La Loves to Dance", and the video copy of the video data 300a is "Teach you to quickly learn a dance in three minutes").
  • the video display interface 3001 there are like controls, comment controls and sharing controls, which can be used for the object to like, comment and share the video data.
  • the terminal device may respond to the trigger operation by displaying an input box in the video display interface 3001 , and object a may input the content desired to be searched in the input box.
  • object a enters the text data "How to make braised crucian carp" in the input box, and the terminal device can obtain the text data "How to make braised crucian carp" entered by object a.
  • the terminal device can send the text data "How to make braised crucian carp" to the business server, and the business server can infer and understand the text data "How to make braised crucian carp” by training the optimized modal representation model.
  • the attention representation vector corresponding to the text data "How to make braised crucian carp” can be output, and the attention representation vector can be distributed to the text modal routing layer.
  • the business server can determine whether the two data are associated based on the similarity between different vectors. For example, for the modal representation vector corresponding to the above text data "How to make braised crucian carp", the modal representation vector of the candidate video data can be obtained first, and then the modal representation vector corresponding to the text data "How to make braised crucian carp" and the modal representation vector of each candidate video data are calculated for similarity. When the similarity is greater than the similarity threshold, the candidate video data can be determined to be the associated video data of the text data "How to make braised crucian carp".
  • the present application can adopt a set of training task groups including unimodal type and cross-modal type to train the modal representation model, wherein, in the modal representation model, a unimodal routing layer and a cross-modal routing layer are included, the unimodal routing layer can perform feature prediction processing on data (attention representation vector) of a unimodal type, and the cross-modal routing layer can perform feature prediction processing on data (attention representation vector) of a cross-modal type, and during training optimization, the unimodal routing layer can be optimized based on the predicted modal representation vector corresponding to the training task group of the unimodal type, and the cross-modal routing layer can be optimized by the predicted modal representation vector corresponding to the training task group of the cross-modal type.
  • the single-modal routing layer and the cross-modal routing layer in the modal characterization model can be trained and optimized at the same time, and during the training and optimization, the training and optimization are performed separately based on data of different modal types, without losing any modal type of data, and the performance of the single-modal routing layer and the cross-modal routing layer can be maintained at the same time, thereby improving the versatility and flexibility of the modal characterization model for single-modal type and cross-modal type.
  • the present application can set a routing distribution mechanism in the modal characterization model, divide the routing layer into a single-modal routing layer and a cross-modal routing layer, and different routing layers can process data of different modal types. During training, the data of different modal types are also trained and optimized separately.
  • Each routing layer will only retain the relevant information of the current modal type, so there is no need to discard any modal type of data, and the cross-modal type and single-modal type of characterization vectors can be output at the same time, which can well improve the accuracy of the characterization results of data of various modal types.
  • the present application can improve the versatility of the modal characterization model and the accuracy of the modal characterization results in the modal characterization business.
  • Figure 6 is a schematic diagram of a process for optimizing a unimodal routing layer provided in an embodiment of the present application.
  • the process may correspond to the embodiment corresponding to Figure 2 above, for the predicted modal representation vector corresponding to the training task group whose modality type is a unimodal type, to optimize the unimodal routing layer.
  • the process is a process described by taking the unimodal type including the first modal type and the second modal type, and the unimodal routing layer including the first modal sub-routing layer corresponding to the first modal type and the second modal sub-routing layer corresponding to the second modal type as an example.
  • the process may include at least the following steps S601-S605:
  • the training task group whose modality type is the first modality type can be named the first modality training task group
  • the training task group whose modality type is the second modality type can be named the second modality training task group.
  • Step S602 obtaining a true modal representation vector corresponding to the first modal training task group and a true modal representation vector corresponding to the second modal training task group.
  • Step S603 performing error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group.
  • the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group can be subjected to error calculation processing (that is, the error value between the predicted modal representation vector and the true modal representation vector is calculated), thereby obtaining the first loss value corresponding to the first modal training task group.
  • the first modality training task group can be a text word sequence
  • the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence
  • the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence.
  • the predicted modality representation vector corresponding to the first modality training task group and the real modality representation vector corresponding to the first modality training task group are subjected to error calculation processing to obtain the first loss value corresponding to the first modality training task group.
  • a masking strategy can be used to mask a certain text word.
  • a text word in a text word sequence that is a noun attribute, a verb attribute, or a position attribute can be masked.
  • the predicted modal representation vector output by the modal representation model includes the predicted representation feature corresponding to the masked text word, and the real modal representation vector also includes the real representation feature corresponding to the masked text word.
  • the similarity between the predicted representation feature and the real representation feature can be calculated, and the similarity between the predicted representation feature and the real representation feature can be used as the first loss value corresponding to the first modal training task group.
  • the first modality type is an image modality type or an audio modality type
  • some elements in the first modality training task group can also be masked.
  • the first modality training task group can be image data, and certain pixel areas in the image data can be masked (specifically, pixel areas with a high mutual intersection ratio can be masked).
  • the predicted modality representation vector output by the modality representation model includes predicted representation features corresponding to the masked pixel areas, and the real modality representation vector also includes real representation features corresponding to the masked pixel areas.
  • the similarity between the predicted representation features and the real representation features can also be calculated.
  • the similarity between the predicted representation features and the real representation features can be used as the first loss value corresponding to the first modality training task group.
  • the loss value between the predicted value and the true value may also be calculated by using feature regression or regional classification targets. This application does not limit the method of calculating the loss value based on the predicted value and the true value.
  • error calculation is performed on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain the second loss value corresponding to the second modal training task group.
  • the specific method can refer to the description of determining the first loss value above. The principles of the two are the same and will not be repeated here.
  • Step S605 Optimize the first modality sub-routing layer corresponding to the first modality type based on the first loss value, and optimize the second modality sub-routing layer corresponding to the second modality type based on the second loss value.
  • the first modal sub-routing layer corresponding to the first modal type can be optimized based on the first loss value
  • the second modal sub-routing layer corresponding to the second modal type can be optimized based on the second loss value.
  • the loss values obtained by the predicted values of the corresponding modal types can be used for optimization.
  • the parameters of each modal sub-routing layer can be adjusted based on the loss value until the loss value meets the convergence condition.
  • the convergence condition here can refer to the number of iterations reaching the iteration threshold (this iteration threshold can be set based on actual business needs), or it can refer to the loss value being less than the loss threshold.
  • the convergence condition can be set accordingly based on specific business needs. This application does not limit it, and it is specifically explained here.
  • Figure 7 is a schematic diagram of a process for optimizing a cross-modal routing layer provided in an embodiment of the present application.
  • the process may correspond to the embodiment corresponding to Figure 2 above, for the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type, the process for optimizing the cross-modal routing layer, the process is based on a training task group set, including at least two training task groups whose modality type is a cross-modal type, at least two training task groups whose modality type is a cross-modal type, including training task group Sj (j is a positive integer), training task group Sj includes a third sample media data whose modality type is a first modality type, and a fourth sample media data whose modality type is a second modality type.
  • the process is explained as an example.
  • the predicted modality corresponding to the training task group Sj is
  • the state representation vector may include the predicted representation feature corresponding to the third sample media data and the predicted representation feature corresponding to the fourth sample media data.
  • the process may include at least the following steps S701 to S703:
  • Step S701 Determine a second feature similarity between a predicted representation feature corresponding to the third sample media data and a predicted representation feature corresponding to the fourth sample media data.
  • the task of cross-modal retrieval is that after the object inputs certain information (such as text data), the platform can recall the image/video/audio content associated with it. Therefore, the core idea of cross-modal retrieval is to align and make consistent between different modalities, that is, the matching pictures and text content or audio content, video content are closest and similar in the feature space, and the unmatched pictures and text content or audio content, video content are inconsistent in the feature space. Based on this, for the training task group containing different modality types, it is necessary to determine the second feature similarity between the predicted representation feature corresponding to the third sample media data and the predicted representation feature corresponding to the fourth sample media data. The second feature similarity should be as large as possible, so that the matching text, image, audio, and video can be consistent in the feature space.
  • Step S702 each training task group whose modality type is a cross-modal type is determined as a cross-modal training task group, and the third feature similarity between the training task group S j and the remaining cross-modal training task groups in at least two cross-modal training task groups is determined; the remaining cross-modal training task groups refer to the cross-modal training task groups in at least two cross-modal training task groups except the training task group S j .
  • Step S703 Optimize the cross-modal routing layer based on the second feature similarity and the third feature similarity.
  • the second feature similarity in this application, it is necessary to maximize it by continuously adjusting the parameters of the cross-modal routing layer.
  • the third feature similarity it is necessary to minimize it by continuously adjusting the parameters of the cross-modal routing layer.
  • a routing distribution mechanism can be set in the modal representation model to divide the routing layer into a single-modal routing layer and a cross-modal routing layer.
  • Different routing layers can process data of different modal types.
  • data of different modal types are also trained and optimized separately.
  • Each routing layer will only retain relevant information of the current modal type. Therefore, there is no need to discard data of any modal type, and representation vectors of cross-modal and single-modal types can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types.
  • Figure 8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application.
  • the data processing device can be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the data processing device can be used to execute the method shown in Figure 2.
  • the data processing device 1 may include: a set acquisition module 11, an attention processing module 12, a routing layer determination module 13, a prediction module 14, a first optimization module 15 and a second optimization module 16.
  • the set acquisition module 11 is used to acquire a training task group set; the training task group set includes a training task group S i , where i is a positive integer; the modality type to which the training task group S i belongs is a single modality type or a cross-modality type; when the modality type to which the training task group S i belongs is a single modality type, the training task group S i includes one sample media data; when the modality type to which the training task group S i belongs is a cross-modality type, the training task group S i includes at least two sample media data of different modality types;
  • the attention processing module 12 is used to perform attention interaction processing on the training task group S i in the modality representation model to obtain the attention representation vector corresponding to the training task group S i ; the attention interaction processing is used to fully interact different elements in the training task group S i ;
  • a routing layer determination module 13 is used to determine a target routing layer that matches the modality type to which the training task group Si belongs, among the single-modality routing layers and the cross-modality routing layers included in the modality representation model;
  • a prediction module 14 is used to perform feature prediction processing on the attention representation vector corresponding to the training task group S i based on the target routing layer to obtain a predicted modality representation vector corresponding to the training task group S i ;
  • the first optimization module 15 is used to optimize the single-modal routing layer based on the prediction modal representation vector corresponding to the training task group whose modality type is a single-modal type when the prediction modal representation vector corresponding to each training task group in the training task group set is determined;
  • the second optimization module 16 is used to optimize the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
  • the specific implementation method of the set acquisition module 11 acquiring the training task group set includes:
  • N is a positive integer
  • a modality type to which a sample media data among the N media data belongs is any one of a first modality type and a second modality type; and the first modality type and the second modality type are different;
  • a task group construction process is performed on the N sample media data to obtain a training task group set.
  • the set acquisition module 11 performs task group construction processing on the N sample media data based on the modality type to which each sample media data belongs in the N sample media data, and obtains a specific implementation method of the training task group set, including:
  • the first sample media data refers to any sample media data of the N sample media data whose modality type is the first modality type;
  • the second sample media data refers to any sample media data among the N sample media data whose modality type is the second modality type;
  • the number of the first sample media data and the number of the second sample media data are both one or more; the one or more first sample media data include the first sample media data M j , where j is a positive integer; the training task group of the modality type is a cross-modality type, and includes the training task group corresponding to the first sample media data M j ;
  • the first sample media data, the associated sample media data, the first identifier and the second identifier are combined to obtain a training task group corresponding to the first sample media data M j .
  • the attention processing module 12 performs attention interaction processing on the training task group Si in the modality representation model to obtain the specific implementation method of the attention representation vector corresponding to the training task group Si , including:
  • multi-head self-attention processing is performed on the media features corresponding to the training task group Si to obtain the attention representation vector corresponding to the training task group Si .
  • the multi-head self-attention network layer in the modality representation model includes Q self-attention sub-network layers; Q is a positive integer; the Q self-attention sub-network layers include a self-attention sub-network layer V k , k is a positive integer;
  • the attention processing module 12 performs multi-head self-attention processing on the media features corresponding to the training task group Si through the multi-head self-attention network layer in the modality representation model to obtain the specific implementation method of the attention representation vector corresponding to the training task group Si , including:
  • the linear transformation matrix is feature integrated to obtain the attention representation sub-vector corresponding to the self-attention sub-network layer Vk ;
  • the Q attention representation sub-vectors are fused to obtain the attention representation vector corresponding to the training task group Si .
  • the single modality type includes a first modality type and a second modality type
  • the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type
  • the specific implementation method of the routing layer determination module 13 determines the target routing layer matching the modality type to which the training task group Si belongs in the single-modality routing layer and the cross-modality routing layer included in the modality representation model, including:
  • the cross-modality routing layer included in the modality representation model is determined as a target routing layer that matches the modality type to which the training task group S i belongs;
  • the first modality sub-routing layer corresponding to the first modality type is determined as the target routing layer that matches the modality type to which the training task group S i belongs;
  • the second modality sub-routing layer corresponding to the second modality type is determined as the target routing layer that matches the modality type to which the training task group Si belongs.
  • the single modality type includes a first modality type and a second modality type
  • the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type
  • the specific implementation method of the first optimization module 15 optimizing the single-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type includes:
  • the first modality sub-routing layer corresponding to the first modality type is optimized based on the first loss value
  • the second modality sub-routing layer corresponding to the second modality type is optimized based on the second loss value.
  • the first modality type is a text modality type
  • the first modality training task group is a text word sequence
  • the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence
  • the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence
  • the first optimization module 15 performs error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the real modal representation vector corresponding to the first modal training task group to obtain a specific implementation method of the first loss value corresponding to the first modal training task group, including:
  • the masked text word refers to a text word that has been masked
  • the prediction modality representation vector corresponding to the first modality training task group obtain the prediction representation feature corresponding to the masked text word, and Obtain the real representation features corresponding to the masked text words from the real modal representation vector corresponding to the modal training task group;
  • the first feature similarity is determined as a first loss value corresponding to the first modality training task group.
  • the training task group set includes at least two training task groups whose modality types are cross-modal types; the at least two training task groups whose modality types are cross-modal types include training task group S j , where j is a positive integer; the training task group S j includes first sample media data whose modality type is a first modality type, and second sample media data whose modality type is a second modality type; the predicted modal representation vector corresponding to the training task group S j includes predicted representation features corresponding to the first sample media data, and predicted representation features corresponding to the second sample media data;
  • the specific implementation method of the second optimization module 16 optimizing the cross-modal routing layer based on the predicted modality representation vector corresponding to the training task group whose modality type is a cross-modal type includes:
  • each training task group whose modality type is a cross-modal type as a cross-modal training task group and determine the third feature similarity between the training task group S j and the remaining cross-modal training task groups in at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the cross-modal training task groups in at least two cross-modal training task groups except the training task group S j ;
  • the cross-modal routing layer is optimized.
  • the data processing device 1 may further include: a task group acquisition module 17 , an interaction processing module 18 , a feature processing module 19 and a category identification module 20 .
  • the task group acquisition module 17 is used to acquire a target task group for describing the media data to be classified; the target task group includes at least two media data of different modal types;
  • An interaction processing module 18 used to perform attention interaction processing on the target task group in the modality representation model to obtain an attention representation vector corresponding to the target task group;
  • a feature processing module 19 is used to perform feature prediction processing on the attention representation vector corresponding to the target task group in the optimized cross-modal routing layer to obtain a predicted modality representation vector corresponding to the target task group;
  • the category identification module 20 is used to perform category identification processing on the predicted modal representation vector corresponding to the target task group to obtain the media category to which the media data to be classified belongs; the media category to which the media data to be classified belongs is used to archive the media data to be classified.
  • a routing distribution mechanism can be set in the modal representation model to divide the routing layer into a single-modal routing layer and a cross-modal routing layer.
  • Different routing layers can process data of different modal types.
  • data of different modal types are also trained and optimized separately.
  • Each routing layer will only retain relevant information of the current modal type. Therefore, there is no need to discard data of any modal type, and representation vectors of cross-modal and single-modal types can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types.
  • Figure 9 is a structural diagram of a computer device provided in an embodiment of the present application.
  • the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005.
  • the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002.
  • the communication bus 8002 is used to realize the connection and communication between these components.
  • the user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 8003 may also include a standard wired interface and a wireless interface.
  • the network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 8005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage.
  • the memory 8005 may also be at least one storage device located away from the aforementioned processor 8001. As shown in Figure 9, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module and device control applications.
  • the network interface 8004 can provide a network communication function;
  • the user interface 8003 is mainly used to provide an input interface for the user; and
  • the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:
  • the training task group set includes a training task group S i , where i is a positive integer; the modality type to which the training task group S i belongs is a single modality type or a cross-modality type; when the modality type to which the training task group S i belongs is a single modality type, the training task group S i includes one sample media data; when the modality type to which the training task group S i belongs is a cross-modality type, the training task group S i includes at least two sample media data of different modality types;
  • attention interaction processing is performed on the training task group Si to obtain the attention representation vector corresponding to the training task group Si ; the attention interaction processing is used to fully interact different elements in the training task group Si ;
  • a target routing layer matching the modal type to which the training task group S i belongs is determined, and feature prediction processing is performed on the attention representation vector corresponding to the training task group S i based on the target routing layer to obtain a predicted modal representation vector corresponding to the training task group S i ;
  • the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
  • the computer device 8000 described in the embodiment of the present application can execute the description of the data processing method in the embodiment corresponding to FIG. 2 above, and can also execute the description of the data processing device 1 in the embodiment corresponding to FIG. 8 above, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.
  • the embodiment of the present application also provides a computer-readable storage medium
  • the above-mentioned computer-readable storage medium stores a computer program executed by the computer device 8000 for data processing mentioned above, and the above-mentioned computer program includes program instructions.
  • the above-mentioned processor executes the above-mentioned program instructions, it can execute the description of the above-mentioned data processing method in the embodiment corresponding to Figure 2 above, so it will not be repeated here.
  • the description of the beneficial effects of using the same method will not be repeated.
  • the computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device.
  • the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device.
  • the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device.
  • the computer-readable storage medium is used to store the computer program and other programs and data required by the computer device.
  • the computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
  • a computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in one aspect of the embodiments of the present application.
  • the method and related device provided by the embodiment of the present application are described with reference to the method flow chart and/or structural diagram provided by the embodiment of the present application, and can be specifically implemented by computer program instructions for each process and/or box of the method flow chart and/or structural diagram, and the combination of the process and/or box in the flow chart and/or block diagram.
  • These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes of the flow chart and/or one box or multiple boxes of the structural diagram.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in one process or multiple processes of the flow chart and/or one box or multiple boxes of the structural diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the structure diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种数据处理方法、装置、设备以及可读存储介质，方法包括：获取训练任务组集合；训练任务组集合中包括训练任务组Si；对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；确定与训练任务组Si所属的模态类型相匹配的目标路由层，基于目标路由层对注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量；基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化，基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化。采用本申请，可以提升模态表征模型的通用性，提升模态表征结果的准确性。

Description

一种数据处理方法、装置、设备以及可读存储介质
本申请要求于2023年05月19日提交中国专利局、申请号为2023105720436、申请名称为“一种数据处理方法、装置、设备以及可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种数据处理方法、装置、设备以及可读存储介质。
背景技术
随着多媒体技术的快速发展，媒体数据(如图片、文本、视频、音频等等)得到海量生产，而随着海量媒体数据的生成，媒体数据的理解任务(如视频理解任务)就变得尤为重要，媒体数据的理解任务能够为媒体数据提供众多丰富且多样的媒体标签(如舞蹈标签、唱歌标签、竞赛标签、游戏标签等)，通过媒体数据的媒体标签可以便于对媒体数据进行多种媒体处理，例如，可以便于对媒体数据进行媒体检索、分类、归档、进行媒体推荐、媒体二次编辑等处理。
在媒体检索任务中，通过媒体标签可以很便利地检索出相关的媒体片段，媒体数据的文本标签、标题信息等，媒体检索对于媒体推荐、媒体加工都有着非常重要的意义。而随着媒体检索需求越来越大，对媒体检索的能力要求也越来越高，例如，在输入一个模态类型为文本类型的媒体数据后，对于检索结果的需求不仅限于是模态类型为文本类型的媒体数据，还需求检索到模态类型为非文本类型(如视频类型、音频类型、图像类型等)的媒体数据。即，在媒体数据的检索业务中，基于多模态的检索性能要求越来越大。
然而，在相关技术中，对于媒体数据多模态的理解任务，会由一个统一的模型来进行理解推理，具体的，针对各个模态会提取每个模态的特征，之后，再将所有模态的特征多模态信息进行融合，最终可以得到多模态信息。但是，相关技术在进行模型的训练时，为了提升多模态信息的准确性，会剔除单模态的个性信息，并提取不同模态之间的共性信息，由此会严重影响模型对于单模态数据的理解准确性，从而严重影响单模态数据在后续任务(如检索任务)中的处理结果的准确性。
发明内容
本申请实施例提供一种数据处理方法、装置、设备以及可读存储介质，可以在模态表征业务中，提升模态表征模型的通用性，提升模态表征结果的准确性。
本申请实施例一方面提供了一种数据处理方法，包括：
获取训练任务组集合；训练任务组集合中包括训练任务组Si，i为正整数；所述训练任务组Si所属模态类型为单模态类型或跨模态类型；在所述训练任务组Si所属模态类型为所述单模态类型时，所述训练任务组Si包含一个样本媒体数据；在所述训练任务组Si所属模态类型为所述跨模态类型时，所述训练任务组Si包含至少两个不同模态类型的样本媒体数据；
在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；注意力交互处理用于将训练任务组Si中的不同元素进行充分交互；
在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层，基于目标路由层对训练任务组Si对应的注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量；
当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单 模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化，基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为跨模态类型的任务组进行特征预测处理。
本申请实施例一方面提供了一种数据处理装置，包括：
集合获取模块，用于获取训练任务组集合；训练任务组集合中包括训练任务组Si，i为正整数；所述训练任务组Si所属模态类型为单模态类型或跨模态类型；在所述训练任务组Si所属模态类型为所述单模态类型时，所述训练任务组Si包含一个样本媒体数据；在所述训练任务组Si所属模态类型为所述跨模态类型时，所述训练任务组Si包含至少两个不同模态类型的样本媒体数据；
注意力处理模块，用于在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；注意力交互处理用于将训练任务组Si中的不同元素进行充分交互；
路由层确定模块，用于在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层；
预测模块，用于基于目标路由层对训练任务组Si对应的注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量；
第一优化模块，用于当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化；
第二优化模块，用于基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为跨模态类型的任务组进行特征预测处理。
在一个实施例中，集合获取模块获取训练任务组集合的具体实现方式，包括：
获取N个样本媒体数据；N为正整数，N个媒体数据中的一个样本媒体数据所属的模态类型，为第一模态类型与第二模态类型中的任意一种；第一模态类型与第二模态类型不同；
基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对N个样本媒体数据进行任务组构建处理，得到训练任务组集合。
在一个实施例中，集合获取模块基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对N个样本媒体数据进行任务组构建处理，得到训练任务组集合的具体实现方式，包括：
将N个样本媒体数据的第一样本媒体数据，与用于表征第一模态类型的第一标识进行组合，得到所属模态类型为第一模态类型的第一训练任务组；第一样本媒体数据是指N个样本媒体数据中，所属模态类型为第一模态类型的任一样本媒体数据；
将N个样本媒体数据中的第二样本媒体数据，与用于表征第二模态类型的第二标识进行组合，得到所属模态类型为第二模态类型的第二训练任务组；第二样本媒体数据是指N个样本媒体数据中，所属模态类型为第二模态类型的任一样本媒体数据；
基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组；
将第一训练任务组与第二训练任务组，均确定为所属模态类型为单模态类型的训练任务组，将所属模态类型为所述跨模态类型的训练任务组，与所属模态类型为所述单模态类型的训练任务组所组成的集合，确定为训练任务组集合。
在一个实施例中，第一样本媒体数据与第二样本媒体数据的数量均为一个或多个；一个或多个第一样本媒体数据中包括第一样本媒体数据Mj，j为正整数；所属模态类型为跨模态类型的训练任务组，包含第一样本媒体数据Mj对应的训练任务组；
集合获取模块基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组的具体实现方式，包括：
将第一样本媒体数据Mj对应的媒体来源渠道，确定为目标媒体来源渠道；
将一个或多个第二样本媒体数据中，媒体来源渠道为目标媒体来源渠道的第二样本媒体数据，确定为第一样本媒体数据Mj对应的关联样本媒体数据；
将第一样本媒体数据、关联样本媒体数据、第一标识以及第二标识进行组合，得到第一样本媒体数据Mj对应的训练任务组。
在一个实施例中，注意力处理模块在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量的具体实现方式，包括：
在模态表征模型中，通过特征提取网络层对训练任务组Si所包含的样本媒体数据进行特征提取处理，得到训练任务组Si对应的媒体特征；
通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，得到训练任务组Si对应的注意力表征向量。
在一个实施例中，模态表征模型中的多头自注意力网络层包括Q个自注意力子网络层；Q为正整数；Q个自注意力子网络层包括自注意力子网络层Vk，k为正整数；
注意力处理模块通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，得到训练任务组Si对应的注意力表征向量的具体实现方式，包括：
获取自注意力子网络层Vk所包含的注意力参数矩阵，将自注意力子网络层Vk所包含的注意力参数矩阵，与训练任务组Si对应的媒体特征进行运算处理，得到注意力参数矩阵对应的线性变换矩阵；
通过自注意力子网络层Vk中的全连接组件，对线性变换矩阵进行特征整合处理，得到自注意力子网络层Vk对应的注意力表征子向量；
当确定出Q个自注意力子网络层分别对应的注意力表征子向量时，将Q个注意力表征子向量进行融合处理，得到训练任务组Si对应的注意力表征向量。
在一个实施例中，单模态类型包含第一模态类型与第二模态类型；单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层；
路由层确定模块在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层的具体实现方式，包括：
若训练任务组Si所属的模态类型为跨模态类型，则将模态表征模型中所包含的跨模态路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；
若训练任务组Si所属的模态类型为第一模态类型，则将第一模态类型对应的第一模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；
若训练任务组Si所属的模态类型为第二模态类型，则将第二模态类型对应的第二模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层。
在一个实施例中，单模态类型包含第一模态类型与第二模态类型；单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层；
第一优化模块基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化的具体实现方式，包括：
将所属模态类型为第一模态类型的训练任务组，确定为第一模态训练任务组，将所属模态类型为第二模态类型的训练任务组，确定为第二模态训练任务组；
获取第一模态训练任务组对应的真实模态表征向量，以及第二模态训练任务组对应的真实模态表征向 量；
对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值；
对第二模态训练任务组对应的预测模态表征向量，与第二模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第二模态训练任务组对应的第二损失值；
基于第一损失值对第一模态类型对应的第一模态子路由层进行优化，基于第二损失值对第二模态类型对应的第二模态子路由层进行优化。
在一个实施例中，第一模态类型为文本模态类型，第一模态训练任务组为文本词序列；第一模态训练任务组对应的预测模态表征向量由文本词序列中，每个文本词分别对应的预测表征特征所组成；第一模态训练任务组对应的真实模态表征向量，由文本词序列中，每个文本词分别对应的真实表征特征所组成；
第一优化模块对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值的具体实现方式，包括：
在文本词序列所包含的文本词中获取遮掩文本词；遮掩文本词是指经过遮掩处理的文本词；
在第一模态训练任务组对应的预测模态表征向量中，获取遮掩文本词对应的预测表征特征，在第一模态训练任务组对应的真实模态表征向量中，获取遮掩文本词对应的真实表征特征；
确定遮掩文本词对应的预测表征特征，与遮掩文本词对应的真实表征特征之间的第一特征相似度；
将第一特征相似度确定为第一模态训练任务组对应的第一损失值。
在一个实施例中，训练任务组集合中，包含至少两个所属模态类型为跨模态类型的训练任务组；至少两个所属模态类型为跨模态类型的训练任务组，包含训练任务组Sj，j为正整数；训练任务组Sj包括所属模态类型为第一模态类型的第一样本媒体数据，以及所属模态类型为第二模态类型的第二样本媒体数据；训练任务组Sj所对应的预测模态表征向量中，包括第一样本媒体数据对应的预测表征特征，以及第二样本媒体数据对应的预测表征特征；
第二优化模块基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化的具体实现方式，包括：
确定第一样本媒体数据对应的预测表征特征，与第二样本媒体数据对应的预测表征特征之间的第二特征相似度；
将每个所属模态类型为跨模态类型的训练任务组，均确定为跨模态训练任务组，确定训练任务组Sj与至少两个跨模态训练任务组中，剩余跨模态训练任务组之间的第三特征相似度；剩余跨模态训练任务组是指至少两个跨模态训练任务组中，除训练任务组Sj以外的跨模态训练任务组；
基于第二特征相似度与第三特征相似度，对跨模态路由层进行优化。
在一个实施例中，数据处理装置还可以包括：
任务组获取模块，用于获取用于描述待分类媒体数据的目标任务组；目标任务组包括至少两个不同模态类型的媒体数据；
交互处理模块，用于在模态表征模型中对目标任务组进行注意力交互处理，得到目标任务组对应的注意力表征向量；
特征处理模块，用于在优化后的跨模态路由层中，对目标任务组对应的注意力表征向量进行特征预测处理，得到目标任务组对应的预测模态表征向量；
类别识别模块，用于对目标任务组对应的预测模态表征向量进行类别识别处理，得到待分类媒体数据所属的媒体类别；待分类媒体数据所属的媒体类别用于对待分类媒体数据进行归档处理。
本申请实施例一方面提供了一种计算机设备，包括：处理器和存储器；
存储器存储有计算机程序，计算机程序被处理器执行时，使得处理器执行本申请实施例中的方法。
本申请实施例一方面提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序包括程序指令，程序指令当被处理器执行时，执行本申请实施例中的方法。
本申请的一个方面，提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机程序，处理器执行该计算机程序，使得该计算机设备执行本申请实施例中一方面提供的方法。
在本申请实施例中，提供一种通用的模态表征模型，可以同时为所属模态类型为单模态类型的数据以及所属模态类型为跨模态类型的数据进行服务。具体的，本申请可以采用包含单模态类型与跨模态类型的训练任务组集合，对模态表征模型进行训练，其中，在模态表征模型中，包含有单模态路由层与跨模态路由层，单模态路由层可以对所属模态类型为单模态类型的数据(注意力表征向量)进行特征预测处理，跨模态路由层可以对所属模态类型为跨模态类型的数据(注意力表征向量)进行特征预测处理，而在训练优化时，单模态路由层可以基于所属模态类型为所述单模态类型的训练任务组所对应的预测模态表征向量来进行优化，跨模态路由层可以由所属模态类型为所述跨模态类型的训练任务组所对应的预测模态表征向量来进行优化。基于此，可以使得模态表征模型中的单模态路由层与跨模态路由层能够同时进行训练优化，且在训练优化时，是基于不同模态类型的数据来分别进行训练优化，无需损失任何一种模态类型的数据，可以同时保持单模态路由层与跨模态路由层的性能，由此可以提升模态表征模型对单模态类型与跨模态类型的通用性与灵活性。综上可知，本申请可以在模态表征模型中设置路由分发机制，将路由层划分为单模态路由层与跨模态路由层，不同的路由层可以对不同模态类型的数据进行处理，在训练时也由不同模态类型的数据分别进行训练优化，每个路由层只会保留当前模态类型的相关信息，由此无需舍弃任何一种模态类型的数据，也能够同时输出跨模态类型与单模态类型的表征向量，可以很好地提升各种模态类型的数据的表征结果的准确性。综上，本申请可以在模态表征业务中，提升模态表征模型的通用性，提升模态表征结果的准确性。
附图说明
图1是本申请实施例提供的一种数据处理系统的网络架构图；
图2是本申请实施例提供的一种数据处理方法的流程示意图；
图3是本申请实施例提供的一种transformer架构的架构示意图；
图4是本申请实施例提供的一种模态路由分发的架构示意图；
图5是本申请实施例提供的一种媒体数据检索的场景示意图；
图6是本申请实施例提供的一种对单模态路由层进行优化的流程示意图；
图7是本申请实施例提供的一种对跨模态路由层进行优化的流程示意图；
图8是本申请实施例提供的一种数据处理装置的结构示意图；
图9是本申请实施例提供的一种计算机设备的结构示意图。
具体实施方式
本申请涉及人工智能及相关概念，为便于理解，以下将对人工智能及相关概念进行简要阐述：
人工智能(Artificial Intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个综合技术，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式做出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。
人工智能技术是一门综合学科，涉及领域广泛，既有硬件层面的技术也有软件层面的技术。人工智能基础技术一般包括如传感器、专用人工智能芯片、云计算、分布式存储、大数据处理技术、操作/交互系统、机电一体化等技术。人工智能软件技术主要包括计算机视觉技术、语音处理技术、自然语言处理技术以及机器学习/深度学习等几大方向。
随着人工智能技术研究和进步，人工智能技术在多个领域展开研究和应用，例如常见的智能家居、智能穿戴设备、虚拟助理、智能音箱、智能营销、无人驾驶、自动驾驶、无人机、机器人、智能医疗、智能客服等，相信随着技术的发展，人工智能技术将在更多的领域得到应用，并发挥越来越重要的价值。
本申请实施例提供的方案属于人工智能领域下属的机器学习(Machine Learning,ML)和自然语言处理(Nature Language processing,NLP)技术。
机器学习(Machine Learning,ML)是一门多领域交叉学科，涉及概率论、统计学、逼近论、凸分析、算法复杂度理论等多门学科。专门研究计算机怎样模拟或实现人类的学习行为，以获取新的知识或技能，重新组织已有的知识结构使之不断改善自身的性能。机器学习是人工智能的核心，是使计算机具有智能的根本途径，其应用遍及人工智能的各个领域。机器学习和深度学习通常包括人工神经网络、置信网络、强化学习、迁移学习、归纳学习、式教学习等技术。
自然语言处理(Nature Language processing,NLP)是计算机科学领域与人工智能领域中的一个重要方向。它研究能实现人与计算机之间用自然语言进行有效通信的各种理论和方法。自然语言处理是一门融语言学、计算机科学、数学于一体的科学。因此，这一领域的研究将涉及自然语言，即人们日常使用的语言，所以它与语言学的研究有着密切的联系。自然语言处理技术通常包括文本处理、语义理解、机器翻译、机器人问答、知识图谱等技术。
本申请实施例具体可通过自然语言处理技术，将媒体数据(如文本数据、视频数据)的相关文本数据(如标题数据、描述数据)进行语义分析处理，以得到该媒体数据的语义分析结果，从而能够更好地理解该媒体数据；同时，本申请实施例可以通过机器学习技术将相关模型(如后续提及的模态表征模型)进行训练优化，以提升模型的输出结果的准确性。
为便于理解，请参见图1，图1是本申请实施例提供的一种数据处理系统的网络架构图。如图1所示，该网络架构可以包括业务服务器1000和终端设备集群，终端设备集群可以包括一个或者多个终端设备，这里将不对终端设备的数量进行限制。如图1所示，多个终端设备可以包括终端设备100a、终端设备100b、终端设备100c、…、终端设备100n；如图1所示，终端设备100a、终端设备100b、终端设备100c、…、终端设备100n可以分别与业务服务器1000进行网络连接，以便于每个终端设备可以通过该网络连接与业务服务器1000之间进行数据交互。另外，终端设备集群100中的任一终端设备可以是指运行有操作系统的智能设备，本申请实施例对终端设备的操作系统不进行具体限定。
如图1所示的数据处理系统中的终端设备可以为智能手机、平板电脑、笔记本电脑、掌上电脑、台式计算机、移动互联网设备(MID，mobile internet device)、POS(Point Of Sales，销售点)机、智能音箱、智能电视、智能手表、智能车载终端、虚拟现实(Virtual Reality，VR)设备、增强现实(Augmented Reality，AR)设备等，但并不局限于此。终端设备往往配置有显示装置，显示装置可以为显示器、显示屏、触摸屏等等，触摸屏可以为触控屏、触控面板等等。
如图1所示的数据处理系统中的业务服务器可以是单个的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN、以及大数据和人工智能平台等基础云计算服务的云服务器。终端设备与业务服务器可以通过有线或无线通信方式进行直接或间接地连接，本申请在此不做限制。
在一种可能的实现方式中，终端设备(如终端设备100a)中运行有应用，如视频应用、浏览器应用、游戏应用、教育应用等，这里将不对应用进行一一举例说明。在本申请各实施例中，以视频应用为例进行 说明。对象可在终端设备中运行视频应用，而视频应用可以以feeds流(feed可以是指消息来源，feed又可译为源料、馈送、资讯提供、供稿、摘要、源、新闻订阅。其是一种资料格式，应用透过它将最新资讯传播给对象，通常以时间轴方式排列)的形式向对象提供媒体数据(如文本数据、视频数据、音频数据、图像数据等等)，对象可在视频应用中浏览到这些媒体数据。对于视频应用，还可以向对象提供检索功能，对象可以通过输入某个信息的方式，来检索与该输入信息相关联的媒体数据(如文本数据、图像数据、音频数据、视频数据等)。例如，对象输入某个文本数据后，视频应用可以输出与该文本数据相关联的另一个文本数据，也可以输出与该文本数据相关联的图像数据、视频数据、音频数据等等；或者，对象输入某个图像数据后，视频应用可以输出与该图像数据相关联的另一个图像数据，也可以输出与该图像数据相关联的视频数据、文本数据、音频数据等等。而对于图像数据、文本数据、音频数据以及视频数据，可以视作不同模态类型的媒体数据，如，对于图像数据，可以理解为是模态类型为图像模态类型的媒体数据，对于视频数据，可以理解为是模态类型为视频模态类型的媒体数据。也就是说，视频应用可以提供多模态类型的媒体数据的检索功能，而为了提升多模态的检索结果的准确性，本申请提供一种基于模态路由分发机制的多模态联合训练框架，在只需训练一个模态表征模型的基础上，同时完成跨模态(跨模态也就是指两个或两个以上模态的模态组合)与单模态的模态表征特征确定的任务，基于各种模态的模态表征特征(可包含跨模态的模态表征特征与单模态的模态表征特征)，即可进行不同模态类型的媒体数据的检索，由此可以减少模型存储大、检索耗时以及重复计算等问题，使得单模态与跨模态任务能够互相促进，分别保持良好地性能，提升通用性与准确性。
具体的，可以在业务服务器1000中部署模态表征模型，而业务服务器1000可以采集到对象在视频应用中所输入的信息(如输入的文本数据)，基于对象在视频应用中所输入的信息，业务服务器1000可以通过模态表征模型对对象所输入的信息进行推理分析，以得到输入信息的模态表征向量，基于该输入信息的模态表征向量，业务服务器1000即可确定出与该输入信息相关联的媒体数据(如文本数据、图像数据、视频数据、音频数据等等)。其中，可以理解的是，部署于业务服务器1000中的模态表征模型，可以包含有不同的模态路由层，例如，具体可以包含用于对文本数据进行路由处理的文本模态路由层、用于对图像数据进行路由处理的图像模态路由层、用于对音频数据进行路由处理的音频模态路由层，还可以包含用于对多模态媒体数据进行路由处理的跨模态路由层。那么对于不同模态类型的输入信息，可以由相应的模态路由层来进行相应的路由处理(这里的路由层可以是指全连接层，路由处理可以是指通过全连接层进行线性变换处理，而由于通过全连接层进行线性变换处理实际上是指通过全连接层进行特征整合预测处理，那么这里的路由处理也可以理解为是进行特征预测处理)，以得到输入信息的模态表征向量。例如，输入信息为文本数据时，可以由文本模态路由层对其进行路由处理，输入信息为图像数据时，可以由图像模态路由层对其进行路由处理。
也就是说，本申请在模态表征模型中，设置了路由分发机制，为不同模态类型设置了不同的模态路由层，对于不同模态类型的输入信息，可以分发到相应的模态路由层来进行路由处理，相较于传统技术中由统一的路由层来对不同模态类型的输入数据进行统一的处理，设置路由分发机制的方式可以提升路由层的针对性，一种模态类型的输入数据可以由对应的模态路由层来进行针对性地路由处理，基于不同的模态路由层可以适应于各种模态类型的输入数据，这样可以很好地提升模态表征模型对不同模态类型的通用性，而且，由于不同模态类型的输入数据是由相应的模态路由层进行针对性处理，那么其处理过程是符合模态类型自身特性的，所得到的处理结果(即输入数据的表征结果)也可以更为准确，即：可以提升不同模态类型的数据的表征结果的准确性。
本申请中，为了提升模态表征模型输出结果的准确性，可以对模态表征模型进行训练优化，而在模态表征模型的训练过程中，由于设置了不同的模态路由层，对于不同的模态路由层，可以采用不同模态类型的训练样本来进行针对性地训练。例如，对于文本模态路由层，可以采用文本样本数据来进行训练，对于 图像模态路由层，可以采用图像样本数据来进行训练，对于跨模态路由层，可以采用跨模态的样本数据(如既包含图像样本数据又包含文本样本数据的样本数据)来进行训练。对于文本模态路由层、图像模态路由层这种仅针对一种模态类型的路由层，本申请可以称之为单模态路由层，而对于针对多种模态类型的路由层，本申请可称之为跨模态路由层，跨模态路由层可以对多模态类型的数据进行处理。而对于单模态路由层，可以由所属模态类型为单模态类型(如文本模态类型、图像模态类型、音频模态类型等)的训练数据(训练任务组)来对其进行训练，对于跨模态路由层，可以由所属模态类型为跨模态类型的训练数据(训练任务组)来对其进行训练。
具体的，本申请可以配置训练任务组集合，例如，可以将一个样本媒体数据与其对应的模态类型，组成一个训练任务组，也可以将两个不同模态类型的样本媒体数据，与其对应的模态类型，组成一个训练任务组，基于此，通过样本媒体数据与模态类型的组合，即可得到包含单模态类型与跨模态类型的训练任务组集合。例如，样本媒体数据为样本文本数据1时，其模态类型为文本模态类型，那么将其进行组合后可以得到一个训练任务组为<样本文本数据1，文本模态类型>，该训练任务组仅包含一个文本数据，该训练任务组所属模态类型为文本模态类型(单模态类型)；又如，样本媒体数据为样本图像数据1时，其模态类型为图像模态类型，那么将其进行组合后可以得到一个训练任务组为<样本图像数据1，图像模态类型>，该训练任务组仅包含一个文本数据，该训练任务组所属模态类型为图像模态类型(单模态类型)；又如，样本媒体数据为样本文本数据1与样本图像数据1时，其模态类型包含文本模态类型与图像模态类型，那么将其进行组合后可以得到一个训练任务组为<样本文本数据1-样本图像数据1，文本模态类型-图像模态类型>，该训练任务组同时包含一个文本数据与一个图像数据，该训练任务组所属模态类型为跨模态类型。也就是说，本申请中的训练任务组可以包含有单模态类型的训练任务组与跨模态类型的训练任务组，对于单模态类型的训练任务组而言，其仅会包含一种模态类型的样本媒体数据，此模态类型可以为文本模态类型、图像模态类型、音频模态类型等；对于跨模态类型的训练任务组而言，其需要包含有两种或两种以上的不同模态类型的样本媒体数据。
进一步地，对于每一个训练任务组，可以调用模态表征模型对其进行注意力交互处理，其中，这里的注意力交互处理可以是指基于多头自注意力机制，对其进行多头自注意力处理，由于经过多头自注意力处理可以使得输入数据中的每一个元素与其他元素进行充分交互联系，所以本申请将多头自注意力处理称为注意力交互处理，通过注意力交互处理所得到的结果称为训练任务组所对应的注意力表征向量。随后，对于每一个注意力表征向量，可以基于训练任务组所属的模态类型，在模态表征模型所包含的单模态路由层与跨模态路由层中，确定出与其相匹配的目标路由层，通过该目标路由层，即可对训练任务组的注意力表征向量进行路由处理(特征预测处理)，最终即可得到训练任务组对应的模态表征向量(训练任务组对应的模态表征向量也就是指目标路由层所输出的用于表征模态类型的特征向量，由于训练阶段需要对模态表征模型进行训练，那么目标路由层所输出的模态表征向量可以理解为一个预测值，所输出的模态表征向量即可称为预测模态表征向量，需要基于该预测模态表征向量对目标路由层进行训练优化，由此使得模态表征模型所输出的模态表征向量越来越准确)。举例而言，假设某个训练任务组所属模态类型为单模态类型中的文本模态类型，那么在得到该训练任务组的注意力表征向量后，可以在模态表征模型所包含的单模态路由层中，将文本模态路由层确定为该训练任务组的目标路由层，基于该文本模态路由层即可输出该训练任务组对应的预测模态表征向量，而基于该训练任务组对应的预测模态表征向量，可以相应地对文本模态路由层进行训练优化。
也就是说，当确定出训练任务组集合中，每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单模态类型(如文本模态类型、图像模态类型、音频模态类型或视频模态类型)的训练任务组所对应的预测模态表征向量，可以对对应的单模态路由层(如文本模态路由层、图像模态路由层、音频模态路由层或视频模态路由层)进行优化，而优化后的单模态路由层，可以对单模态类型的数据进行路由处 理；同理，基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，可以对跨模态路由层进行优化，优化后的跨模态路由层，可以对跨模态类型的数据进行路由处理。
应当理解，本申请实施例提供一种基于模态路由分发机制的多模态联合训练框架，在只需训练一个模态表征模型的基础上，可以同时完成跨模态与单模态的模态表征向量输出的任务，基于各个模态的模态表征特征，即可进行不同模态类型的媒体数据的检索，由此可以减少模型存储大、检索耗时以及重复计算等问题，使得单模态与跨模态任务能够互相促进，分别保持良好地性能，提升通用性与准确性。
可以理解的是，本申请实施例所提供的方法可以由计算机设备执行，计算机设备包括但不限于图1中所提及的终端设备或业务服务器。
需要说明的是，在本申请的具体实施方式中，涉及到用户信息、用户数据(如上述对象所输入的信息)等相关的数据，均是需要经过用户手动授权许可(即经过用户同意)才进行获取得到的。也就是说，当本申请以上实施例运用到具体产品或技术中时，本申请实施例所提供的方法与相关功能是在获得用户许可或者同意下所运行的(可以由用户主动开启本申请实施例所提供的功能)，且相关数据的收集、使用和处理需要遵守相关地区的相关法律法规和标准。
为便于理解，接下来将结合附图对本申请实施例提供的数据处理方法进行详细描述。请参见图2，图2是本申请实施例提供的一种数据处理方法的流程示意图。其中，该方法可以由计算机设备执行，该计算机设备可以为终端设备(例如，上述图1所示的终端设备集群中的任一终端设备，如终端设备100a)，该计算机设备也可以为服务器(如上述图1所对应实施例中的业务服务器1000)，该计算机设备还可以由终端设备和服务器。为便于理解，本实施例以该方法由终端设备执行为例进行说明。如图2所示，该数据处理方法至少可以包括以下步骤S101-步骤S104：
步骤S101，获取训练任务组集合；训练任务组集合中包括训练任务组Si，i为正整数；训练任务组Si所属模态类型为单模态类型或跨模态类型；在训练任务组Si所属模态类型为单模态类型时，训练任务组Si包含一个样本媒体数据；在训练任务组Si所属模态类型为跨模态类型时，训练任务组Si包含至少两个不同模态类型的样本媒体数据。
本申请中，单模态类型可以是指媒体数据所属的模态类型，具体可以包含文本模态类型、图像模态类型、音频模态类型以及视频模态类型，某个训练任务组所属的模态类型可以基于训练任务组所包含的样本媒体数据所确定，例如，训练任务组仅包含一个样本媒体数据，且该样本媒体数据所属的模态类型为文本模态类型(即该样本媒体数据为样本文本数据)，那么该训练任务组所属的模态类型可为单模态类型，且具体可为文本模态类型；又如，训练任务组仅包含一个样本媒体数据，且该样本媒体数据所属的模态类型为图像模态类型(即该样本媒体数据为样本图像数据)，那么该训练任务组所属的模态可为单模态类型，且具体为图像模态类型。也就是说，当某个训练任务组仅包含一种模态类型的样本媒体数据时，该训练任务组所属的模态类型即可为单模态类型，且具体的模态类型是基于所包含的样本媒体数据的模态类型所确定的。同理可推导知，在某个训练任务组包含两个或两个以上的样本媒体数据，且这些样本媒体数据所属的模态类型不同时，即可确定该训练任务组所属的模态类型为跨模态类型(或多模态类型)。例如，某个训练任务组既包含样本文本数据(模态类型为文本模态类型的样本媒体数据)，又包含有样本视频数据(模态类型为视频模态类型的样本媒体数据)，那么即可确定该训练任务组所属的模态类型为跨模态类型。
也就是说，本申请中的样本媒体数据，可以是指作为训练样本的媒体数据，其具体可以包含样本文本数据、样本图像数据、样本音频数据或样本视频数据，当某个训练任务组由一种模态类型的样本媒体数据所组成时，可以确定该训练任务组所属的模态类型为单模态类型，且可以将该样本媒体数据所属的模态类型确定为该训练任务组所属的模态类型；而当某个训练任务组由多种(两种及两种以上)模态类型的样本媒体数据所组成时，可以确定该训练任务组所属的模态类型为跨模态类型或多模态类型。
具体的，对于获取训练任务组集合的具体实现方式可为：可以获取N(N为正整数)个样本媒体数据； 其中，N个媒体数据中的一个样本媒体数据所属的模态类型，为第一模态类型与第二模态类型中的任意一种；且第一模态类型与第二模态类型不同；基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，可以对N个样本媒体数据进行任务组构建处理，由此可以得到训练任务组集合。
可以理解的是，这里的第一模态类型与第二模态类型可以是指媒体数据不同的模态类型，第一模态类型与第二模态类型均可以是指文本模态类型、图像模态类型、音频模态类型或视频模态类型，且第一模态类型与第二模态类型不同，例如，第一模态类型为文本模态类型时，第二模态类型可以是指图像模态类型、音频模态类型或视频模态类型中的任意一种或多种，第一模态类型为图像模态类型时，第二模态类型可以是指文本模态类型、音频模态类型或视频模态类型中的任意一种或多种。本申请中预先获取到不同模态类型的多个作为样本的样本媒体数据，随后再基于各个样本媒体数据所属的模态类型，来进行任务组构建处理，由此来得到包含不同样本媒体数据的训练任务组，进而得到训练任务组集合。
具体的，对于基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对N个样本媒体数据进行任务组构建处理，得到训练任务组集合的具体实现方式可为：可以将N个样本媒体数据的第一样本媒体数据，与用于表征第一模态类型的第一标识进行组合，由此可以得到所属模态类型为第一模态类型的第一训练任务组；其中，第一样本媒体数据是指N个样本媒体数据中，所属模态类型为第一模态类型的任一样本媒体数据；应当理解的是，N个样本媒体数据中包含有第一模态类型的媒体数据与第二模态类型的媒体数据，可以将所属模态类型为第一模态类型的任一媒体数据，均称之为第一样本媒体数据，那么对于任意一个第一样本媒体数据而言，可以将其于第一模态类型的标识进行组合，由此可以得到一个训练任务组(此训练任务组可以称之为第一训练任务组)，由于第一样本媒体数据的数量可能并非为一个，所以第一训练任务组的数量也可能为多个；同理，可以将N个样本媒体数据中的第二样本媒体数据，与用于表征第二模态类型的第二标识进行组合，由此可以得到所属模态类型为第二模态类型的第二训练任务组；其中，第二样本媒体数据是指N个样本媒体数据中，所属模态类型为第二模态类型的任一样本媒体数据；应当理解的是，N个样本媒体数据中包含有第一模态类型的媒体数据与第二模态类型的媒体数据，可以将所属模态类型为第二模态类型的任一媒体数据，均称之为第二样本媒体数据，那么对于任意一个第二样本媒体数据而言，可以将其于第二模态类型的标识进行组合，由此可以得到一个训练任务组(此训练任务组可以称之为第二训练任务组)，由于第二样本媒体数据的数量可能并非为一个，所以第二训练任务组的数量也可能为多个；基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，可以对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组；进一步地，可以将第一训练任务组与第二训练任务组，均确定为所属模态类型为单模态类型的训练任务组，并将所属模态类型为所述跨模态类型的训练任务组，与所属模态类型为所述单模态类型的训练任务组所组成的集合，确定为训练任务组集合。
可以理解的是，以第一模态类型为文本模态类型，第二模态类型包含图像模态类型与视频模态类型为例，N个样本媒体数据可以包含样本文本数据1(模态类型为文本模态类型)、样本文本数据2(模态类型为文本模态类型)、样本图像数据1(模态类型为图像模态类型)、样本视频数据1(模态类型为视频模态类型)、样本视频数据2(模态类型为视频模态类型)。其中，用于表征文本模态类型的标识可为“text”，用于表征图像模态类型的标识可为“vision”，用于表征视频模态类型的标识可为“video”，那么可以将样本文本数据1以及用于表征文本模态类型的标识“text”进行组合，由此可以得到一个训练任务组为<样本文本数据1，text>，也可以将样本文本数据1以及用于表征文本模态类型的标识“text”进行组合，由此可以得到一个训练任务组为<样本文本数据2，text>，应当理解，由于训练任务组<样本文本数据1，text>与<样本文本数据2，text>均只包含了文本模态类型的样本媒体数据，则这两个训练任务组所属的模态类型可以为单模态类型，所属的模态类型具体可为文本模态类型；同理，可以将样本图像数据1与用于表征图像模态类型的标识“vision”进行组合，由此可以得到一个训练任务组为<样本图像数据1，vision>，该训练 任务组只包含了图像模态类型的样本媒体数据，则这个训练任务组所属的模态类型可以为单模态类型，且所属的模态类型具体可为图像模态类型；同理，可以将样本视频数据1与样本视频数据2分别与用于表征视频模态类型的标识“video”进行组合，由此可以得到训练任务组<样本视频数据1，video>与<样本视频数据2，video>，这两个训练任务组均只包含了视频模态类型的样本媒体数据，则这个训练任务组所属的模态类型可以为单模态类型，且所属的模态类型具体可为视频模态类型。也就是说，对于样本媒体数据，可以将某种模态类型的样本媒体数据，与其对应的模态类型的标识进行组合，由此即可得到模态类型为单模态类型的训练任务组，同样，也可以将不同模态类型的样本媒体数据进行跨模态组合，由此即可得到模态类型为多模态类型(或称为跨模态类型)的训练任务组。换言之，所属模态类型为单模态类型的一个训练任务组，其应当包含一个样本媒体数据与该样本媒体数据所属的模态类型的标识。
具体的，可以将媒体来源渠道(媒体来源渠道可以是指样本媒体数据的获取方式或获取位置)相同的不同模态类型的样本媒体数据进行组合，以第一样本媒体数据与第二样本媒体数据的数量均为一个或多个，一个或多个第一样本媒体数据中包括第一样本媒体数据Mj(j为正整数)，所属模态类型为跨模态类型的训练任务组，包含第一样本媒体数据Mj对应的训练任务组为例，对于基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组的具体实现方式可为：可以将第一样本媒体数据Mj对应的媒体来源渠道，确定为目标媒体来源渠道；随后，可以将一个或多个第二样本媒体数据中，媒体来源渠道为目标媒体来源渠道的第二样本媒体数据，确定为第一样本媒体数据Mj对应的关联样本媒体数据；最后可以将第一样本媒体数据、关联样本媒体数据、第一标识以及第二标识进行组合，即可得到第一样本媒体数据Mj对应的训练任务组。
举例而言，假设N个样本媒体数据包含样本文本数据a、样本文本数据b、样本图像数据a、样本视频数据h1、样本视频数据h2，对于样本文本数据a与样本图像数据a均来自于样本视频数据h1(样本文本数据a为样本视频数据h1的视频描述信息，样本图像数据a是样本视频数据h1的某个画面帧)，那么可以将样本文本数据a与样本图像数据a确定为是互相关联的(样本文本数据a为样本图像数据a的关联媒体数据，同理，样本图像数据a也是样本文本数据a的关联媒体数据)，样本文本数据a与样本视频数据h1也是互相关联的，样本图像数据a与样本视频数据h1也是互相关联的，基于此，可以将样本文本数据a与样本图像数据a及其对应的标识(用于表征文本模态类型的标识“text”与用于表征图像模态类型的标识“vision”)进行组合，得到训练任务组为<样本文本数据a-样本图像数据a，text-vision>，也可以将样本文本数据a与样本视频数据h1及其对应的标识(用于表征文本模态类型的标识“text”与用于表征视频模态类型的标识“video”)进行组合，得到训练任务组为<样本文本数据a-样本视频数据h1，text-video>，也可以将样本图像数据a与样本视频数据h1及其对应的标识(用于表征图像模态类型的标识“vision”与用于表征视频模态类型的标识“video”)进行组合，得到训练任务组为<样本图像数据a-样本视频数据h1，vision-video>。可见，所属模态类型为跨模态类型的一个训练任务组，其应当包含两个及两个以上的样本媒体数据以及各个样本媒体数据所属的模态类型的标识。
综上可见，本申请中的任务组构建处理是指按照各个样本媒体数据所属的模态类型、各个样本媒体数据所属模态类型的标识，将样本媒体数据与模态类型的标识进行相应组合，以得到所属模态类型为单模态类型的训练任务组与跨模态类型的训练任务组，本申请在组合样本媒体数据得到训练任务组时，各个训练任务组的格式均是统一的，无论是单模态类型的训练任务组还是多模态类型的训练任务组，均是由样本媒体数据及其对应的用于表征模态类型的标识所组合得到的，通过用于表征模态类型的标识，即可清晰明了地表明该训练任务组包含的模态类型是单模态类型还是跨模态类型，进而可以反映该训练任务组所属的模态类型为单模态类型还是跨模态类型。此外，本申请中的训练任务组Si可以是指训练任务组集合中的任一个训练任务组，而对于训练任务组集合中的所属模态类型为单模态类型的训练任务组，其主要是由一种模 态类型(如文本模态类型、图像模态类型、视频模态类型或音频模态类型中的任一种)的样本媒体数据所组成的，且该样本媒体数据的数量为一个，那么对于该训练任务组集合中的某个训练任务组(如训练任务组Si)而言，在该训练任务组为单模态类型时，该训练任务组中可仅包含一个样本媒体数据；而对于训练任务组集合中的所属模态类型为跨模态类型的训练任务组，其需要由至少两种不同的模态类型的样本媒体数据所组成，且每种模态类型下的样本媒体数据的数量需要至少一个，那么对于该训练任务组集合中的某个训练任务组(如训练任务组Si)而言，在该训练任务组为跨模态类型时，该训练任务组中至少应包含有两个不同模态类型的样本媒体数据。
步骤S102，在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；注意力交互处理用于将训练任务组Si中的不同元素进行充分交互。
本申请中，模态表征模型可以是指基于transformer架构的模型，通过transformer架构可以对每个训练任务组进行特征提取建模处理，通过特征提取建模处理可以使得训练任务组中的各个特征级的元素进行充分交互，让各个元素更能关注到与自身关联性较强的元素，所以也可以将特征提取建模处理称为注意力交互处理。具体的，transformer架构的整个网络结构完全是由注意力机制(Attention机制)组成，更准确地讲，transformer架构由且仅由self-Attenion和前馈神经网络(Feed Forward Neural Network，FFN)组成，本申请中，基于Transformer的可训练的神经网络(即模态表征模型)可以通过堆叠Transformer的形式进行搭建，采用Attention机制可以解决传统的RNN(或者LSTM，GRU等)的计算context遗忘性问题。为便于理解transformer架构，请一并参见图3，图3是本申请实施例提供的一种transformer架构的架构示意图。如图3所示，在transformer架构中，包含有编码器与解码器，编码器中可以包含有多头注意力层与路由层，通过多头注意力层可以将输入特征进行多头注意力处理，通过编码器中的路由层可以对多头注意力层所处理的内容进行路由处理，其中，这里的路由层可以是指一个FFN，具体可以是指一个多层的全连接层，通过路由层可以对多头注意力层所处理的内容进行全连接处理(特征整合处理，也可以称之为路由处理)。进一步地，编码器所编码得到的特征(即路由层所输出的内容)，可以输入至解码器中，通过解码器可以对编码器所编码得到的特征进行解码处理，最终解码器所解码输出的内容，可以作为输入特征所对应的模态表征向量。
基于上述可知，模态表征模型中可以包含有多头注意力层，本申请可以将其确定为多头自注意力网络层，而这里的在模态表征模型中对训练任务组Si进行注意力交互处理，即可为是在多头自注意力网络层中对训练任务组Si进行注意力交互处理。其具体实现方式可为：在模态表征模型中，可以通过特征提取网络层对训练任务组Si所包含的样本媒体数据进行特征提取处理，由此可以得到训练任务组Si对应的媒体特征；进一步地，可以通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，由此即可得到训练任务组Si对应的注意力表征向量。
可以理解的是，这里的特征提取网络层可以是指embedding层，通过embedding层可以提取训练任务组中，样本媒体数据的特征，得到媒体特征。其中，这里的媒体特征可以基于样本媒体数据所属的模态类型来确定，当样本媒体数据为文本模态类型时，样本媒体数据的媒体特征可以是指每一个文本词对应的词特征；当样本媒体数据为图像模态类型时，样本媒体数据的媒体特征可以是指像素特征(一个图像可以划分为多个像素网络，媒体特征可以由每一个像素网格对应的像素特征所组成)；当样本媒体数据为音频模态类型时，样本媒体数据的媒体特征可以是指音素特征；当样本媒体数据为视频模态类型时，样本媒体数据的媒体特征可以由视频帧对应的帧特征所组成。而当样本媒体数据为跨模态类型时，训练任务组的媒体特征可以由不同模态类型的样本媒体数据的媒体特征所共同组成，即可以由词特征、像素特征、音素特征、帧特征中的两种或两种以上的特征共同组成。
其中，多头自注意力网络层可以包含有Q(Q为正整数)个自注意力子网络层，对于多头自注意力网络层的输出结果，可以是由这些自注意力子网络层的输出结果进行融合后所得到的。具体的，以Q个自注 意力子网络层包括自注意力子网络层Vk(k为正整数)为例，对于通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，得到训练任务组Si对应的注意力表征向量的具体实现方式可为：可以获取自注意力子网络层Vk所包含的注意力参数矩阵，随后可以将自注意力子网络层Vk所包含的注意力参数矩阵，与训练任务组Si对应的媒体特征进行运算处理，由此可以得到注意力参数矩阵对应的线性变换矩阵；进一步地，可以通过自注意力子网络层Vk中的全连接组件，对线性变换矩阵进行特征整合处理，由此可得到自注意力子网络层Vk对应的注意力表征子向量；当确定出Q个自注意力子网络层分别对应的注意力表征子向量时，可以将Q个注意力表征子向量进行融合处理，由此可以得到训练任务组Si对应的注意力表征向量。
可以理解的是，在某个自注意力子网络层中，所包含的注意力参数矩阵可以包括WQ、WK以及WV矩阵，可以将训练任务组Si对应的媒体特征分别与WQ、WK以及WV进行相乘处理，可以得到WQ矩阵对应的线性变换矩阵、WK矩阵对应的线性变换矩阵以及WV矩阵对应的线性变换矩阵。对于线性变换矩阵WQ与线性变换矩阵WK，可以将其进行相乘运算处理，对于这两者相乘运算的运算结果，可以将其输入至全连接层(如Softmax层)进行全连接处理(也就是特征整合处理)，对于全连接处理得到的结果，可以与线性变换矩阵WV又一次进行相乘运算处理，与线性变换矩阵WV又一次进行相乘运算处理所得到的运算结果，即可作为一个自注意力子网络层的输出结果(即注意力表征子向量)。每个自注意力子网络层中均包含有不同的WQ、WK以及WV矩阵，采用同样的原理，可以得到不同的注意力表征子向量，最后，可以将这些不同的注意力表征子向量进行融合处理，即可得到最终的注意力表征向量。
对于基于各个自注意力子网络层的注意力表征子向量，得到最终的注意力表征向量的具体实现方式可如公式(1)所示：
其中，Eattn可用于表征最终的注意力表征向量；可用于表征自注意力子网络层Hn的注意力表征子向量；Concat()可用于表征拼接处理。也就是说，可以将各个注意力表征子向量进行融合(拼接)，融合(拼接)后即可得到注意力表征向量。
步骤S103，在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层，基于目标路由层对训练任务组Si对应的注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量。
本申请中，基于上述可知，在模态表征模型所包含的编码器与解码器中，均会包含有多头注意力层与路由层，路由层可以将多头注意力层所输出的注意力表征向量进行特征整合处理(即路由处理，本申请称之为特征预测处理)，以得到最终的模态表征向量，该模态表征向量即可是指对训练任务组进行推理理解后所得到的特征，而为了提升对不同模态类型的数据的理解准确性，本申请可以将模态表征模型中的路由层设置为不同的模态路由层，将各个模态路由层与不同的模态类型对应起来，例如，可以将路由层设置为文本模态路由层、图像模态路由层、音频模态路由层、视频模态路由层以及跨模态路由层，文本模态路由层可以用于对模态类型为文本模态类型的数据进行处理，图像模态路由层可以用于对模态类型为图像模态类型的数据进行处理，音频模态路由层可以用于对模态类型为音频模态类型的数据进行处理，视频模态路由层可以用于对模态类型为视频模态类型的数据进行处理，而跨模态路由层可以对模态类型为跨模态类型的数据进行处理。对于文本模态路由层、图像模态路由层、音频模态路由层、视频模态路由层，均可以理解为是一个单模态路由层，那么也就是说，在模态表征模型中，可以同时包含有单模态路由层与跨模态路由层，那么基于此，对于训练任务组Si而言，在确定出训练任务组Si对应的注意力表征向量后，可以在模态表征模型所包含的单模态路由层与跨模态路由层中，确定出与训练任务组Si所属的模态类型相匹配的模态路由层作为目标路由层，再基于该目标路由层来对训练任务组Si对应的注意力表征向量进行特征预测处理，以得到该训练任务组Si对应的模态表征向量(预测模态表征向量)。
以单模态类型包含第一模态类型与第二模态类型，单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层为例，对于在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层的具体实现方式可为：若训练任务组Si所属的模态类型为跨模态类型，则可以将模态表征模型中所包含的跨模态路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；若训练任务组Si所属的模态类型为第一模态类型，则可以将第一模态类型对应的第一模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；而若训练任务组Si所属的模态类型为第二模态类型，则可以将第二模态类型对应的第二模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层。
可以理解的是，第一模态类型与第二模态类型可以是指不同的模态类型，例如，第一模态类型可以是指文本模态类型、图像模态类型、音频模态类型以及视频模态类型中的一种或多种，而第二模态类型则可以是指文本模态类型、图像模态类型、音频模态类型以及视频模态类型中，除第一模态类型以外的剩余的模态类型，那么在训练任务组Si所属的模态类型为文本模态类型时，即可将文本模态路由层确定为该训练任务组Si对应的目标路由层；在训练任务组Si所属的模态类型为图像模态类型时，即可将图像模态路由层确定为该训练任务组Si对应的目标路由层；在训练任务组Si所属的模态类型为音频模态类型时，即可将音频模态路由层确定为该训练任务组Si对应的目标路由层；在训练任务组Si所属的模态类型为视频模态类型时，即可将视频模态路由层确定为该训练任务组Si对应的目标路由层；在训练任务组Si所属的模态类型为跨模态类型时，即可将跨模态路由层确定为该训练任务组Si对应的目标路由层。
为便于理解路由分发机制，请一并参见图4，图4是本申请实施例提供的一种模态路由分发的架构示意图。如图4所示，单模态路由层具体可以为文本模态路由层、图像模态路由层、音频模态路由层或视频模态路由层，可以用于对对应的模态类型的数据进行路由处理。当输入特征为文本模态类型对应的媒体特征时，在通过多头注意力层输出注意力表征向量后，可以分发至文本模态路由层，由文本模态路由层对其进行路由处理；当输入特征为图像模态类型对应的媒体特征时，在通过多头注意力层输出注意力表征向量后，可以分发至图像模态路由层，由图像模态路由层对其进行路由处理；输入特征为音频模态类型对应的媒体特征时，在通过多头注意力层输出注意力表征向量后，可以分发至音频模态路由层，由音频模态路由层对其进行路由处理；输入特征为视频模态类型对应的媒体特征时，在通过多头注意力层输出注意力表征向量后，可以分发至视频模态路由层，由视频模态路由层对其进行路由处理；输入特征为多模态类型对应的媒体特征时，在通过多头注意力层输出注意力表征向量后，可以分发至跨模态路由层，由跨模态路由层对其进行路由处理。其中，每个模态路由层，均可以是指一个多层结构的全连接层。
步骤S104，当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化，基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为跨模态类型的任务组进行特征预测处理。
本申请中，当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，也可以由相应模态类型的训练任务组所对应的预测模态表征向量，来对相应的模态路由层进行训练优化处理。例如，可以基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，来对单模态路由层进行优化，可以基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，来对跨模态路由层进行优化。而优化后的单模态路由层，可以用于对所属模态类型为单模态类型的任务组进行特征预测处理；优化后的跨模态路由层，可以用于对所属模态类型为跨模态类型的任务组进行特征预测处理。具体的，基于所属模态类型为文本模态类型的训练任务组所对应的预测模态表征向量，可以对文本模态路由层进行训练优化，基于所属模态类型为图像模态类型的训练任务组所对应的预测模态表征向量，可以对图像模态路 由层进行训练优化，基于所属模态类型为音频模态类型的训练任务组所对应的预测模态表征向量，可以对音频模态路由层进行训练优化，基于所属模态类型为视频模态类型的训练任务组所对应的预测模态表征向量，可以对视频模态路由层进行训练优化。训练优化后的文本模态路由层，可以对文本数据进行特征预测处理，训练优化后的图像模态路由层，可以对图像数据进行特征预测处理，训练优化后的音频模态路由层，可以对音频数据进行特征预测处理，训练优化后的视频模态路由层，可以对视频数据进行特征预测处理。
以跨模态类型的目标任务组为例，可以基于优化后的跨模态路由层对其进行处理。具体的，可以获取用于描述待分类媒体数据的目标任务组；其中，目标任务组包括至少两个不同模态类型的媒体数据；随后，可以在模态表征模型中对目标任务组进行注意力交互处理，由此可以得到目标任务组对应的注意力表征向量；进一步地，可以在优化后的跨模态路由层中，对目标任务组对应的注意力表征向量进行特征预测处理，由此可以得到目标任务组对应的预测模态表征向量；随后，可以对目标任务组对应的预测模态表征向量进行类别识别处理，由此即可得到待分类媒体数据所属的媒体类别。在确定出待分类媒体数据所属的媒体类别后，即可将该待分类媒体数据按照所属的媒体类别进行归档处理。
可以理解的是，这里是以将媒体数据进行归档的场景为例，说明了本申请实施例可以应用的场景，假设待分类媒体数据为某个视频数据，那么可以将来自于该视频数据的文本数据与图像数据作为任务组，再通过该训练优化后的跨模态路由层对其进行推理理解，以得到该视频数据的准确的模态表征向量，基于该视频数据的模态表征向量，即可确定出该视频数据所属的媒体类别(如舞蹈类别、竞赛类别、体操类别等等)，通过该视频数据所属的媒体类别，即可准确地将视频数据进行归档处理。
当然，对于本申请实施例所提供的方法，应用场景并不限制于此，例如，还可以应用于多模态内容的检索场景中、媒体数据推荐的场景中等等，为便于理解，请一并参见图5，图5是本申请实施例提供的一种媒体数据检索的场景示意图。如图5所示，在视频显示界面3001中，当前向对象a曝光的视频数据为视频数据300a(该视频数据300a的创作者为名称为“啦啦爱跳舞”的对象，该视频数据300a的视频文案为“三分钟教你快速学会一支舞蹈”)，在该视频显示界面3001中显示有点赞控件、评论控件以及分享控件，可以用于对象将该视频数据进行点赞、评论以及分享。在该视频显示界面3001中，还包含有搜索控件30a，可以用于对象a进行搜索相关内容。如图3所示，在对象a对该搜索控件30a产生触发操作后，终端设备可以响应这一触发操作，在该视频显示界面3001中显示输入框，对象a可以在该输入框中输入期望查阅的内容。
如图3所示，对象a在该输入框中输入文本数据“红烧鲫鱼做法”，而终端设备可以获取到该对象a所输入的文本数据“红烧鲫鱼做法”，终端设备可将该文本数据“红烧鲫鱼做法”发送至业务服务器，业务服务器可以通过训练优化后的模态表征模型，对该文本数据“红烧鲫鱼做法”进行推理理解。具体的，通过模态表征模型中的多头自注意力网络层，可以输出该文本数据“红烧鲫鱼做法”对应的注意力表征向量，可以将该注意力表征向量分发至文本模态路由层，通过该文本模态路由层，即可输出该文本数据“红烧鲫鱼做法”所对应的模态表征向量。而基于该文本数据对应的模态表征向量，即可查找到与之相关的视频数据。如图3所示，假设与之相关的视频数据为视频数据300b(该视频数据300b的创作者为名称为“美美做美食”的对象，且该视频数据300b的视频文本为“鲫鱼买来这样做，香气扑鼻！”)，业务服务器可以将该视频数据300b返回至终端设备，而终端设备可以在该视频显示界面3001中显示该视频数据300b。
需要说明的是，在确定出文本数据对应的模态表征向量后，业务服务器可以基于不同向量之间的相似性，来确定两个数据之间是否是关联的。例如，对于上述文本数据“红烧鲫鱼做法”所对应的模态表征向量而言，可以先获取到候选视频数据的模态表征向量，再将文本数据“红烧鲫鱼做法”所对应的模态表征向量，与各个候选视频数据的模态表征向量分别计算相似度，相似度大于相似度阈值时，即可确定候选视频数据为文本数据“红烧鲫鱼做法”的关联视频数据。
在本申请实施例中，提供一种通用的基于模态路由分发机制的模态表征模型，可以同时为所属模态类 型为单模态类型的数据以及所属模态类型为跨模态类型的数据提供相应的数据处理功能。具体的，本申请可以采用包含单模态类型与跨模态类型的训练任务组集合，对模态表征模型进行训练，其中，在模态表征模型中，包含有单模态路由层与跨模态路由层，单模态路由层可以对所属模态类型为单模态类型的数据(注意力表征向量)进行特征预测处理，跨模态路由层可以对所属模态类型为跨模态类型的数据(注意力表征向量)进行特征预测处理，而在训练优化时，单模态路由层可以基于所属模态类型为所述单模态类型的训练任务组所对应的预测模态表征向量来进行优化，跨模态路由层可以由所属模态类型为所述跨模态类型的训练任务组所对应的预测模态表征向量来进行优化。基于此，可以使得模态表征模型中的单模态路由层与跨模态路由层能够同时进行训练优化，且在训练优化时，是基于不同模态类型的数据来分别进行训练优化，无需损失任何一种模态类型的数据，可以同时保持单模态路由层与跨模态路由层的性能，由此可以提升模态表征模型对单模态类型与跨模态类型的通用性与灵活性。综上可知，本申请可以在模态表征模型中设置路由分发机制，将路由层划分为单模态路由层与跨模态路由层，不同的路由层可以对不同模态类型的数据进行处理，在训练时也由不同模态类型的数据分别进行训练优化，每个路由层只会保留当前模态类型的相关信息，由此无需舍弃任何一种模态类型的数据，也能够同时输出跨模态类型与单模态类型的表征向量，可以很好地提升各种模态类型的数据的表征结果的准确性。综上，本申请可以在模态表征业务中，提升模态表征模型的通用性，提升模态表征结果的准确性。
进一步地，请参见图6，图6是本申请实施例提供的一种对单模态路由层进行优化的流程示意图。其中，该流程可以对应于上述图2所对应实施例中，对于基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化的流程。该流程是以单模态类型包含第一模态类型与第二模态类型，单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层为例进行说明的流程。如图6所示，该流程可以至少包括以下步骤S601-步骤S605：
步骤S601，将所属模态类型为第一模态类型的训练任务组，确定为第一模态训练任务组，将所属模态类型为第二模态类型的训练任务组，确定为第二模态训练任务组。
具体的，为便于区别，可以将所属模态类型为第一模态类型的训练任务组，命名为第一模态训练任务组，并将所属模态类型为第二模态类型的训练任务组，命名为第二模态训练任务组。
步骤S602，获取第一模态训练任务组对应的真实模态表征向量，以及第二模态训练任务组对应的真实模态表征向量。
具体的，可以获取到第一模态训练任务组对应的真实模态表征向量，以及第二模态训练任务组对应的真实模态表征向量。真实模态表征向量可以是指作为训练标签的真实的模态表征向量。
步骤S603，对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值。
具体的，可以将第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理(也就是计算出预测模态表征向量与真实模态表征向量之间的误差值)，由此可以得到第一模态训练任务组对应的第一损失值。以第一模态类型为文本模态类型为例，那么第一模态训练任务组则可以为文本词序列，第一模态训练任务组对应的预测模态表征向量由文本词序列中，每个文本词分别对应的预测表征特征所组成，且第一模态训练任务组对应的真实模态表征向量，由文本词序列中，每个文本词分别对应的真实表征特征所组成，那么此时对于对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值的具体实现方式可为：可以在文本词序列所包含的文本词中获取遮掩文本词；其中，遮掩文本词是指经过遮掩处理的文本词；随后，可以在第一模态训练任务组对应的预测模态表征向量中，获取遮掩文本词对应的预测表征特征，并在第一模态训练任务组对应的真实模态表征向量中，获取遮掩文本词对应的真实表征特征；可以确定遮掩文本词对应的预测表征特征，与遮掩文本词对应的真实表征特征之 间的第一特征相似度；随后，即可将第一特征相似度确定为第一模态训练任务组对应的第一损失值。
可以理解的是，对于文本词序列，本申请中，可以采用掩蔽策略将某个文本词进行掩蔽处理，例如，可以将文本词序列中的为名词属性、动词属性或位置属性的文本词，进行掩蔽处理，而在模态表征模型所输出的预测模态表征向量中，包含有该掩蔽文本词对应的预测表征特征，在真实模态表征向量中，也包含有该掩蔽文本词对应的真实表征特征，可以计算该预测表征特征与真实表征特征之间的相似度，而该预测表征特征与真实表征特征之间的相似度，即可作为该第一模态训练任务组对应的第一损失值。
同理，当第一模态类型为图像模态类型或音频模态类型时，也可以将第一模态训练任务组中的部分元素进行掩蔽，例如，当第一模态类型为图像模态类型时，第一模态训练任务组可以为图像数据，可以将该图像数据中某些像素区域进行掩蔽处理(具体可以将相互交集比例较高的像素区域进行掩蔽)，在模态表征模型所输出的预测模态表征向量中，包含有该掩蔽像素区域对应的预测表征特征，在真实模态表征向量中，也包含有该掩蔽像素区域对应的真实表征特征，同样可以计算该预测表征特征与真实表征特征之间的相似度，该预测表征特征于真实表征特征之间的相似度，即可作为该第一模态训练任务组对应的第一损失值。
需要说明的是，对于第一模态训练任务组，也可以不进行掩蔽处理，可以直接将预测模态表征向量与真实模态表征向量进行相似度计算处理，所得到的整体的相似度可以作为最终的第一损失值。还需要说明的是，除了可以通过计算相似度得到第一损失值的方式以外，还可以利用特征回归或区域分类目标的方式来计算预测值与真实值之间的损失值，本申请对于基于预测值与真实值计算损失值的方式，并不进行限制。
步骤S604，对第二模态训练任务组对应的预测模态表征向量，与第二模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第二模态训练任务组对应的第二损失值。
具体的，对于对第二模态训练任务组对应的预测模态表征向量，与第二模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第二模态训练任务组对应的第二损失值的具体方式，可以参考上述确定第一损失值的描述，二者原理相同，这里将不再进行赘述。
步骤S605，基于第一损失值对第一模态类型对应的第一模态子路由层进行优化，基于第二损失值对第二模态类型对应的第二模态子路由层进行优化。
具体的，基于第一损失值可以对第一模态类型对应的第一模态子路由层进行优化，而基于第二损失值可以对第二模态类型对应的第二模态子路由层进行优化。换言之，对于不同模态类型的模态子路由层，可以由相应的模态类型的预测值所得到的损失值来进行优化，具体可以是基于损失值来调整各个模态子路由层的参数，直至损失值满足收敛条件。其中，这里的收敛条件可以是指迭代次数达到迭代阈值(此迭代阈值可以基于实际的业务需求进行设定)，也可以是指损失值小于损失阈值，收敛条件可以基于具体业务需求进行相应设定，本申请对其不进行限制，特在此说明。
在本申请实施例中，可以在模态表征模型中设置路由分发机制，将路由层划分为单模态路由层与跨模态路由层，不同的路由层可以对不同模态类型的数据进行处理，在训练时也由不同模态类型的数据分别进行训练优化，每个路由层只会保留当前模态类型的相关信息，由此无需舍弃任何一种模态类型的数据，也能够同时输出跨模态类型与单模态类型的表征向量，可以很好地提升各种模态类型的数据的表征结果的准确性。
进一步地，请参见图7，图7是本申请实施例提供的一种对跨模态路由层进行优化的流程示意图。其中，该流程可以对应于上述图2所对应实施例中，对于基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化的流程，该流程是以训练任务组集合中，包含至少两个所属模态类型为跨模态类型的训练任务组，至少两个所属模态类型为跨模态类型的训练任务组，包含训练任务组Sj(j为正整数)，训练任务组Sj包括所属模态类型为第一模态类型的第三样本媒体数据，以及所属模态类型为第二模态类型的第四样本媒体数据为例进行说明的流程，此时，训练任务组Sj所对应的预测模 态表征向量中，可以包括第三样本媒体数据对应的预测表征特征，以及第四样本媒体数据对应的预测表征特征。如图7所示，该流程可以至少包括以下步骤S701-步骤S703：
步骤S701，确定第三样本媒体数据对应的预测表征特征，与第四样本媒体数据对应的预测表征特征之间的第二特征相似度。
具体的，对于跨模态检索的任务在于对象输入某个信息(如文本数据)后，平台可以召回与其相关联的图像/视频/音频内容。因此跨模态检索的核心思想就是进行不同模态间的对齐和一致，即匹配的图片和文本内容或音频内容、视频内容在特征空间最接近和相似，不能匹配的图片和文本内容或音频内容、视频内容在特征空间不一致。那么基于此，对于包含不同模态类型的训练任务组，需要确定第三样本媒体数据对应的预测表征特征，与第四样本媒体数据对应的预测表征特征之间的第二特征相似度，该第二特征相似度应该尽可能大，由此即可使得相匹配文本、图像、音频、视频在特征空间上是一致的。
步骤S702，将每个所属模态类型为跨模态类型的训练任务组，均确定为跨模态训练任务组，确定训练任务组Sj与至少两个跨模态训练任务组中，剩余跨模态训练任务组之间的第三特征相似度；剩余跨模态训练任务组是指至少两个跨模态训练任务组中，除训练任务组Sj以外的跨模态训练任务组。
具体的，基于上述可知，对于媒体来源渠道不同的样本媒体数据，需要将其的相似度控制得尽可能小，那么对于训练任务组Sj与其他各个跨模态训练任务组，需要将其产生的第三特征相似度进行逐步减小。
步骤S703，基于第二特征相似度与第三特征相似度，对跨模态路由层进行优化。
具体的，基于上述可知，对于第二特征相似度，本申请中，需要通过不断调整跨模态路由层的参数以将其进行极大化处理，同时，对于第三特征相似度，需要通过不断调整跨模态路由层的参数以将其进行极小化处理，直至第二特征相似度为最大值，且第三特征相似度为最小值时，确定该跨模态路由层满足了收敛条件，可以不再将跨模态路由层的参数进行调整，确定此时的跨模态路由层为最优。
在本申请实施例中，可以在模态表征模型中设置路由分发机制，将路由层划分为单模态路由层与跨模态路由层，不同的路由层可以对不同模态类型的数据进行处理，在训练时也由不同模态类型的数据分别进行训练优化，每个路由层只会保留当前模态类型的相关信息，由此无需舍弃任何一种模态类型的数据，也能够同时输出跨模态类型与单模态类型的表征向量，可以很好地提升各种模态类型的数据的表征结果的准确性。
进一步地，请参见图8，图8是本申请实施例提供的一种数据处理装置的结构示意图。该数据处理装置可以是运行于计算机设备中的一个计算机程序(包括程序代码)，例如该数据处理装置为一个应用软件；该数据处理装置可以用于执行图2所示的方法。如图8所示，该数据处理装置1可以包括：集合获取模块11、注意力处理模块12、路由层确定模块13、预测模块14、第一优化模块15以及第二优化模块16。
集合获取模块11，用于获取训练任务组集合；训练任务组集合中包括训练任务组Si，i为正整数；训练任务组Si所属模态类型为单模态类型或跨模态类型；在训练任务组Si所属模态类型为单模态类型时，训练任务组Si包含一个样本媒体数据；在训练任务组Si所属模态类型为跨模态类型时，训练任务组Si包含至少两个不同模态类型的样本媒体数据；
注意力处理模块12，用于在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；注意力交互处理用于将训练任务组Si中的不同元素进行充分交互；
路由层确定模块13，用于在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层；
预测模块14，用于基于目标路由层对训练任务组Si对应的注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量；
第一优化模块15，用于当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化；
第二优化模块16，用于基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为跨模态类型的任务组进行特征预测处理。
其中，集合获取模块11、注意力处理模块12、路由层确定模块13、预测模块14、第一优化模块15以及第二优化模块16的具体实现方式，可以参见上述图2所对应实施例中步骤S101-步骤S104的描述，这里将不再进行赘述。
在一个实施例中，集合获取模块11获取训练任务组集合的具体实现方式，包括：
获取N个样本媒体数据；N为正整数，N个媒体数据中的一个样本媒体数据所属的模态类型，为第一模态类型与第二模态类型中的任意一种；第一模态类型与第二模态类型不同；
基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对N个样本媒体数据进行任务组构建处理，得到训练任务组集合。
在一个实施例中，集合获取模块11基于N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对N个样本媒体数据进行任务组构建处理，得到训练任务组集合的具体实现方式，包括：
将N个样本媒体数据的第一样本媒体数据，与用于表征第一模态类型的第一标识进行组合，得到所属模态类型为第一模态类型的第一训练任务组；第一样本媒体数据是指N个样本媒体数据中，所属模态类型为第一模态类型的任一样本媒体数据；
将N个样本媒体数据中的第二样本媒体数据，与用于表征第二模态类型的第二标识进行组合，得到所属模态类型为第二模态类型的第二训练任务组；第二样本媒体数据是指N个样本媒体数据中，所属模态类型为第二模态类型的任一样本媒体数据；
基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组；
将第一训练任务组与第二训练任务组，均确定为所属模态类型为单模态类型的训练任务组，将所属模态类型为所述跨模态类型的训练任务组，与所属模态类型为所述单模态类型的训练任务组所组成的集合，确定为训练任务组集合。
在一个实施例中，第一样本媒体数据与第二样本媒体数据的数量均为一个或多个；一个或多个第一样本媒体数据中包括第一样本媒体数据Mj，j为正整数；所属模态类型为跨模态类型的训练任务组，包含第一样本媒体数据Mj对应的训练任务组；
集合获取模块11基于第一样本媒体数据与第二样本媒体数据分别对应的媒体来源渠道，对第一样本媒体数据与第二样本媒体数据进行跨模态组合处理，得到所属模态类型为跨模态类型的训练任务组的具体实现方式，包括：
将第一样本媒体数据Mj对应的媒体来源渠道，确定为目标媒体来源渠道；
将一个或多个第二样本媒体数据中，媒体来源渠道为目标媒体来源渠道的第二样本媒体数据，确定为第一样本媒体数据Mj对应的关联样本媒体数据；
将第一样本媒体数据、关联样本媒体数据、第一标识以及第二标识进行组合，得到第一样本媒体数据Mj对应的训练任务组。
在一个实施例中，注意力处理模块12在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量的具体实现方式，包括：
在模态表征模型中，通过特征提取网络层对训练任务组Si所包含的样本媒体数据进行特征提取处理，得到训练任务组Si对应的媒体特征；
通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，得到训练任务组Si对应的注意力表征向量。
在一个实施例中，模态表征模型中的多头自注意力网络层包括Q个自注意力子网络层；Q为正整数；Q个自注意力子网络层包括自注意力子网络层Vk，k为正整数；
注意力处理模块12通过模态表征模型中的多头自注意力网络层，对训练任务组Si对应的媒体特征进行多头自注意力处理，得到训练任务组Si对应的注意力表征向量的具体实现方式，包括：
获取自注意力子网络层Vk所包含的注意力参数矩阵，将自注意力子网络层Vk所包含的注意力参数矩阵，与训练任务组Si对应的媒体特征进行运算处理，得到注意力参数矩阵对应的线性变换矩阵；
通过自注意力子网络层Vk中的全连接组件，对线性变换矩阵进行特征整合处理，得到自注意力子网络层Vk对应的注意力表征子向量；
当确定出Q个自注意力子网络层分别对应的注意力表征子向量时，将Q个注意力表征子向量进行融合处理，得到训练任务组Si对应的注意力表征向量。
在一个实施例中，单模态类型包含第一模态类型与第二模态类型；单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层；
路由层确定模块13在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层的具体实现方式，包括：
若训练任务组Si所属的模态类型为跨模态类型，则将模态表征模型中所包含的跨模态路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；
若训练任务组Si所属的模态类型为第一模态类型，则将第一模态类型对应的第一模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层；
若训练任务组Si所属的模态类型为第二模态类型，则将第二模态类型对应的第二模态子路由层，确定为与训练任务组Si所属的模态类型相匹配的目标路由层。
在一个实施例中，单模态类型包含第一模态类型与第二模态类型；单模态路由层包括第一模态类型对应的第一模态子路由层，与第二模态类型对应的第二模态子路由层；
第一优化模块15基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化的具体实现方式，包括：
将所属模态类型为第一模态类型的训练任务组，确定为第一模态训练任务组，将所属模态类型为第二模态类型的训练任务组，确定为第二模态训练任务组；
获取第一模态训练任务组对应的真实模态表征向量，以及第二模态训练任务组对应的真实模态表征向量；
对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值；
对第二模态训练任务组对应的预测模态表征向量，与第二模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第二模态训练任务组对应的第二损失值；
基于第一损失值对第一模态类型对应的第一模态子路由层进行优化，基于第二损失值对第二模态类型对应的第二模态子路由层进行优化。
在一个实施例中，第一模态类型为文本模态类型，第一模态训练任务组为文本词序列；第一模态训练任务组对应的预测模态表征向量由文本词序列中，每个文本词分别对应的预测表征特征所组成；第一模态训练任务组对应的真实模态表征向量，由文本词序列中，每个文本词分别对应的真实表征特征所组成；
第一优化模块15对第一模态训练任务组对应的预测模态表征向量，与第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到第一模态训练任务组对应的第一损失值的具体实现方式，包括：
在文本词序列所包含的文本词中获取遮掩文本词；遮掩文本词是指经过遮掩处理的文本词；
在第一模态训练任务组对应的预测模态表征向量中，获取遮掩文本词对应的预测表征特征，在第一模 态训练任务组对应的真实模态表征向量中，获取遮掩文本词对应的真实表征特征；
确定遮掩文本词对应的预测表征特征，与遮掩文本词对应的真实表征特征之间的第一特征相似度；
将第一特征相似度确定为第一模态训练任务组对应的第一损失值。
在一个实施例中，训练任务组集合中，包含至少两个所属模态类型为跨模态类型的训练任务组；至少两个所属模态类型为跨模态类型的训练任务组，包含训练任务组Sj，j为正整数；训练任务组Sj包括所属模态类型为第一模态类型的第一样本媒体数据，以及所属模态类型为第二模态类型的第二样本媒体数据；训练任务组Sj所对应的预测模态表征向量中，包括第一样本媒体数据对应的预测表征特征，以及第二样本媒体数据对应的预测表征特征；
第二优化模块16基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化的具体实现方式，包括：
确定第一样本媒体数据对应的预测表征特征，与第二样本媒体数据对应的预测表征特征之间的第二特征相似度；
将每个所属模态类型为跨模态类型的训练任务组，均确定为跨模态训练任务组，确定训练任务组Sj与至少两个跨模态训练任务组中，剩余跨模态训练任务组之间的第三特征相似度；剩余跨模态训练任务组是指至少两个跨模态训练任务组中，除训练任务组Sj以外的跨模态训练任务组；
基于第二特征相似度与第三特征相似度，对跨模态路由层进行优化。
在一个实施例中，数据处理装置1还可以包括：任务组获取模块17、交互处理模块18、特征处理模块19以及类别识别模块20。
任务组获取模块17，用于获取用于描述待分类媒体数据的目标任务组；目标任务组包括至少两个不同模态类型的媒体数据；
交互处理模块18，用于在模态表征模型中对目标任务组进行注意力交互处理，得到目标任务组对应的注意力表征向量；
特征处理模块19，用于在优化后的跨模态路由层中，对目标任务组对应的注意力表征向量进行特征预测处理，得到目标任务组对应的预测模态表征向量；
类别识别模块20，用于对目标任务组对应的预测模态表征向量进行类别识别处理，得到待分类媒体数据所属的媒体类别；待分类媒体数据所属的媒体类别用于对待分类媒体数据进行归档处理。
其中，任务组获取模块17、交互处理模块18、特征处理模块19以及类别识别模块20的具体实现方式，可以参见上述图2所对应实施例中步骤S104中的描述，这里将不再进行赘述。
在本申请实施例中，可以在模态表征模型中设置路由分发机制，将路由层划分为单模态路由层与跨模态路由层，不同的路由层可以对不同模态类型的数据进行处理，在训练时也由不同模态类型的数据分别进行训练优化，每个路由层只会保留当前模态类型的相关信息，由此无需舍弃任何一种模态类型的数据，也能够同时输出跨模态类型与单模态类型的表征向量，可以很好地提升各种模态类型的数据的表征结果的准确性。
进一步地，请参见图9，图9是本申请实施例提供的一种计算机设备的结构示意图。如图9所示，上述计算机设备8000可以包括：处理器8001，网络接口8004和存储器8005，此外，上述计算机设备8000还包括：用户接口8003，和至少一个通信总线8002。其中，通信总线8002用于实现这些组件之间的连接通信。其中，用户接口8003可以包括显示屏(Display)、键盘(Keyboard)，可选用户接口8003还可以包括标准的有线接口、无线接口。网络接口8004可选的可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器8005可以是高速RAM存储器，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。存储器8005可选的还可以是至少一个位于远离前述处理器8001的存储装置。如图9所示，作为一种计算机可读存储介质的存储器8005中可以包括操作系统、网络通信模块、用户接口模块 以及设备控制应用程序。
在图9所示的计算机设备8000中，网络接口8004可提供网络通讯功能；而用户接口8003主要用于为用户提供输入的接口；而处理器8001可以用于调用存储器8005中存储的设备控制应用程序，以实现：
获取训练任务组集合；训练任务组集合中包括训练任务组Si，i为正整数；训练任务组Si所属模态类型为单模态类型或跨模态类型；在训练任务组Si所属模态类型为单模态类型时，训练任务组Si包含一个样本媒体数据；在训练任务组Si所属模态类型为跨模态类型时，训练任务组Si包含至少两个不同模态类型的样本媒体数据；
在模态表征模型中对训练任务组Si进行注意力交互处理，得到训练任务组Si对应的注意力表征向量；注意力交互处理用于将训练任务组Si中的不同元素进行充分交互；
在模态表征模型所包含的单模态路由层与跨模态路由层中，确定与训练任务组Si所属的模态类型相匹配的目标路由层，基于目标路由层对训练任务组Si对应的注意力表征向量进行特征预测处理，得到训练任务组Si对应的预测模态表征向量；
当确定出训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为单模态类型的训练任务组所对应的预测模态表征向量，对单模态路由层进行优化，基于所属模态类型为跨模态类型的训练任务组所对应的预测模态表征向量，对跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为跨模态类型的任务组进行特征预测处理。
应当理解，本申请实施例中所描述的计算机设备8000可执行前文图2所对应实施例中对该数据处理方法的描述，也可执行前文图8所对应实施例中对该数据处理装置1的描述，在此不再赘述。另外，对采用相同方法的有益效果描述，也不再进行赘述。
此外，这里需要指出的是：本申请实施例还提供了一种计算机可读存储介质，且上述计算机可读存储介质中存储有前文提及的数据处理的计算机设备8000所执行的计算机程序，且上述计算机程序包括程序指令，当上述处理器执行上述程序指令时，能够执行前文图2所对应实施例中对上述数据处理方法的描述，因此，这里将不再进行赘述。另外，对采用相同方法的有益效果描述，也不再进行赘述。对于本申请所涉及的计算机可读存储介质实施例中未披露的技术细节，请参照本申请方法实施例的描述。
上述计算机可读存储介质可以是前述任一实施例提供的数据处理装置或者上述计算机设备的内部存储单元，例如计算机设备的硬盘或内存。该计算机可读存储介质也可以是该计算机设备的外部存储设备，例如该计算机设备上配备的插接式硬盘，智能存储卡(smart media card，SMC)，安全数字(secure digital，SD)卡，闪存卡(flash card)等。进一步地，该计算机可读存储介质还可以既包括该计算机设备的内部存储单元也包括外部存储设备。该计算机可读存储介质用于存储该计算机程序以及该计算机设备所需的其他程序和数据。该计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。
本申请的一个方面，提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机程序，处理器执行该计算机程序，使得该计算机设备执行本申请实施例中一方面提供的方法。
本申请实施例的说明书和权利要求书及附图中的术语“第一”、“第二”等是用于区别不同对象，而非用于描述特定顺序。此外，术语“包括”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、装置、产品或设备没有限定于已列出的步骤或模块，而是可选地还包括没有列出的步骤或模块，或可选地还包括对于这些过程、方法、装置、产品或设备固有的其他步骤单元。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技 术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例提供的方法及相关装置是参照本申请实施例提供的方法流程图和/或结构示意图来描述的，具体可由计算机程序指令实现方法流程图和/或结构示意图的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。这些计算机程序指令可提供到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或结构示意图一个方框或多个方框中指定的功能的装置。这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或结构示意图一个方框或多个方框中指定的功能。这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或结构示意一个方框或多个方框中指定的功能的步骤。
以上所揭露的仅为本申请较佳实施例而已，当然不能以此来限定本申请之权利范围，因此依本申请权利要求所作的等同变化，仍属本申请所涵盖的范围。

Claims (15)

  1. 一种数据处理方法，其特征在于，所述方法由计算机设备所执行，所述方法包括：
    获取训练任务组集合；所述训练任务组集合中包括训练任务组Si，i为正整数；所述训练任务组Si所属模态类型为单模态类型或跨模态类型；在所述训练任务组Si所属模态类型为所述单模态类型时，所述训练任务组Si包含一个样本媒体数据；在所述训练任务组Si所属模态类型为所述跨模态类型时，所述训练任务组Si包含至少两个不同模态类型的样本媒体数据；
    在模态表征模型中对所述训练任务组Si进行注意力交互处理，得到所述训练任务组Si对应的注意力表征向量；所述注意力交互处理用于将所述训练任务组Si中的不同元素进行充分交互；
    在所述模态表征模型所包含的单模态路由层与跨模态路由层中，确定与所述训练任务组Si所属的模态类型相匹配的目标路由层，基于所述目标路由层对所述训练任务组Si对应的注意力表征向量进行特征预测处理，得到所述训练任务组Si对应的预测模态表征向量；
    当确定出所述训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为所述单模态类型的训练任务组所对应的预测模态表征向量，对所述单模态路由层进行优化，基于所属模态类型为所述跨模态类型的训练任务组所对应的预测模态表征向量，对所述跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为所述单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为所述跨模态类型的任务组进行特征预测处理。
  2. 根据权利要求1所述的方法，其特征在于，所述获取训练任务组集合，包括：
    获取N个样本媒体数据；N为正整数，所述N个媒体数据中的一个样本媒体数据所属的模态类型，为第一模态类型与第二模态类型中的任意一种；所述第一模态类型与所述第二模态类型不同；
    基于所述N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对所述N个样本媒体数据进行任务组构建处理，得到所述训练任务组集合。
  3. 根据权利要求1～2任一项所述的方法，其特征在于，所述基于所述N个样本媒体数据中，每个样本媒体数据分别所属的模态类型，对所述N个样本媒体数据进行任务组构建处理，得到所述训练任务组集合，包括：
    将所述N个样本媒体数据的第一样本媒体数据，与用于表征所述第一模态类型的第一标识进行组合，得到所属模态类型为所述第一模态类型的第一训练任务组；所述第一样本媒体数据是指所述N个样本媒体数据中，所属模态类型为所述第一模态类型的任一样本媒体数据；
    将所述N个样本媒体数据中的第二样本媒体数据，与用于表征所述第二模态类型的第二标识进行组合，得到所属模态类型为所述第二模态类型的第二训练任务组；所述第二样本媒体数据是指所述N个样本媒体数据中，所属模态类型为所述第二模态类型的任一样本媒体数据；
    基于所述第一样本媒体数据与所述第二样本媒体数据分别对应的媒体来源渠道，对所述第一样本媒体数据与所述第二样本媒体数据进行跨模态组合处理，得到所属模态类型为所述跨模态类型的训练任务组；
    将所述第一训练任务组与所述第二训练任务组，均确定为所属模态类型为所述单模态类型的训练任务组，将所述所属模态类型为所述跨模态类型的训练任务组，与所述所属模态类型为所述单模态类型的训练任务组所组成的集合，确定为所述训练任务组集合。
  4. 根据权利要求3所述的方法，其特征在于，所述第一样本媒体数据与所述第二样本媒体数据的数量均为一个或多个；一个或多个第一样本媒体数据中包括第一样本媒体数据Mj，j为正整数；所述所属模态类型为所述跨模态类型的训练任务组，包含所述第一样本媒体数据Mj对应的训练任务组；
    所述基于所述第一样本媒体数据与所述第二样本媒体数据分别对应的媒体来源渠道，对所述第一样本媒体数据与所述第二样本媒体数据进行跨模态组合处理，得到所属模态类型为所述跨模态类型的训练任务组，包括：
    将所述第一样本媒体数据Mj对应的媒体来源渠道，确定为目标媒体来源渠道；
    将一个或多个第二样本媒体数据中，媒体来源渠道为所述目标媒体来源渠道的第二样本媒体数据，确定为所述第一样本媒体数据Mj对应的关联样本媒体数据；
    将所述第一样本媒体数据、所述关联样本媒体数据、所述第一标识以及所述第二标识进行组合，得到所述第一样本媒体数据Mj对应的训练任务组。
  5. 根据权利要求1～4任一项所述的方法，其特征在于，所述在模态表征模型中对所述训练任务组Si进行注意力交互处理，得到所述训练任务组Si对应的注意力表征向量，包括：
    在所述模态表征模型中，通过特征提取网络层对所述训练任务组Si所包含的样本媒体数据进行特征提取处理，得到训练任务组Si对应的媒体特征；
    通过所述模态表征模型中的多头自注意力网络层，对所述训练任务组Si对应的媒体特征进行多头自注意力处理，得到所述训练任务组Si对应的注意力表征向量。
  6. 根据权利要求5所述的方法，其特征在于，所述模态表征模型中的多头自注意力网络层包括Q个自注意力子网络层；Q为正整数；所述Q个自注意力子网络层包括自注意力子网络层Vk，k为正整数；
    所述通过所述模态表征模型中的多头自注意力网络层，对所述训练任务组Si对应的媒体特征进行多头自注意力处理，得到所述训练任务组Si对应的注意力表征向量，包括：
    获取所述自注意力子网络层Vk所包含的注意力参数矩阵，将所述自注意力子网络层Vk所包含的注意力参数矩阵，与所述训练任务组Si对应的媒体特征进行运算处理，得到所述注意力参数矩阵对应的线性变换矩阵；
    通过自注意力子网络层Vk中的全连接组件，对所述线性变换矩阵进行特征整合处理，得到所述自注意力子网络层Vk对应的注意力表征子向量；
    当确定出Q个自注意力子网络层分别对应的注意力表征子向量时，将Q个注意力表征子向量进行融合处理，得到所述训练任务组Si对应的注意力表征向量。
  7. 根据权利要求1～6任一项所述的方法，其特征在于，所述单模态类型包含第一模态类型与第二模态类型；所述单模态路由层包括所述第一模态类型对应的第一模态子路由层，与所述第二模态类型对应的第二模态子路由层；
    所述在所述模态表征模型所包含的单模态路由层与跨模态路由层中，确定与所述训练任务组Si所属的模态类型相匹配的目标路由层，包括：
    若所述训练任务组Si所属的模态类型为所述跨模态类型，则将所述模态表征模型中所包含的所述跨模态路由层，确定为与所述训练任务组Si所属的模态类型相匹配的目标路由层；
    若所述训练任务组Si所属的模态类型为所述第一模态类型，则将所述第一模态类型对应的第一模态子路由层，确定为与所述训练任务组Si所属的模态类型相匹配的目标路由层；
    若所述训练任务组Si所属的模态类型为所述第二模态类型，则将所述第二模态类型对应的第二模态子路由层，确定为与所述训练任务组Si所属的模态类型相匹配的目标路由层。
  8. 根据权利要求1～7任一项所述的方法，其特征在于，所述单模态类型包含第一模态类型与第二模 态类型；所述单模态路由层包括所述第一模态类型对应的第一模态子路由层，与所述第二模态类型对应的第二模态子路由层；
    所述基于所属模态类型为所述单模态类型的训练任务组所对应的预测模态表征向量，对所述单模态路由层进行优化，包括：
    将所属模态类型为所述第一模态类型的训练任务组，确定为第一模态训练任务组，将所属模态类型为所述第二模态类型的训练任务组，确定为第二模态训练任务组；
    获取所述第一模态训练任务组对应的真实模态表征向量，以及所述第二模态训练任务组对应的真实模态表征向量；
    对所述第一模态训练任务组对应的预测模态表征向量，与所述第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到所述第一模态训练任务组对应的第一损失值；
    对所述第二模态训练任务组对应的预测模态表征向量，与所述第二模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到所述第二模态训练任务组对应的第二损失值；
    基于所述第一损失值对所述第一模态类型对应的第一模态子路由层进行优化，基于所述第二损失值对所述第二模态类型对应的第二模态子路由层进行优化。
  9. 根据权利要求8所述的方法，其特征在于，所述第一模态类型为文本模态类型，所述第一模态训练任务组为文本词序列；所述第一模态训练任务组对应的预测模态表征向量由所述文本词序列中，每个文本词分别对应的预测表征特征所组成；所述第一模态训练任务组对应的真实模态表征向量，由所述文本词序列中，每个文本词分别对应的真实表征特征所组成；
    所述对所述第一模态训练任务组对应的预测模态表征向量，与所述第一模态训练任务组对应的真实模态表征向量，进行误差计算处理，得到所述第一模态训练任务组对应的第一损失值，包括：
    在所述文本词序列所包含的文本词中获取遮掩文本词；所述遮掩文本词是指经过遮掩处理的文本词；
    在所述第一模态训练任务组对应的预测模态表征向量中，获取所述遮掩文本词对应的预测表征特征，在所述第一模态训练任务组对应的真实模态表征向量中，获取所述遮掩文本词对应的真实表征特征；
    确定所述遮掩文本词对应的预测表征特征，与所述遮掩文本词对应的真实表征特征之间的第一特征相似度；
    将所述第一特征相似度确定为所述第一模态训练任务组对应的第一损失值。
  10. 根据权利要求1～9任一项所述的方法，其特征在于，所述训练任务组集合中，包含至少两个所属模态类型为所述跨模态类型的训练任务组；至少两个所属模态类型为所述跨模态类型的训练任务组，包含训练任务组Sj，j为正整数；所述训练任务组Sj包括所属模态类型为第一模态类型的第三样本媒体数据，以及所属模态类型为第二模态类型的第四样本媒体数据；所述训练任务组Sj所对应的预测模态表征向量中，包括所述第三样本媒体数据对应的预测表征特征，以及所述第四样本媒体数据对应的预测表征特征；
    所述基于所属模态类型为所述跨模态类型的训练任务组所对应的预测模态表征向量，对所述跨模态路由层进行优化，包括：
    确定所述第三样本媒体数据对应的预测表征特征，与所述第四样本媒体数据对应的预测表征特征之间的第二特征相似度；
    将每个所属模态类型为所述跨模态类型的训练任务组，均确定为跨模态训练任务组，确定所述训练任务组Sj与至少两个跨模态训练任务组中，剩余跨模态训练任务组之间的第三特征相似度；所述剩余跨模态训练任务组是指所述至少两个跨模态训练任务组中，除所述训练任务组Sj以外的跨模态训练任务组；
    基于所述第二特征相似度与所述第三特征相似度，对所述跨模态路由层进行优化。
  11. 根据权利要求1～10任一项所述的方法，其特征在于，所述方法还包括：
    获取用于描述待分类媒体数据的目标任务组；所述目标任务组包括至少两个不同模态类型的媒体数据；
    在所述模态表征模型中对所述目标任务组进行注意力交互处理，得到所述目标任务组对应的注意力表征向量；
    在所述优化后的跨模态路由层中，对所述目标任务组对应的注意力表征向量进行特征预测处理，得到所述目标任务组对应的预测模态表征向量；
    对所述目标任务组对应的预测模态表征向量进行类别识别处理，得到所述待分类媒体数据所属的媒体类别；所述待分类媒体数据所属的媒体类别用于对所述待分类媒体数据进行归档处理。
  12. 一种数据处理装置，其特征在于，所述装置应用于计算机设备，所述装置包括：
    集合获取模块，用于获取训练任务组集合；所述训练任务组集合中包括训练任务组Si，i为正整数；所述训练任务组Si所属模态类型为单模态类型或跨模态类型；在所述训练任务组Si所属模态类型为所述单模态类型时，所述训练任务组Si包含一个样本媒体数据；在所述训练任务组Si所属模态类型为所述跨模态类型时，所述训练任务组Si包含至少两个不同模态类型的样本媒体数据；
    注意力处理模块，用于在模态表征模型中对所述训练任务组Si进行注意力交互处理，得到所述训练任务组Si对应的注意力表征向量；所述注意力交互处理用于将所述训练任务组Si中的不同元素进行充分交互；
    路由层确定模块，用于在所述模态表征模型所包含的单模态路由层与跨模态路由层中，确定与所述训练任务组Si所属的模态类型相匹配的目标路由层；
    预测模块，用于基于所述目标路由层对所述训练任务组Si对应的注意力表征向量进行特征预测处理，得到所述训练任务组Si对应的预测模态表征向量；
    第一优化模块，用于当确定出所述训练任务组集合中每个训练任务组分别对应的预测模态表征向量时，基于所属模态类型为所述单模态类型的训练任务组所对应的预测模态表征向量，对所述单模态路由层进行优化；
    第二优化模块，用于基于所属模态类型为所述跨模态类型的训练任务组所对应的预测模态表征向量，对所述跨模态路由层进行优化；优化后的单模态路由层，用于对所属模态类型为所述单模态类型的任务组进行特征预测处理，优化后的跨模态路由层，用于对所属模态类型为所述跨模态类型的任务组进行特征预测处理。
  13. 一种计算机设备，其特征在于，包括：处理器、存储器以及网络接口；
    所述处理器与所述存储器、所述网络接口相连，其中，所述网络接口用于提供网络通信功能，所述存储器用于存储计算机程序，所述处理器用于调用所述计算机程序，以使所述计算机设备执行权利要求1-11任一项所述的方法。
  14. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有计算机程序，所述计算机程序适于由处理器加载并执行权利要求1-11任一项所述的方法。
  15. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序，所述计算机程序存储在计算机可读存储介质中，所述计算机程序适于由处理器读取并执行，以使得具有所述处理器的计算机设备执行权利要求1-11任一项所述的方法。
PCT/CN2024/083332 2023-05-19 2024-03-22 一种数据处理方法、装置、设备以及可读存储介质 Ceased WO2024239783A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US19/233,363 US20250299072A1 (en) 2023-05-19 2025-06-10 Data processing method and apparatus, device, and readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310572043.6A CN116975016A (zh) 2023-05-19 2023-05-19 一种数据处理方法、装置、设备以及可读存储介质
CN202310572043.6 2023-05-19

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/233,363 Continuation US20250299072A1 (en) 2023-05-19 2025-06-10 Data processing method and apparatus, device, and readable storage medium

Publications (1)

Publication Number Publication Date
WO2024239783A1 true WO2024239783A1 (zh) 2024-11-28

Family

ID=88480436

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/083332 Ceased WO2024239783A1 (zh) 2023-05-19 2024-03-22 一种数据处理方法、装置、设备以及可读存储介质

Country Status (3)

Country Link
US (1) US20250299072A1 (zh)
CN (1) CN116975016A (zh)
WO (1) WO2024239783A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116975016A (zh) * 2023-05-19 2023-10-31 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质
CN118569865B (zh) * 2024-08-01 2024-10-01 四川千行你我科技股份有限公司 一种多平台聚合支付的数据处理方法及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112668671A (zh) * 2021-03-15 2021-04-16 北京百度网讯科技有限公司 预训练模型的获取方法和装置
CN113240056A (zh) * 2021-07-12 2021-08-10 北京百度网讯科技有限公司 多模态数据联合学习模型训练方法及装置
CN113762322A (zh) * 2021-04-22 2021-12-07 腾讯科技（北京）有限公司 基于多模态表示的视频分类方法、装置和设备及存储介质
CN114419514A (zh) * 2022-01-26 2022-04-29 腾讯科技（深圳）有限公司 一种数据处理方法、装置、计算机设备及存储介质
US20220382975A1 (en) * 2021-05-28 2022-12-01 Adobe Inc. Self-supervised document representation learning
US20220392637A1 (en) * 2021-06-02 2022-12-08 Neumora Therapeutics, Inc. Multimodal dynamic attention fusion
CN116975016A (zh) * 2023-05-19 2023-10-31 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112668671A (zh) * 2021-03-15 2021-04-16 北京百度网讯科技有限公司 预训练模型的获取方法和装置
CN113762322A (zh) * 2021-04-22 2021-12-07 腾讯科技（北京）有限公司 基于多模态表示的视频分类方法、装置和设备及存储介质
US20220382975A1 (en) * 2021-05-28 2022-12-01 Adobe Inc. Self-supervised document representation learning
US20220392637A1 (en) * 2021-06-02 2022-12-08 Neumora Therapeutics, Inc. Multimodal dynamic attention fusion
CN113240056A (zh) * 2021-07-12 2021-08-10 北京百度网讯科技有限公司 多模态数据联合学习模型训练方法及装置
CN114419514A (zh) * 2022-01-26 2022-04-29 腾讯科技（深圳）有限公司 一种数据处理方法、装置、计算机设备及存储介质
CN116975016A (zh) * 2023-05-19 2023-10-31 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质

Also Published As

Publication number Publication date
CN116975016A (zh) 2023-10-31
US20250299072A1 (en) 2025-09-25

Similar Documents

Publication Publication Date Title
CN114283351B (zh) 视频场景分割方法、装置、设备及计算机可读存储介质
WO2025123998A1 (zh) 多模态大语言模型训练方法、相关性计算及标签生成方法
CN116821457B (zh) 一种基于多模态大模型的智能咨询及舆情处理系统
WO2025001486A1 (zh) 一种推荐方法以及装置
US20250299072A1 (en) Data processing method and apparatus, device, and readable storage medium
WO2019154411A1 (zh) 词向量更新方法和装置
WO2023168818A1 (zh) 视频和文本相似度确定方法、装置、电子设备、存储介质
CN116821307B (zh) 内容交互方法、装置、电子设备和存储介质
CN112149642A (zh) 一种文本图像识别方法和装置
WO2025031067A1 (zh) 图像处理方法、装置、设备和计算机可读存储介质
CN115168609A (zh) 一种文本匹配方法、装置、计算机设备和存储介质
CN117789099B (zh) 视频特征提取方法及装置、存储介质及电子设备
EP4384987A1 (en) Method and system for selecting marker for modifying a scene within an augmented reality based computing environment
WO2024174804A1 (zh) 服务提供方法、设备和存储介质
CN114281934A (zh) 文本识别方法、装置、设备及存储介质
CN113761933B (zh) 检索方法、装置、电子设备及可读存储介质
CN115114545B (zh) 内容处理方法以及相关设备
CN115269961B (zh) 内容搜索方法以及相关设备
CN121189437A (zh) 一种基于大模型的图谱更新方法、装置、电子设备及介质
CN117032517A (zh) 基于电力工程项目管理的信息检索方法、设备及存储介质
CN115705705A (zh) 基于机器学习的视频识别方法、装置、服务器和存储介质
CN119599030B (zh) 一种问答处理方法、装置、介质及设备
CN116561350B (zh) 一种资源生成方法及相关装置
CN113505246B (zh) 数据处理方法、装置、终端设备以及存储介质
WO2023058899A1 (en) Method and system for selecting marker for modifying a scene within an augmented reality based computing environment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24810032

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE